The more networks have evolved and the more complex data center architectures have become, more and more organizations are realizing they've got time bombs latent in their network, just waiting for the right set of circumstances to take down critical portions of their infrastructure. In the past, a configuration error might cause a blip in the network that went unnoticed. With today’s networks so intimately tied to business, each blip can cost many millions of dollars and become front-page news, as seen in the accompanying headlines.
A surprisingly simple way out of this dilemma is network verification. With this new search, analysis and certification approach, it’s possible to analyze any network today in minutes to quickly find and eradicate these potential risks.
Our customers show us configuration errors that are as seemingly simple as a maximum transmission unit (MTU) size mismatches that went unnoticed until a new application revision moved to jumbo frames, or as complex as a failover error that triggered only when specific paths with slightly different configurations went down. Not only can these problems cause downtime and lost business, but our customers even see unidentified inefficiencies from configuration errors go on for months and years, degrading performance, service quality and driving up costs. A famous example of the latter was a major bank that added new spine switches to their network but did not configure them as part of the ECMP groups from the leaves for data traffic, so for months, their smokin’ fast new routers were acting as expensive space heaters.
Why do these problems persist despite the costs? Chalk it up to frequent changes, network complexity, poor documentation, inconsistencies across vendors, overloaded admins and the rush to keep up with business. But, the main culprit has to be that we've been taking the wrong approach to find, isolate and head off faults in the first place!
The process for network updates and change windows to align new policies and services with network behavior usually involves a great deal of testing. Test the connection from subnet-A to subnet-B. Test the new firewall configuration. Test ACLs with different traffic to specific applications. However, some problems arise when a real-world scenario doesn’t align with our test case: did you test a connection with a ping (ICMP packet), when the connection may behave differently for normal TCP traffic? Some problems arise when we miss a test case: did you test reachability but not across every alternate path? Some problems arise because there’s no easy way to test the complex reality of a complete network: did you test configurations box-by-box, but not consider every possible interaction of protocols, on all paths, under all packet sizes?
“Pretty Sure” may be the status quo, but it will never be good enough. “Pretty sure” has cost enterprises millions of dollars in downtime, and kept many network admins at the office over a long weekend. Instead, we want “Absolutely Sure”.
“Absolutely Sure” means taking a leap from testing just what we can think of, now, to confirming that every behavior in the network is intended. There’s even a name for new technology that moves us towards network nirvana: ‘Verification’.
A verification system doesn't merely test a finite number of specific scenarios. It runs a mathematical and logical analysis of the behavior of the network under all possible conditions, all device configurations, all forwarding states, and all end-to-end traffic paths - holistically. It doesn't rely on explicit packets or a single path, but will model all packet types under all possible paths and expose latent configuration problems, unexpected routes and open vulnerabilities. In essence, network verification can assure that your network is indeed a reflection of your business and policy intent (or not!).
To be more concrete, think of network verification like a UL certification for your network. A UL certification tests that an electronic product won’t catch fire, won’t emit electromagnetic waves, and doesn’t contain dangerous chemicals. Similarly, network verification tests that a network won’t see routing-loop fires, won’t leak packets, and doesn’t contain harmful configuration errors.
With a real verification platform, not only can network admins accelerate their workflows, but you've got automated reports that can verify compliance and target audit requirements. If a verification system can understand network behavior, it can make that available to the network admins and user to speed up the typical questions and tasks that live at the front of each trouble ticket, while making hours-long diagnoses a search away. If UL certification effectively assures that electronic devices won't cause major damage, how much more trust could you have in your network allowing you to sleep at night? What about the rest of your organization? Or your customers? What if you had a paper trail to deflect liability in case something did go wrong? Could that benefit your business or your career?
And perhaps the best part: Network Verification is available today. It doesn't have to be a disruptive technology to your environment. It can run on any existing network.
There should be no agents to install. No upgrades to the infrastructure. It's non-intrusive because a verification can be done away from your live network, with changes prototyped and tested in a virtual sand-box, eliminating any risk. Installation of a verification platform can take minutes, so you can start deriving benefits from day one, as soon as your topology and device configurations are collected.
(Note: Brandon Heller did a review of verification in other fields of technology in an earlier blog post, and why it's quickly becoming a critical requirement of all datacenter designs and processes.)
Want to know more? It turns out that network verification is becoming an interesting first step towards the new vision of Intent-Based Networking (it even says so here). Gartner Group also says, “Keep an eye out for Forward Networks…” in a blog about Intent-Based Networking. In part 2 of this blog post (coming soon), I will go into more detail about how we can deliver on network verification in our own Intent-Based Networking system, Forward Enterprise, and walk through some typical use cases.
Can't wait? For a live demo on how it all works and to see how it could model and analyze your network environment, sign up here.
[Special thanks to Brandon Heller and Matthias Schroeder who contributed to the development of this blog post.]