November 8, 2017

Recent Internet Outage Likely Prevented with Forward Networks Intent-based Verification

by Lisa Garvey

Monday morning this week, Internet access in many areas across the U.S. were severely compromised and routes through several carriers reported problems. Although remediation was fast (about 90 minutes), a simple router misconfiguration in the Border Gateway Protocol (BGP) set-up through one of the carriers appears to be the culprit. And this time it appears to be a simple human error.

Down Detector’s user-generated outage map shows the scope of the problem:

The particular error, as noted by several sources, appears to be a route leak between ISPs. ISP’s use Autonomous Systems (AS) to discover the best paths through other service provider networks using BGP. In this case misconfigured router caused one ISP to send inefficient or bad routing path information to others, causing packets to be delayed or dropped completely.

As Wired further reported:

Route leaks can be malicious, sometimes called "route hijacks" or "BGP hijacks," but Monday's incident seems to have been caused by a simple mistake that ballooned to have national impact. Large outages caused by accidental route leaks have cropped up before...

Internet outages of all sizes caused by route leaks have occurred occasionally, but consistently, for decades. ISPs attempt to minimize them using "route filters" that check the IP routes their peers and customers intend to use to send and receive packets, and attempt to catch any problematic plans. But these filters are difficult to maintain on the scale of the modern internet, and can have their own mistakes.

Monday's outages reinforce how precarious connectivity really is, and how certain aspects of the internet's architecture—offering flexibility and ease-of-use—can introduce instability into what has become a vital service.

The key takeaways from this event are:

  1. Basic network connectivity can hardly been taken for granted in a world of more complex global networks. The infrastructure seems more brittle than we often assume.
  2. Managing the infrastructure of any large network can be an overwhelming task when a tedious manual error on one device can bring down large sections of the system.
  3. Any small error resulting in large scale outage can bring business to a halt, impacting revenue and increasing exposure to business liability.

The final takeaway is that this is exactly the kind of misconfiguration error that Forward Enterprise is designed to detect and report, well before reaching this kind of performance degradation and business impact. Our intent-based verification system can find the needle in the haystack to focus on the potential errors and how they can be remediated, across very large telco-scale networks.

As Wired noted, state of the art in detecting routing leaks has traditionally relied on route filters that are difficult to maintain on the scale of the Internet. On the other hand, Forward Networks proactively identifies misconfiguration errors like this ahead of time because we compare the network intent (or policy requirements like network A sends to network C through B) to the de facto BGP router configurations and can quickly alert any discrepancy. Forward maintains a mathematical model of all device configurations and routing paths and would expose where route leaks exist and which devices need to be addressed.

It’s a revolutionary approach and platform to address a myriad of tedious manual network configuration errors and policy violations. Interested to learn more? Get a quick demo of how we can analyze your network device configurations, network states and head off any disruptions in your business.

Learn the modern way that companies stay off the front page

Experience a demo of the Forward Platform

Subscribe to our blog!

RELATED FORWARD CONTENT 
September 13, 2023
A Financial Services Company Saved “7 Figures” By Improving Network Inventory Management

Everyone knows inventory management is important – but so are the 100+ other things we need to do, and let’s face it, the inventory is not on fire. Given the benefits one customer experienced, maybe it should be.   On September 14, at 2:00pm Eastern time, we’re hosting a webinar, featuring special guests, Michael Wynston, Director of Network Architecture and […]

Read More
September 7, 2023
What’s worse than a toothache?

For me, I’d have to say it’s sitting through a high-pressure demo with a sales guy who needs to close business. Given the choice, I’ll take the dentist office visit anytime, at least they give you meds! We realize that sales demos aren’t always pleasant. And while we strive to create a comfortable environment for […]

Read More
August 23, 2023
How do you Monitor and Manage a Network Without Borders?

There are only two options for managing a global multi-cloud network: either by using a combination of inference, hope, and intuition or with mathematical certainty. When conducting 5 million financial transactions daily, it’s essential to operate with certainty, regardless of your network’s size or geographical distribution. Auditors don’t accept inferences; they demand certainty when determining […]

Read More
crossmenu linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram