November 8, 2017

Recent Internet Outage Likely Prevented with Forward Networks Intent-based Verification

by Charlie Elliott

Monday morning this week, Internet access in many areas across the U.S. were severely compromised and routes through several carriers reported problems. Although remediation was fast (about 90 minutes), a simple router misconfiguration in the Border Gateway Protocol (BGP) set-up through one of the carriers appears to be the culprit. And this time it appears to be a simple human error.

Down Detector’s user-generated outage map shows the scope of the problem:

Down Detector Outage Map

Source: downdetector.com

The particular error, as noted by several sources, appears to be a route leak between ISPs. ISP’s use Autonomous Systems (AS) to discover the best paths through other service provider networks using BGP. In this case misconfigured router caused one ISP to send inefficient or bad routing path information to others, causing packets to be delayed or dropped completely.

As Wired further reported:

Route leaks can be malicious, sometimes called "route hijacks" or "BGP hijacks," but Monday's incident seems to have been caused by a simple mistake that ballooned to have national impact. Large outages caused by accidental route leaks have cropped up before...

Internet outages of all sizes caused by route leaks have occurred occasionally, but consistently, for decades. ISPs attempt to minimize them using "route filters" that check the IP routes their peers and customers intend to use to send and receive packets, and attempt to catch any problematic plans. But these filters are difficult to maintain on the scale of the modern internet, and can have their own mistakes.

Monday's outages reinforce how precarious connectivity really is, and how certain aspects of the internet's architecture—offering flexibility and ease-of-use—can introduce instability into what has become a vital service.

The key takeaways from this event are:

  1. Basic network connectivity can hardly been taken for granted in a world of more complex global networks. The infrastructure seems more brittle than we often assume.
  2. Managing the infrastructure of any large network can be an overwhelming task when a tedious manual error on one device can bring down large sections of the system.
  3. Any small error resulting in large scale outage can bring business to a halt, impacting revenue and increasing exposure to business liability.

The final takeaway is that this is exactly the kind of misconfiguration error that Forward Enterprise is designed to detect and report, well before reaching this kind of performance degradation and business impact. Our intent-based verification system can find the needle in the haystack to focus on the potential errors and how they can be remediated, across very large telco-scale networks.

As Wired noted, state of the art in detecting routing leaks has traditionally relied on route filters that are difficult to maintain on the scale of the Internet. On the other hand, Forward Networks proactively identifies misconfiguration errors like this ahead of time because we compare the network intent (or policy requirements like network A sends to network C through B) to the de facto BGP router configurations and can quickly alert any discrepancy. Forward maintains a mathematical model of all device configurations and routing paths and would expose where route leaks exist and which devices need to be addressed.

It’s a revolutionary approach and platform to address a myriad of tedious manual network configuration errors and policy violations. Interested to learn more? Get a quick demo of how we can analyze your network device configurations, network states and head off any disruptions in your business.

Learn the modern way that companies stay off the front page

Experience a demo of the Forward Platform

Subscribe to our blog!

RELATED FORWARD CONTENT 
November 10, 2022
Why You Should Care About Vendor Hack Weeks

Hack weeks and hack-a-thons are like foosball tables; if you don’t have them, are you even a tech company? These events, once revered for innovation, are now relegated to being blasé and often perceived as little more than playtime for engineers. As someone who’s worked in tech for longer than I care to admit, I […]

Read More
October 13, 2022
How do you know a software rep is lying?

A customer posed this question to me recently; after pausing and smiling (a little too) broadly, he continued, “Their lips are moving.” I thought this would be funnier if it weren’t partly true. The software industry has over-promised and under-delivered for years, making technical executives rightfully skeptical when they hear a new promise. Unfortunately, it’s […]

Read More
September 20, 2022
When is a Digital Twin Entertaining?

When it’s ensuring that tens of thousands of visitors have the best experience possible every single day. Keeping people entertained is a 24/7 endeavor, even the smallest hiccup results in a social media firestorm. Keeping things running requires thousands of dedicated employees and a staggeringly complex network that sprawls the area of a major city […]

Read More

Sign up for our newsletter

crossmenu linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram