SEE DEMO

November 8, 2017

Recent Internet Outage Likely Prevented with Forward Networks Intent-based Verification

by Charlie Elliott

Monday morning this week, Internet access in many areas across the U.S. were severely compromised and routes through several carriers reported problems. Although remediation was fast (about 90 minutes), a simple router misconfiguration in the Border Gateway Protocol (BGP) set-up through one of the carriers appears to be the culprit. And this time it appears to be a simple human error.

Down Detector’s user-generated outage map shows the scope of the problem:

Down Detector Outage Map

Source: downdetector.com

The particular error, as noted by several sources, appears to be a route leak between ISPs. ISP’s use Autonomous Systems (AS) to discover the best paths through other service provider networks using BGP. In this case misconfigured router caused one ISP to send inefficient or bad routing path information to others, causing packets to be delayed or dropped completely.

As Wired further reported:

Route leaks can be malicious, sometimes called "route hijacks" or "BGP hijacks," but Monday's incident seems to have been caused by a simple mistake that ballooned to have national impact. Large outages caused by accidental route leaks have cropped up before...

Internet outages of all sizes caused by route leaks have occurred occasionally, but consistently, for decades. ISPs attempt to minimize them using "route filters" that check the IP routes their peers and customers intend to use to send and receive packets, and attempt to catch any problematic plans. But these filters are difficult to maintain on the scale of the modern internet, and can have their own mistakes.

Monday's outages reinforce how precarious connectivity really is, and how certain aspects of the internet's architecture—offering flexibility and ease-of-use—can introduce instability into what has become a vital service.

The key takeaways from this event are:

  1. Basic network connectivity can hardly been taken for granted in a world of more complex global networks. The infrastructure seems more brittle than we often assume.
  2. Managing the infrastructure of any large network can be an overwhelming task when a tedious manual error on one device can bring down large sections of the system.
  3. Any small error resulting in large scale outage can bring business to a halt, impacting revenue and increasing exposure to business liability.

The final takeaway is that this is exactly the kind of misconfiguration error that Forward Enterprise is designed to detect and report, well before reaching this kind of performance degradation and business impact. Our intent-based verification system can find the needle in the haystack to focus on the potential errors and how they can be remediated, across very large telco-scale networks.

As Wired noted, state of the art in detecting routing leaks has traditionally relied on route filters that are difficult to maintain on the scale of the Internet. On the other hand, Forward Networks proactively identifies misconfiguration errors like this ahead of time because we compare the network intent (or policy requirements like network A sends to network C through B) to the de facto BGP router configurations and can quickly alert any discrepancy. Forward maintains a mathematical model of all device configurations and routing paths and would expose where route leaks exist and which devices need to be addressed.

It’s a revolutionary approach and platform to address a myriad of tedious manual network configuration errors and policy violations. Interested to learn more? Get a quick demo of how we can analyze your network device configurations, network states and head off any disruptions in your business.

Learn the modern way that companies stay off the front page

Experience a demo of the Forward Platform

Subscribe to our blog!

RELATED FORWARD CONTENT 
June 23, 2022
What’s it Like to work at a Great Place 

Forward Networks just was named a Great Place to Work in the Bay Area by Fortune. We’re excited to be on this list in an area known for setting the bar when it comes to treating employees well. Perks are nice, but perks alone don’t get 100% of employees to say they work at a […]

Read More
June 20, 2022
Forward Networks named one of the Best Workplaces in the Bay Area™ in 2022 by Great Place to Work® and Fortune Magazine

SANTA CLARA, Calif., June 20, 2022 /PRNewswire/ -- Great Place to Work and Fortune magazine have honored Forward Networks as one of this year's Best Workplaces in the Bay Area. This is Forward Networks' first time applying for and being named to this prestigious list. Earning a spot means that Forward Networks is one of the best companies to work […]

Read More
June 7, 2022
Forward Networks Wins 2022 Fortress Cyber Security Award

SANTA CLARA, Calif., June 7, 2022 /PRNewswire/ -- The Business Intelligence Group today announced that Forward Networks has won the 2022 Fortress Cyber Security Awards in the Network Security category. The industry awards program sought to identify and reward the world's leading companies and products that are working to keep our data and electronic assets safe among a growing […]

Read More

Sign up for our newsletter

clockcrossmenu linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram