Customer Case Study

Global Entertainment Company Saves Hundreds of IT Hours Every Year with
Forward Networks

Read how one Fortune 50 company increased the overall efficiency of the IT department by automating manual network tasks

The Company

A near century-old entertainment and media company knows a few things about resilience, agility, and the importance of generating the right kinds of headlines while preventing the bad ones.

Every day, this company entertains enough guests to populate a city the size of Buffalo, New York. Almost every visitor is carrying a cell phone that could lead to any mishap going viral, making predictability key to their operation.

The behind-the-scenes logistics of running such an operation are staggeringly complex. Maintaining this environment takes several hundred highly trained technical experts, managing a system that is comprised of tens of thousands of networking devices spread across an area roughly the same size as Manhattan. This network has been evolving for nearly 50 years and includes devices from every major networking vendor and connects to multiple public clouds.

The Environment

Tens of thousands of networking devices
Millions of endpoints
Hundreds of IT professionals
Tens of thousands of acres
The goals of the technical leadership team included:
Achieving a better understanding of the network to support modernization efforts.
Increasing the overall efficiency of the IT department by automating laborious and time-consuming manual tasks.
Continuously ensuring that the network configurations remain in-policy.

The Challenge

Over time and with the introduction of new technologies, the network had become difficult to manage. Seemingly simple tasks like locating devices and conducting path searches took prohibitively long. Like many companies with large networks, they lacked an accurate, in-depth understanding of network topology. These limitations inhibited efforts to modernize the network and caused an unnecessary number of expensive outages.

The Solution

The team turned to Forward Networks to help them achieve their goals. Specifically, they wanted to automate path searches and reduce Mean-Time-to-Resolution (MTTR) while enhancing situational awareness. The Forward Networks feature set easily delivered on these goals while creating a path for further modernization and optimization of the network.
 

 

Achieving a Better Understanding of the Network

The first step in utilizing Forward Enterprise is setting the software to automatically scan the network to build a digital twin of all devices, their state, and configuration. This collection is used to create a visual topology of the network and provides the information needed by the mathematical model to calculate all possible paths a packet could take.

The Network Query Engine (NQE) ingests the data collected by the Forward Enterprise platform, effectively turning the network into a database. Using plain language, engineers can search the network and extract detailed, actionable data for problem-solving and decision-making. According to one user, “NQE queries are solving problems that we couldn’t tackle before. We are much more efficient now and can take on projects that were impossible before because we have better knowledge of our network. I can’t wait to explore new use cases.”

Based on internal calculations across four regular maintenance tasks, a minuscule subset of the overall project scope, NQE reduced the time it would have taken to complete the tasks (e.g. understanding device utilization, finding mismatched timers, understanding port utilization) from over 700 person-hours in a year to just under 60 seconds in total.

The path search technology in Forward Enterprise reduced the amount of time spent on search and troubleshooting tasks by half, according to the company’s internal calculations. One of them engineers remarked, “I’m not sure if Forward has made me lazy or smart, but I don’t try to troubleshoot on my own anymore; I go straight to the platform for immediate answers.

Increasing Overall Efficiency

Another main goal of the company was to increase the efficiency of the IT team by automating tasks. Forward Enterprise is helping here as well. As part of its annual maintenance, the company tests its emergency power by shutting down the electricity and checking that backup sources such as UPS and generators are working. The IT department is tasked with verifying the state of all network devices before the test, while the main power source is cut, and after it’s restored. This manual process took a considerable number of people and a significant amount of time. In the event a device failed to come back online, or returned in a “hung” state, the only solution was physically deploying engineers on an expensive scavenger hunt throughout the location. Leveraging NQE helps them quickly identify devices that don’t come back online. Technicians are no longer going in search of devices that need their attention; they can go directly to an impacted device. The customer calls this a “game-changer” for their team.

Continuously Ensuring the Network is In Policy

Like most large enterprises, the customer struggled with ensuring network compliance in an environment that was constantly undergoing changes and upgrades. Given its size and the number of engineers working within the environment, it would have been impossible to ensure that all updates were in policy and would not cause performance or security issues. It would also be nearly impossible to determine what changes caused an incident after the fact. When hundreds of thousands of customers depend on the network, this is an untenable situation.

Using the custom verification feature in Forward Enterprise, administrators configured regular checks of the network to search for misconfigurations, lack of connectivity, or lack of redundancy. Being able to run these checks allows the team to proactively identify and remediate non-compliant issues before they cause an incident.

This capability helped the team prevent two potential outages which could have generated unwanted media attention. In both cases, the verification checks identified missing redundancy; had the main path failed without a backup in place, a brandnew attraction would have shut down and a park-wide evening performance would have been canceled.

Another feature the team finds extremely useful is behavioral diffs. When the network runs checks to verify compliance and device configuration and state, it saves a snapshot. The behavioral diffs function allows engineers to easily compare two snapshots and see exactly what changed in a given time frame. This makes it very easy to find the causes of unexpected network behavior.

Looking Forward

There are so many potential use cases for the Forward Networks platform that the company held a hackathon to find ways to extract value in the Fall of 2021. Many of the ideas brought to light during that event are already in production, and they plan to continue co-hosting this event with the Forward Networks engineering leadership.

Another opportunity they are exploring is replacing an aging network verification system with custom NQE checks. This will free up engineering resources previously dedicated to maintaining this system and provide additional features they didn’t have the resources to develop.

Lastly, they are starting to take advantage of Forward’s hybrid cloud functionality, which extends all the features of Forward Enterprise to AWS, GCP, and Azure. This provides additional value by delivering hop-by-hop visibility from the point of origin through the clouds and to the Internet. The company anticipates cost savings and enhanced security policy enforcement from having the ability to view their entire hybrid, multi-cloud network in a single pane of glass with the ability to drill down to individual network devices and cloud object instances.

Top cross