Did you hear about the change window that went exactly as planned? No? That’s because the odds of winning the PowerBall without buying a ticket are better than the odds of executing a change window on a global network without a glitch. 

What about the story of the tier one network engineer that diagnosed and resolved an ACL in seconds? That one also seems as mythical as staying friends with your ex—but it’s not. 

Instead of telling you the story, I want to show you how it’s done, which is why I recently hosted a workshop showcasing how we use search and intent verification within the Forward Networks Platform to tame ACLs (Access Control Lists). 

I’ve spent untold hours trying to troubleshoot an ACL issue after a change window and that was on a network I’d been running for decades, for tier-one admin, or even a more advanced engineer working on a new (or newly blended) network, it’s like trying to find a needle in a haystack while wearing a blindfold and being chased by rabid badgers.

On the face of it, the process for resolving ACL issues is pretty straightforward:

  1. Determine where your ACLs are running (which interfaces)
  2. Locate the ACL creating the issue
  3. Analyze the ACL to find the problem and resolve the issue

Except—networks have evolved over decades and include tens of thousands of devices from dozens of vendors and cloud providers running billions of lines of config. The fact is network complexity is outpacing IT support capabilities. Today, nothing about running a global network is straightforward without a comprehensive understanding of the network’s behavior and detailed visualization of traffic paths. 

Managing ACLs  shouldn’t be that hard

At Forward Networks, we think that the hard stuff should be easy, so we’ve done something unique. We developed a mathematical model that creates a network digital twin with Google-like search capabilities. By collecting and analyzing device state and packet forwarding data over time, we provide more than network visualization – we put the humans back in control of the network by providing them synthesized, actionable insights around network behavior. 

The Morning After the Change Window Before

The call comes in—a user can’t access an application – or worse, unauthorized users are accessing a secure app. What to do?  The network team always gets the call first, but the firewall tribe and security squad were also making changes – so how do you know which change created the problem?

The Forward Networks Platform (which functions as SaaS or be loaded onto an on-site VM) collects snapshots of the network over time including state data (ARP tables, route tables, interface tables, and so on) to develop a behavioral model of the network, providing detailed information on how packets are forwarded, filtered, and mutated. The end result is not only detailed visualization of the network but also advanced behavior modeling. For the ACL workshop, I focused on two ways to solve the issue, search and intent verification.

Search Two Ways: Text and Behavioral Path

Wouldn’t it be great if your network was indexed the same way the Internet is, and you could search it as easily as using Google? Ima ‘bout to rock your world by doing it right in front of your eyes.

Maybe you only know the IP address of a device that’s misbehaving. Our text search bar lets you enter that IP address (or any other atomic network information) and instantly gives you everything you need to know about that device (including which ACL rules/policies are applied to it). Maybe you want to search by ACL names—you can do that as well, and the platform returns config information with the ACL-related lines highlighted. This is ridiculously helpful when firewall configs have tens of thousands of lines. Now, even Tier-one support engineers can diagnose the problem and route it to the correct team with the context they need to immediately resolve the issue—no more searching manuals or paging through thousands of lines of config. 

By conducting a behavioral path search from the Internet to a specific application, you can see the exact path(s) traffic takes to the application in blue.  The gray lines denote detailed information about what happens to the packets as they flow through the network and the functions that are applied to them which is explained in the path’s pane. The platform serves up the relevant information without the network admin having to know details about the firewall or its syntax. The search shown above tells us that there is a path, and helps us easily identify that there are issues are with the firewall config, saving tons of time (conversely, it would tell us if the network path is broken). 

Behavioral searches can be saved as expected behaviors (intents) so that anytime the platform gathers information about the network, it will confirm that path is working as expected. In the workshop, I show how this function also can be used to verify if the “fix” applied by our friends in the tribe of firewall worked as expected (spoiler—it didn’t but network operations saves the day) without any risk to the production network, by using the predictive capabilities of the platform within the network digital twin. 

NQE – Your ACL management BFF 

In-App NQE (Network Query Engine), checks the data collected from the network and looks for states in the network that should (or should not) exist. For instance, an NQE Check can look for ACLs that are defined on a device but not applied to an interface. Custom checks can be written from inside the browser using syntax within the browser. There’s nothing to download; all of the reference information such as the data model and documentation is available within the browser window. This is a much better way to roll than my days of custom coding queries trying to pull information from the dozens of tabs I’ve opened to write code in the past. 

Sound interesting?  Watch the full ACL workshop (30 minutes of live-demo content). We host Forward Fix Live every month – On April 21, 2020 we’re going to dive deeper into one of our most popular features—NQE. There are two sessions, so no matter what time zone you are in! one for the East Coast and one

April 21, 2021 10:00 a.m. Eastern Time

April 21, 2021 10:00 a.m. Pacific Time

Only have a few minutes but you want to see more content by engineers for engineers?  Check out our YouTube playlist Forward Fixes – no hype, just actionable information, in roughly five-minute chunks. 

In network operations, it’s never the same day twice.

Most network engineers love this aspect, but it has a dark side. The best plans often fall to the wayside—in an instant work stops and firefighting begins.

In the last year, I’ve been part of a whole-day colo move, diagnosed an outage in the middle of the night, and resolved a slow performance issue. I know what the networking operations experience is like, and I know how much better it can be. 

Enabling others to solve every network problem at “global enterprise-scale”—faster and with more confidence is… let’s just say, very motivating. Especially when the networks are composed of multiple clouds, tens of thousands of devices, and are managed by multiple operations teams. I think about it like this:

If network behavior and insights were instantly available, you could speed up pretty much every network operations or engineering task.

In over seven years, I haven’t come across anyone who disagrees!  Everyone who has personally felt the stress of an outage, wasted a week tracking down a problem that ultimately was outside the network, or even spent too long with a simple ticket, doesn’t just agree—they feel it.

People in network operations and engineering wonder—is this even possible. The first questions are always of the “does it really work,” “how long will it take to set up,” “how much risk does it add,” and “can my team use it” variety. Not only do I hear these questions—I ask them of my vendors. Yes, it’s possible; we’ve been doing it at full scale for lots of companies you know, including Goldman Sachs for years.

Network operators and engineers don’t just need to see it to believe it. They need to deploy it, use it, and then have their coworkers use it, to believe it. 

The first step is seeing it. We joined Networking Field Day 24 to show what a day in a network operations professional’s life using the Forward Enterprise Platform looks like, from unboxing to integrations—covering killer use cases between. Instead of death-by-PPT, our field engineers, the technical experts who work side-by-side with our users to deploy Forward Enterprise, gave live demos and took questions. To make it easy for you to find content that’s relevant, we chunked it into short segments.

If the potential of instant network insight excites you—and you think maybe, just maybe—more time in the day could enable your team to be more proactive—then I’d like you to pick one thing you’ve recently had to spend time on, and check out the corresponding video below.

With the hands of our field team driving this, you’ll see what it’s like with the Forward Enterprise Platform. And if that passes your sniff test, as it’s done for many Fortune 500 enterprises already—reach out and schedule a personal demo. We’ll answer your toughest questions. We want to!

In fact, I dare you to pick one task from the list below that you or your team have done recently, and show me why instant access to info and insights WOULD NOT transform the speed of that task, and get your team on a path to faster, more proactive operations. 

Here’s what we covered, over a complete “day in the life”:

Unboxing to Up-to-Date, Searchable Network Model—15 minutes to Insight

Knowing the network topology’s detailed state is the first step in ensuring that your network is agile, predictable, and secure. Watch our Technical Solutions Architecture team leader, Elyor Khakimov, create a usable map and comprehensive collection of network data in less than 15 minutes without disrupting the network.

Path Analysis—Using Automation to Combat Complexity

After spending 20 years in the field helping network operations teams resolve issues, Technical Solution Architect Glen Turner knows that immediate access to actionable network behavior information is key to solving complex problems quickly. In this live demo, watch Glen use the search functionality within the Forward Networks Platform to analyze paths and reduce time spent troubleshooting to the seconds it takes him to type in a query into a search bar. 

Security Breach—Going back in time to resolve a leak

Need to find and resolve a data-leak issue but don’t have hours to do it? Armed with only four MAC address characters and the Forward Enterprise search bar, Senior Technical Solution Architect Scot Wilson shows how he’s used the Forward Networks platform to do it in four steps and under 10 minutes.

Audit—Search Billions of Lines of Config in Seconds

A simple typo caused a major network outage. The Forward Networks Network Query Engine (NQE) ‘s Google-like search capabilities helped resolve the issue in seconds – not hours. Customer Success Manager Jack Shen demonstrates how he did it and how NQE makes audits faster and more accurate.

Workflow Integrations—Solve Problems Faster by Getting the Right Data to the Right People

Without context, even the best applications only partially streamline ticket resolution. Senior Technical Solutions Architect Kevin Kuhls takes you through a live demonstration of our ServiceNow and Splunk integrations to show how quickly incidents can be resolved when context is automatically shared. 

Do you want to see more content by engineers for engineers and have only 5 minutes?  Check out our YouTube playlist Forward Fixes – no hype, just actionable information, in roughly five-minute chunks.

Still skeptical? I get it, and I challenge you to put us to the test, request a demo and give us your toughest challenges.

A new feature in Forward Enterprise now allows customers to simplify the analysis of network access issues between the network and security teams. We call this feature ACL-less analysis, or permit-all mode. First some context why multiple customers asked us to develop this feature, and the use case benefits they are seeing.

Forward Enterprise allows customers to quickly drill down into network and security configuration issues to isolate and expose the root cause of policy violations and deviations from intended network behavior. For example, why is this destination unreachable? Why is server access from the WAN impeded? What is blocking traffic between two sites or subnets? Forward Enterprise allows you to compare end-to-end path behavior with desired policies rather than focusing on individual device configurations and box-by-box analysis the old-fashioned way. Overall, this greatly accelerates Mean-Time-To-Repair (MTTR) and increases operational efficiency for IT teams.

When dealing with uncertain root-cause across large networks, many organizations are challenged to bridge the silos between network and security teams. It’s only natural. Visibility to both policies and implementations between two large technical organizations is rarely complete. It’s easy to start with a reasonable amount of finger-pointing. And when dealing with a connectivity or accessibility issues, sometimes it’s the network devices and topology, and sometimes it’s an unintended consequence of a security policy or access control issue.

When Forward Networks started putting our next-generation analytical tools and troubleshooting insights into the hands of large enterprise organizations, we uncovered some of these Layer 8 (political) problems ourselves. Several of our customers that have distinct network and security policy teams subsequently asked us to provide capability in our system to separate root-cause analysis between networking configuration and Access Control List (ACL) rules.

The motivation was at least two-fold: 1) It provides an immediate way of isolating any access or connectivity issues to network devices or security rules, and 2) It clearly indicates which team should be addressing the problem and further refines where remediation should best be applied. This usually decreases the MTTI (Mean Time to Innocence) for the networking team as well as avoiding tedious work and delays trying to definitively prove the lack of existence of some uncertain error.

How does this work in practice? Starting with the Verify view in Forward Enterprise, where a user has defined a set of policies to validate, we see a single failed policy check for the existence of at least one path between two IP addresses in different data centers, through a specific firewall, with traffic delivered between the sites via an MPLS backbone.

Clicking on the “failed” link allows the user to explore the configuration issues associated with this policy failure. This brings up a new view as depicted in Figure 2. The failing policy statement is displayed in the top search bar, which we can refine or broaden to help analyze the situation further.

The result of a Forward Enterprise query statement is always the full set of network paths that meet the requirements of the query or search. In this case, as expected, we see “No results found”, because no such path exists. All traffic is being dropped in this scenario between the two IP addresses 10.117.170.01 and 10.110.57.34. And no paths are highlighted in our topology diagram, only the individual devices included in the query.

At this point we don’t know if this is a network connectivity error, or security policy issue. The new permit-all mode in Forward Enterprise allows users to determine this immediately. By clicking on “permit-all mode”, the platform runs the same analytical query bypassing all the ACL rules, to see if there is network reachability and if traffic would flow in the absence of any security enforcement.

For those not familiar with Forward Enterprise, our platform is based on a behaviorally-accurate software model of your live network. These types of hypothetical analysis are very easy in our system, and never impact the live network where you can’t turn-off security enforcement just for the sake of analysis and testing. Checking the expected behavior of future traffic under any hypothetical change or scenario is one of many ways we aid in the analysis and troubleshooting of network and security issues.

In Figure 3, we see the top search updated with permit all, and now we are seeing that, indeed that are many (128) possible paths between these systems, due to the several pairs of redundant devices at most hops in the network. We are highlighting one path through the network, and focusing on an initial access layer switch that enforces ACLs.

We have highlighted in the hop details how the deny here, which is being applied to all packets, is being ignored, and the policy violation is not a network connectivity configuration issue after all. At this point, we can refer the ticket to the security team or administrator responsible for this particular device for further analysis or remediation. A key policy alert was detected, isolated and handed off to the responsible team in only a few clicks.

Another ACL-less scenario would be an application team wanting to know if the current network configuration supported access to a requested server. The current security policy would likely not support this policy a priori, but a key first step would be to know what network connectivity would allow in the absence of security rules. ACL-less analysis ignores the firewalls and ACL rules and can either confirm or deny network support for the application team request. This scenario is detailed in the YouTube video below.

This new capability, referred to as ACL-less or permit all mode, is having increasing interest across our entire user base that have separate security and network teams. We are interested to learn how it might help your organization and your IT processes in dealing with trouble tickets and how it may help overcome any Layer 8 problems you may have.

For more information, check out our YouTube video or get a live demo of ACL-less mode and the rest of the features in Forward Enterprise.

Top cross