February 23, 2017

Network path not found: How to use an army of tests to understand and diagnose your network

by Peyman Kazemian

You’ve probably heard about how hard it is to operate networks. The complexity of the devices, the diversity of protocols operating simultaneously, and the sheer size of the network are all well-known reasons for getting something wrong. As a result, network changes require stressful, carefully planned change windows to ensure the desired outcome, with no unintended consequences.

In this post, I want to discuss an additional dimension to the complexity of the problem: cross-vendor, cross-platform and cross-version behavior differences. If not well understood or glanced over by the network operator, these little-known differences may result in security breaches, connectivity loss or network downtime, even when they affect a single device.

A Simple Firewall Configuration, Right?

Imagine you are the network operator for an enterprise network. All the hosts in your network are firewalled and NATed by a Palo Alto Security Platform. All your hosts have a private IP address and their access is restricted by the firewall to match the security policy of the organization. In particular, the security policy of the organization dictates that the host with IP address, which hosts sensitive financial data, should have all of its web communications to the outside be encrypted. Here is the configuration you may put on the the Palo Alto box:

NAT configuration and security rules on Palo Alto Security Platform

This configuration simply translates the private IP address of to a public IP address of, and denies any web browsing application other than SSL-based when it comes from

After a few months, as part of a hardware refresh, you are asked to replace this firewall with a Cisco ASA. The box you purchased is running ASA OS version 8.2. You do a bit of research online and translate each piece of config in your Palo Alto device to equivalent ASA configuration:

Pretty simple, right? Not so much…

We just introduced a security hole.

The subtle catch here is that the order of applying NAT and firewall rules can change the outcome of the configuration above. If the device applies firewall rules first, then NATs the traffic, the above configurations will work fine. But as it turns out, Palo Alto firewalls perform Firewall and NAT in that order, but on ASA versions 8.2 and lower, firewall rules are applied after the translation and therefore the rules matching on as source IP address will not have any effect. Instead, we should have applied those rules to the post-translated address,

Six months later, Cisco comes out with ASA version 8.4 and you decide to upgrade. You may think this time everything should be fine. After all, it’s the same vendor! But, not quite… To start, you notice some syntax differences in how you define NATs, which with the help of CLI, you fix. But at this point, can you be sure that the new configuration meets your desired behavior?

As it turns out, the answer is again, no! Going from version 8.2 to 8.4, the order of applying NAT and Firewall is changed and now it matches the original Palo Alto firewall! Bummer!

As the example above illustrates, different vendors, and even different OS versions or platforms coming from the same vendor may have subtle differences in the implementation of the same concept, protocol or standard.

While syntactical differences can be caught by the device’s CLI or management UI, behavioral differences are very hard to catch.

What makes it even worse is that sometimes these subtleties are not well-documented. They can lead to error-prone configurations, security breaches, or network downtime if not well understood by network operators.


Even the simplest configuration constructs have surprisingly large differences between implementations. We have seen these subtleties mislead network operators in some of our customer networks in the past. For example, some vendors require each VLAN to be explicitly declared before using them on an access or trunk interface (e.g. Cisco IOS and NX-OS), and forgetting to do so will silently drop any traffic arriving on that interface with the undefined VLAN tag.

Similarly, some vendors enforce referential consistency, which means that each object used in ACLs should be defined, while some others don’t have such requirements (e.g. many Cisco routers). In these cases, the ACL rule that references an undefined object will be ignored. As a result, a simple typo in the naming of an object may open up a security hole in the network.

In the best case, not fully understanding these nuances can waste hours of your time; in the worst case, it can lead to a security breach that puts your company in the news.

Subscribe to our blog!

September 13, 2023
A Financial Services Company Saved “7 Figures” By Improving Network Inventory Management

Everyone knows inventory management is important – but so are the 100+ other things we need to do, and let’s face it, the inventory is not on fire. Given the benefits one customer experienced, maybe it should be.   On September 14, at 2:00pm Eastern time, we’re hosting a webinar, featuring special guests, Michael Wynston, Director of Network Architecture and […]

Read More
September 7, 2023
What’s worse than a toothache?

For me, I’d have to say it’s sitting through a high-pressure demo with a sales guy who needs to close business. Given the choice, I’ll take the dentist office visit anytime, at least they give you meds! We realize that sales demos aren’t always pleasant. And while we strive to create a comfortable environment for […]

Read More
August 23, 2023
How do you Monitor and Manage a Network Without Borders?

There are only two options for managing a global multi-cloud network: either by using a combination of inference, hope, and intuition or with mathematical certainty. When conducting 5 million financial transactions daily, it’s essential to operate with certainty, regardless of your network’s size or geographical distribution. Auditors don’t accept inferences; they demand certainty when determining […]

Read More
crossmenu linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram