SEE DEMO

February 23, 2017

Network path not found?

by Charlie Elliott

[vc_row][vc_column width="1/1"][vc_custom_heading heading_semantic="h1" text_size="h1"]Network path not found?[/vc_custom_heading][/vc_column][/vc_row][vc_row row_height_percent="0" overlay_alpha="50" gutter_size="3" shift_y="0" desktop_visibility="yes" medium_visibility="yes" mobile_visibility="yes"][vc_column][vc_column_text]

How to use an army of tests to understand and diagnose your network!

You’ve probably heard about how hard it is to operate networks. The complexity of the devices, the diversity of protocols operating simultaneously, and the sheer size of the network are all well-known reasons for getting something wrong. As a result, network changes require stressful, carefully planned change windows to ensure the desired outcome, with no unintended consequences.

In this post, I want to discuss an additional dimension to the complexity of the problem: cross-vendor, cross-platform and cross-version behavior differences. If not well understood or glanced over by the network operator, these little-known differences may result in security breaches, connectivity loss or network downtime, even when they affect a single device.

A Simple Firewall Configuration, Right?

Imagine you are the network operator for an enterprise network. All the hosts in your network are firewalled and NATed by a Palo Alto Security Platform. All your hosts have a private IP address and their access is restricted by the firewall to match the security policy of the organization. In particular, the security policy of the organization dictates that the host with IP address 10.1.1.10, which hosts sensitive financial data, should have all of its web communications to the outside be encrypted. Here is the configuration you may put on the the Palo Alto box:

NAT configuration and security rules on Palo Alto Security Platform

This configuration simply translates the private IP address of 10.1.1.10 to a public IP address of 2.2.2.10, and denies any web browsing application other than SSL-based when it comes from 10.1.1.10.

After a few months, as part of a hardware refresh, you are asked to replace this firewall with a Cisco ASA. The box you purchased is running ASA OS version 8.2. You do a bit of research online and translate each piece of config in your Palo Alto device to equivalent ASA configuration:

NAT configuration and security rules on Cisco ASA Platform running OS version 8.2

Pretty simple, right? Not so much…

We just introduced a security hole.

The subtle catch here is that the order of applying NAT and firewall rules can change the outcome of the configuration above. If the device applies firewall rules first, then NATs the traffic, the above configurations will work fine. But as it turns out, Palo Alto firewalls perform Firewall and NAT in that order, but on ASA versions 8.2 and lower, firewall rules are applied after the translation and therefore the rules matching on 10.1.1.10 as source IP address will not have any effect. Instead, we should have applied those rules to the post-translated address, 2.2.2.10.

The order of applying NAT and firewall rules matters!!

Six months later, Cisco comes out with ASA version 8.4 and you decide to upgrade. You may think this time everything should be fine. After all, it’s the same vendor! But, not quite… To start, you notice some syntax differences in how you define NATs, which with the help of CLI, you fix. But at this point, can you be sure that the new configuration meets your desired behavior?

NAT configuration and security rules on Cisco ASA Platform, running OS version 8.4

As it turns out, the answer is again, no! Going from version 8.2 to 8.4, the order of applying NAT and Firewall is changed and now it matches the original Palo Alto firewall! Bummer!

As the example above illustrates, different vendors, and even different OS versions or platforms coming from the same vendor may have subtle differences in the implementation of the same concept, protocol or standard.

While syntactical differences can be caught by the device’s CLI or management UI, behavioral differences are very hard to catch.

What makes it even worse is that sometimes these subtleties are not well-documented. They can lead to error-prone configurations, security breaches or network downtime, if not well understood by network operators.

It’s not just Firewalls

Even the simplest configuration constructs have surprisingly large differences between implementations. We have seen these subtleties mislead network operators in some of our customer networks in the past. For example, some vendors require each VLAN to be explicitly declared before using them on an access or trunk interface (e.g. Cisco IOS and NX-OS), and forgetting to do so will silently drop any traffic arriving on that interface with the undefined VLAN tag.

Similarly, some vendors enforce referential consistency, which means that each object used in ACLs should be defined, while some others don’t have such requirement (e.g. many Cisco routers). In these cases, the ACL rule that references an undefined object will be ignored. As a result, a simple typo in naming of an object may open up a security hole in the network.

In the best case, not fully understanding these nuances can waste hours of your time; in the worst case, it can lead to a security breach that puts your company in the news.

How do we deal with these hidden subtleties at Forward Networks?

Imagine the most complex enterprise network you can think of, running every type of device from every major vendor, along with pretty much every single OS version in use - as one customer put it, a “museum of network history”. This is one key challenge we face: to comprehensively test a product that works with and understands today’s deployed networks.

To scale out our testing while keeping our sanity, we’ve developed a comprehensive framework that automatically discovers and tests these nuances across a massive set of carefully crafted test configurations running on racks of devices from various vendors. Every day, we load these devices with a diverse range of OS versions to discover and confirm any cross-vendor, cross-platform and cross-version behavioral differences.

Why go through this pain to understand everything in such great detail? Simply put, an accurate model of network behavior enables a new, and better, network admin experience.

At Forward Networks, we provide a single pane of glass through which network operators can see an accurate end-to-end picture of how traffic will be forwarded in the network. This enables network operators to ensure that the network is always secure, always forwarding traffic where they want, diagnose tickets faster, upgrade to newer OS versions, and add new devices, all without placing the network at risk. After seeing this for the first time, (like at our Networking Field Day launch), many show their excitement:[/vc_column_text][/vc_column][/vc_row][vc_row row_height_percent="0" override_padding="yes" h_padding="2" top_padding="0" bottom_padding="0" overlay_alpha="50" gutter_size="3" shift_y="0" desktop_visibility="yes" medium_visibility="yes" mobile_visibility="yes"][vc_column][vc_raw_html]JTNDYmxvY2txdW90ZSUyMGNsYXNzJTNEJTIydHdpdHRlci10d2VldCUyMiUyMGRhdGEtbGFuZyUzRCUyMmVuJTIyJTNFJTNDcCUyMGxhbmclM0QlMjJlbiUyMiUyMGRpciUzRCUyMmx0ciUyMiUzRU5ldyUyMCUzQ2ElMjBocmVmJTNEJTIyaHR0cHMlM0ElMkYlMkZ0d2l0dGVyLmNvbSUyRmhhc2h0YWclMkZORkQxMyUzRnNyYyUzRGhhc2glMjIlM0UlMjNORkQxMyUzQyUyRmElM0UlMjBibG9nJTIwcG9zdCUzQSUyMCUzQ2ElMjBocmVmJTNEJTIyaHR0cHMlM0ElMkYlMkZ0LmNvJTJGRkM4VmdlYmc4VyUyMiUzRWh0dHBzJTNBJTJGJTJGdC5jbyUyRkZDOFZnZWJnOFclM0MlMkZhJTNFJTNDJTJGcCUzRSUyNm1kYXNoJTNCJTIwUm9iJTIwQ29vdGUlMjAlMjglNDBSb2JfQ29vdGUlMjklMjAlM0NhJTIwaHJlZiUzRCUyMmh0dHBzJTNBJTJGJTJGdHdpdHRlci5jb20lMkZSb2JfQ29vdGUlMkZzdGF0dXMlMkY4MDgzODUxNDIyMjU5NjUwNTYlMjIlM0VEZWNlbWJlciUyMDEyJTJDJTIwMjAxNiUzQyUyRmElM0UlM0MlMkZibG9ja3F1b3RlJTNFJTBBJTNDc2NyaXB0JTIwYXN5bmMlMjBzcmMlM0QlMjIlMkYlMkZwbGF0Zm9ybS50d2l0dGVyLmNvbSUyRndpZGdldHMuanMlMjIlMjBjaGFyc2V0JTNEJTIydXRmLTglMjIlM0UlM0MlMkZzY3JpcHQlM0U=[/vc_raw_html][/vc_column][/vc_row][vc_row row_height_percent="0" override_padding="yes" h_padding="2" top_padding="0" bottom_padding="0" overlay_alpha="50" gutter_size="3" shift_y="0" desktop_visibility="yes" medium_visibility="yes" mobile_visibility="yes"][vc_column][vc_raw_html]JTNDYmxvY2txdW90ZSUyMGNsYXNzJTNEJTIydHdpdHRlci10d2VldCUyMiUyMGRhdGEtbGFuZyUzRCUyMmVuJTIyJTNFJTNDcCUyMGxhbmclM0QlMjJlbiUyMiUyMGRpciUzRCUyMmx0ciUyMiUzRUdsYWQlMjBJJTIwd2FzbiUyNiUyMzM5JTNCdCUyMHRoZSUyMG9ubHklMjBvbmUlMjBleGNpdGVkJTIwd2l0aCUyMHdoYXQlMjBJJTIwc2F3JTIwYXQlMjAlM0NhJTIwaHJlZiUzRCUyMmh0dHBzJTNBJTJGJTJGdHdpdHRlci5jb20lMkZmb3J3YXJkbmV0d29ya3MlMjIlM0UlNDBmb3J3YXJkbmV0d29ya3MlM0MlMkZhJTNFLiUyMEdyZWF0JTIwd3JpdGUlMjB1cCUyMCUzQ2ElMjBocmVmJTNEJTIyaHR0cHMlM0ElMkYlMkZ0d2l0dGVyLmNvbSUyRm92ZXJsYWlkJTIyJTNFJTQwb3ZlcmxhaWQlM0MlMkZhJTNFJTIxJTIwJTNDYSUyMGhyZWYlM0QlMjJodHRwcyUzQSUyRiUyRnR3aXR0ZXIuY29tJTJGaGFzaHRhZyUyRk5GRDEzJTNGc3JjJTNEaGFzaCUyMiUzRSUyM05GRDEzJTNDJTJGYSUzRSUyMCUzQ2ElMjBocmVmJTNEJTIyaHR0cHMlM0ElMkYlMkZ0LmNvJTJGMFVaS2NTUm9EdSUyMiUzRWh0dHBzJTNBJTJGJTJGdC5jbyUyRjBVWktjU1JvRHUlM0MlMkZhJTNFJTNDJTJGcCUzRSUyNm1kYXNoJTNCJTIwUmljaCUyMFN0cm9mZm9saW5vJTIwJTI4JTQwTXJBbnRocm9wb2xvZ3klMjklMjAlM0NhJTIwaHJlZiUzRCUyMmh0dHBzJTNBJTJGJTJGdHdpdHRlci5jb20lMkZNckFudGhyb3BvbG9neSUyRnN0YXR1cyUyRjgwMzMwNzg2MDg5Nzc1NTEzNiUyMiUzRU5vdmVtYmVyJTIwMjglMkMlMjAyMDE2JTNDJTJGYSUzRSUzQyUyRmJsb2NrcXVvdGUlM0UlMEElM0NzY3JpcHQlMjBhc3luYyUyMHNyYyUzRCUyMiUyRiUyRnBsYXRmb3JtLnR3aXR0ZXIuY29tJTJGd2lkZ2V0cy5qcyUyMiUyMGNoYXJzZXQlM0QlMjJ1dGYtOCUyMiUzRSUzQyUyRnNjcmlwdCUzRQ==[/vc_raw_html][/vc_column][/vc_row][vc_row row_height_percent="0" overlay_alpha="50" gutter_size="3" shift_y="0" desktop_visibility="yes" medium_visibility="yes" mobile_visibility="yes"][vc_column][vc_column_text]But the value extends beyond visibility. By using the Forward platform, network operators can directly benefit from this army of tests to ensure that their understanding of the network is always accurate. Forward Search provides an easy-to-understand, end-to-end view of traffic paths throughout the network, Forward Verify ensures that the correct connectivity is implemented at all times, and Forward Predict guarantees that no future change will violate the correctness criteria.

Instantaneous and accurate path analysis with Forward Search pointing to relevant device configuration lines]

[/vc_column_text][vc_column_text]

Free Offer:
Audit your network with the Forward Platform:

Forward Networks is offering every prospective customer free access to our Platform to run a limited audit of the configuration and policy correctness of some subset of their network. In this effort, you would be able to:

  • deploy the Forward Platform for a limited time, either on-premise or in our cloud offering to generate a mathematically accurate model of a subset of your network devices
  • perform searches across the network to better understand device inventory, traffic paths, and end-to-end network behaviors
  • run a series of device audit and configuration correctness checks atop the network model to identify issues and potential problems (from simple network hygiene issues to more critical outage-inducing device misconfigurations or higher-level traffic or security policy violations)
  • isolate the exact lines of device configuration code that need to be fixed in the production boxes to remediate the misconfiguration or errors
  • file tickets to repair these issues in the production equipment

As a one-time audit of a portion of your production network, you would experience the value of the Forward Platform first-hand and free-of-charge.

Request a demo and get a free network audit

[contact-form-7 id="55294" title="Request a Demo"][/vc_column_text][/vc_column][/vc_row][vc_row row_height_percent="0" overlay_alpha="50" gutter_size="3" shift_y="0" desktop_visibility="yes" medium_visibility="yes" mobile_visibility="yes"][vc_column width="1/1"][vc_empty_space][/vc_column][/vc_row][vc_row row_height_percent="0" overlay_alpha="50" gutter_size="3" shift_y="0"][vc_column][vc_column_text]

How to use an army of tests to understand and diagnose your network!

You’ve probably heard about how hard it is to operate networks. The complexity of the devices, the diversity of protocols operating simultaneously, and the sheer size of the network are all well-known reasons for getting something wrong. As a result, network changes require stressful, carefully planned change windows to ensure the desired outcome, with no unintended consequences.

In this post, I want to discuss an additional dimension to the complexity of the problem: cross-vendor, cross-platform and cross-version behavior differences. If not well understood or glanced over by the network operator, these little-known differences may result in security breaches, connectivity loss or network downtime, even when they affect a single device.

A Simple Firewall Configuration, Right?

Imagine you are the network operator for an enterprise network. All the hosts in your network are firewalled and NATed by a Palo Alto Security Platform. All your hosts have a private IP address and their access is restricted by the firewall to match the security policy of the organization. In particular, the security policy of the organization dictates that the host with IP address 10.1.1.10, which hosts sensitive financial data, should have all of its web communications to the outside be encrypted. Here is the configuration you may put on the the Palo Alto box:

NAT configuration and security rules on Palo Alto Security Platform

This configuration simply translates the private IP address of 10.1.1.10 to a public IP address of 2.2.2.10, and denies any web browsing application other than SSL-based when it comes from 10.1.1.10.

After a few months, as part of a hardware refresh, you are asked to replace this firewall with a Cisco ASA. The box you purchased is running ASA OS version 8.2. You do a bit of research online and translate each piece of config in your Palo Alto device to equivalent ASA configuration:

NAT configuration and security rules on Cisco ASA Platform running OS version 8.2

Pretty simple, right? Not so much…

We just introduced a security hole.

The subtle catch here is that the order of applying NAT and firewall rules can change the outcome of the configuration above. If the device applies firewall rules first, then NATs the traffic, the above configurations will work fine. But as it turns out, Palo Alto firewalls perform Firewall and NAT in that order, but on ASA versions 8.2 and lower, firewall rules are applied after the translation and therefore the rules matching on 10.1.1.10 as source IP address will not have any effect. Instead, we should have applied those rules to the post-translated address, 2.2.2.10.

The order of applying NAT and firewall rules matters!!

Six months later, Cisco comes out with ASA version 8.4 and you decide to upgrade. You may think this time everything should be fine. After all, it’s the same vendor! But, not quite… To start, you notice some syntax differences in how you define NATs, which with the help of CLI, you fix. But at this point, can you be sure that the new configuration meets your desired behavior?

NAT configuration and security rules on Cisco ASA Platform, running OS version 8.4

As it turns out, the answer is again, no! Going from version 8.2 to 8.4, the order of applying NAT and Firewall is changed and now it matches the original Palo Alto firewall! Bummer!

As the example above illustrates, different vendors, and even different OS versions or platforms coming from the same vendor may have subtle differences in the implementation of the same concept, protocol or standard.

While syntactical differences can be caught by the device’s CLI or management UI, behavioral differences are very hard to catch.

What makes it even worse is that sometimes these subtleties are not well-documented. They can lead to error-prone configurations, security breaches or network downtime, if not well understood by network operators.

Conclusion

Even the simplest configuration constructs have surprisingly large differences between implementations. We have seen these subtleties mislead network operators in some of our customer networks in the past. For example, some vendors require each VLAN to be explicitly declared before using them on an access or trunk interface (e.g. Cisco IOS and NX-OS), and forgetting to do so will silently drop any traffic arriving on that interface with the undefined VLAN tag.

Similarly, some vendors enforce referential consistency, which means that each object used in ACLs should be defined, while some others don’t have such requirement (e.g. many Cisco routers). In these cases, the ACL rule that references an undefined object will be ignored. As a result, a simple typo in naming of an object may open up a security hole in the network.

In the best case, not fully understanding these nuances can waste hours of your time; in the worst case, it can lead to a security breach that puts your company in the news.[/vc_column_text][/vc_column][/vc_row][vc_row row_height_percent="0" override_padding="yes" h_padding="2" top_padding="0" bottom_padding="2" overlay_alpha="50" gutter_size="3" shift_y="0" desktop_visibility="yes" medium_visibility="yes" mobile_visibility="yes"][vc_column][vc_column_text]

How to close these security holes?

1. Get a free demo version of the Forward Platform
2. Run a network audit and identify issues
4. Isolate the exact lines of device configuration code that need to be fixed

Forward Networks is offering free access to our Platform to run a limited audit of the configuration and policy correctness of some subset of their network.

Request a free demo version

[contact-form-7 id="55294" title="Request a Demo"][/vc_column_text][/vc_column][/vc_row][vc_row row_height_percent="0" override_padding="yes" h_padding="2" top_padding="0" bottom_padding="2" overlay_alpha="50" gutter_size="3" shift_y="0" mobile_visibility="yes"][vc_column][vc_column_text]

Video: How to run an effective audit and close the security holes?

Get access to this video

[contact-form-7 id="55323" title="Resources - Video Series - Audit"][/vc_column_text][/vc_column][/vc_row][vc_row row_height_percent="0" override_padding="yes" h_padding="2" top_padding="0" bottom_padding="2" overlay_alpha="50" gutter_size="3" shift_y="0" desktop_visibility="yes" medium_visibility="yes"][vc_column][vc_column_text]

Don't miss out

Get updates on new articles, webinars and other opportunities:

[contact-form-7 id="55247" title="Blog subscription"][/vc_column_text][/vc_column][/vc_row][vc_row][vc_column width="1/1"][vc_empty_space][/vc_column][/vc_row]

Subscribe to our blog!

RELATED FORWARD CONTENT 
June 23, 2022
What’s it Like to work at a Great Place 

Forward Networks just was named a Great Place to Work in the Bay Area by Fortune. We’re excited to be on this list in an area known for setting the bar when it comes to treating employees well. Perks are nice, but perks alone don’t get 100% of employees to say they work at a […]

Read More
June 20, 2022
Forward Networks named one of the Best Workplaces in the Bay Area™ in 2022 by Great Place to Work® and Fortune Magazine

SANTA CLARA, Calif., June 20, 2022 /PRNewswire/ -- Great Place to Work and Fortune magazine have honored Forward Networks as one of this year's Best Workplaces in the Bay Area. This is Forward Networks' first time applying for and being named to this prestigious list. Earning a spot means that Forward Networks is one of the best companies to work […]

Read More
June 7, 2022
Forward Networks Wins 2022 Fortress Cyber Security Award

SANTA CLARA, Calif., June 7, 2022 /PRNewswire/ -- The Business Intelligence Group today announced that Forward Networks has won the 2022 Fortress Cyber Security Awards in the Network Security category. The industry awards program sought to identify and reward the world's leading companies and products that are working to keep our data and electronic assets safe among a growing […]

Read More

Sign up for our newsletter

clockcrossmenu linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram