• Financial Services
• Several data centers located in geographically separated areas connected by a managed core network
• Mission-critical applications running. across data center infrastructure
• Large virtualization presence on server side
• Limited network virtualization
• Mixed leaf-spine or three-tier architecture based for each data center
• Combination virtual and physical load balancers
• 10+ network equipment vendors
• 20+ models of hardware
• 100+ unique hardware and software combinations
In dynamic, agile network environments, ordinary tasks include updating existing devices and making changes to current configurations. Multiple vendors and diversified networks increase the complexity and reliability of such changes, making standard configurations harder to enforce. Additionally, every change window can create an opportunity for “configuration drift” from a given desired configuration.
At a large financial services firm, the configuration drifts started to cause network outages. After every change, software or hardware, an army of people was employed to manually verify that the changes were not introducing any error or outage-inducing problem. However, the manual testing and verification was never as exhaustive coverage as the one offered by a mathematical model of a given network. Remediation efforts required teams of engineers to perform regular audits as new errors were found. Specifically, simple things like reusing IP addresses (inserted by cut and paste errors) were causing high CPU and increasing troubleshooting time.