As any company that specializes in live entertainment knows, you're only as good as your last success. In fact, most entertainment venues are well aware that the difference between good and bad press attention often comes down to something as simple as whether a single bulb lights up at the right second.
Proper maintenance for such venues is so vital, in fact, that the International Association of Amusement Parks and Attractions (IAAPA) says ride and attraction maintenance is the second-highest expense they face – with only employee payroll costing more. The average cost of an attraction “going down” is $100K per hour, not including the impacts of negative press.
However, as one such entertainment venue recently discovered, it's not only vital to ensure that the proverbial light bulb is turned on at the right second, it's just as important to ensure that backup systems work properly should a failsafe become necessary.
One network engineer who works in infrastructure operations and service delivery with such a venue recently encountered this situation when his team was tasked with verifying the network design. Traditionally, his team has had to assign several engineers to manually verify the integrity of designs for rides and attractions – a process of logging into network devices and visually reviewing configurations. Not surprisingly, that process can eat up hours of effort, and it's not unusual for problems to be overlooked.
By utilizing Forward Networks' NQE checks for design verification, however, his team reduced the process to less than five minutes, creating massive savings in terms of labor and time. But most importantly, NQE checks enabled the team to validate circuit redundancy and identify missing links. Had the primary circuits or links failed during a live show, the redundant circuits or links would not have kicked in to keep the show running smoothly – or at all.
"It was a problem we didn't even know we had, and we weren't even looking for it," he said. "If we hadn't run these checks on NQE, any failover scenario we might have had would have been unsuccessful, causing massive segments of the show to go down. We avoided an outage that would have been impossible to calculate by preventing a spectacular customer service fail."
That experience led him to expand the use of NQE with Forward Enterprise during the company's biannual Power Redundancy Drills. During these events, the venue effectively kills the commercial power feeds to different areas of the park to ensure that UPS, battery strings, and backup diesel generators function properly. Prior to doing so, the IT team is tasked with verifying the operational state of all network devices that will be affected, including verifying device states while they're on backup power and when commercial power is restored.
Traditionally, these processes have been completely manual, requiring considerable time and personnel resources. When a device fails to come back online or enters a "hung" state, the team is forced to deploy “boots on the ground” to conduct a building-by-building, closet-by-closet hunt to visually inspect the physical devices.
"Leveraging Forward Enterprise and NQE enables us to quickly identify any network device that doesn't come back in the proper operational state within minutes," he said. "In the event that we have to direct our technicians to physically inspect a network device, we can do so with far greater precision and minimize – if not entirely eliminate – the need to hunt for a failed element. This is a game-changer for our team."
To learn more about how you can use Forward Networks to simplify network operations, schedule a demo today. Be sure to read our other blogs in this series about how Forward Networks is impacting enterprise networks around the world, including UNDER PRESSURE: Enterprise IT Teams Use NQE To Reduce Time-Intensive Processes.