Monitoring BOC infrastructure resource health is neither a luxury nor a trivial task. With the integration of broadcast and IT technologies along with the resultant exponential increase in infrastructure complexity, neglecting to implement centralized monitoring methodologies will result in confusion and stress.
Among the challenges of monitoring infrastructure health:
- The larger the facility, the more physically disperse the equipment.
- Support personnel with experience in broadcast and IT technologies are rare.
- Integration of discreet monitoring systems under one GUI.
These issues make it essential that equipment be centrally monitored so that when failures occur, they can be swiftly localized, clear instructions are given to first responders and all relevant information for each subsystem is available for analysis.
Monitoring technology implementations
Systems are available that individually monitor the physical, media network, application or security layers of the BOC infrastructure. In a centralized monitoring system, each layer will report its condition in a window on a single display, the central station monitor (CSM). The accompanying illustration depicts a CSM display.
A central station monitor with physical, media network, application and security monitoring status displayed in individual windows. Click here to view an enlarged version of this diagram.
Physical layer monitoring of video and audio signals will include routing and distribution equipment. Digital and analog signals are monitored for presence, level and conformance to industry specifications. Closed captioning regulatory compliance can be verified.
Media networks are monitored for faults, configuration and performance. This includes network congestion, traffic patterns and device health. Storage is monitored for capacity, bandwidth and network interface activity.
The application layer resource monitor can ensure that programs are running, network communication is alive, the OS is stable and hardware is healthy. Additionally, platform configuration data reporting will keep track of OS patches and application updates.
Security must be constantly monitored. It is important to know if you are under attack or infected. If the infrastructure has been breeched, verify that corrective actions have been taken, and the threat has been isolated. And most important, an audit trail must be recorded: where did the incident originate and who is the attacker?
Design and deployment
Centralized monitoring is an integral part of the BOC design. Simultaneous deployment of monitoring tools will aid in bringing up the infrastructure and in having confidence that it performs to engineering design specifications and workflow process requirements.
Deploy a core pilot system that monitors essential infrastructure resources, such as power and temperature. Establish whether systems integrators, vendors or in house staff will configure equipment for monitoring. Assigning IP addresses to a few distribution amplifier frames is not an issue, but for 50 or 100 frames will be very time consuming.
Plan to monitor normal and stressed operation of all layers of the infrastructure before going into production. Be sure the monitoring system catches meaningful faults. You want to know not only what will fail, but what the impact of a failure is and plan for a response and contingency work around.
Since multiple departments are involved, it is important that implementation scope issues don't fall through the cracks and undermine deployment and day-to-day monitoring system operation. Involve all stakeholders to determine where and what the key points are to monitor. Experienced operational personnel are best qualified to determine what should constitute an alarm condition. Support personnel can define resolution protocols for their areas of first responder responsibility.
Broadcast or IT
When designing a CSM system, the first question to answer is whether a broadcast or IT application should be used as the core. Dedicated monitoring applications are best suited to monitor either broadcast or IT resources.
Harris’ Broadcast Manager is a broadcast monitoring system that can be extended to IT resources. IT application suites, such as Spectrum Aprisma and HP OpenView, can be used for SNMP-enabled facility monitoring.
Because many broadcasters now include SNMP functionality in their equipment, SNMP can be used to communicate with the monitoring application. However, many broadcast resources use GPI, RS-422 and other communication interfaces for fault reporting. A monitoring system must be able to process any of these signals.
In either case, there will be a lot of adaptation and customizing to create a comprehensive system. Lack of standardized MIBs in broadcast equipment is an issue with SNMP-based implementations. SMPTE is addressing the need for a common MIB structure, and a draft standard PRO-AV MIB-1 is now in committee (available as a Registered Disclosure Document.)
Facility modeling via a block diagram can help in navigating the infrastructure and isolatinge faults. Yet developing the diagrams from scratch is a time consuming and error prone process. Import of AutoCad schematics and incorporation of network discovery topologies would automate the process.
Problem resolution protocol
Alarm or warning conditions must be analyzed and divided into air-critical, air threatening and non-air response levels. The most appropriate first responder will be dispatched. If the problem must be elevated, the appropriate departments will be contacted and more experienced personnel dispatched.
Simultaneously, a trouble ticket will be generated. Incident occurrence and resolution actions will be amended to the asset history. In this way, a centralized asset history database will be maintained. E-mail, cell phone or other alerts can be sent to support personnel.
Finding the root cause of cascaded alarm conditions can be difficult. The prioritizing of cascaded alarm data needs to be established during facility commissioning and in the initial few days and refined in the following weeks of operation.
A decision needs to be made regarding automated failover capabilities. Will you depend on the monitoring system to take corrective action without operator intervention or should an operator be notified and give the OK?
Pop up windows with problem logging, resolution and escalation procedures can be activated by alarm conditions. Caution and warning messages can also insure that the wrong, possibly catastrophic, corrective actions are not taken. Incident responses should be regularly reviewed and refined.
With the establishment of a BOC support knowledge base, nothing will be left to chance. Commercially available help desk, trouble ticket and asset management application suites will facilitate development of a support database.
One for all and all for one
A recent Broadcast Engineering Webcast discussed the implementation of a centralized monitoring and control system of systems. ILC, a monitoring application provider and Nebraska Educational Telecommunications (NET), who installed an ILC system, described a system from the broadcast perspective. The Webcast is available at http://event.on24.com/eventRegistration/eventRegistrationServlet?eventid=15098&sessionid=1&key=4E1B140291E1273CA2BC19E022720D3C&referrer=http%3A%2F%2Fbroadcastengineering.com%2F
CSM is a mandatory requirement in maintaining the operational integrity of critical broadcast systems. Among the benefits that will be realized:
- Simplifying problem isolation and resolution
- Reduction of response and repair time
- Proactive measures to prevent critical failures
- Minimize down time
- Status information is available anywhere on the BOC LAN
An off-the-shelf, turnkey perfectly integrated centralized monitoring system might be impossible to find. But it is possible however to build a comprehensive monitoring system for each of the four BOC layers, using existing applications.
Additional reading
Physical Layer: Multi Video Processors
- Miranda: Kaleido-K2: Advanced modular multi-image display processor
www.miranda.com/pdf/custom/Kaleido-K2_applications.pdfDatasheet
www.miranda.com/pdf/datasheet/Kaleido-K2.en.pdf - Evertz
MVP Multi-Signal Monitoring Solution
www.evertz.com/mvp.php
Media Network
- Harris Broadcast Manager
www.broadcast.harris.com/recon/network.asp - ILC Max View
www.ilc.com - HP OpenView
www.managementsoftware.hp.com/solutions/index.html - Spectrum Aprisma
www.aprisma.com/products - SNMP Tutorial
Simple Network Management Protocol, Cisco
www.cisco.com/univercd/cc/td/doc/cisintwk/ito_doc/snmp.htm
Application Layer
- StatePointPlus System Integrity Controller
www.statepointplus.com/downloads - Microsoft
www.microsoft.com/smserver/default.mspx
Security
Knowledge Base, Trouble Tickets & Asset Management
- Remedy Help Desk – Data Sheet www.remedy.com/solutions/documents/datasheets/BMC_Remedy_HelpDesk.pdf
Correction
Thomas E. Fahy of the Broadcast Strategy Group informed Broadcast Engineering that the Partnership for Public Warning is no longer operational. Fahy was the project manager of the 2005 NASBA Summit on EAS and will serve in the same capacity next year. NASBA is the National Alliance of State Broadcast Associations, whose members continue the effort for modernization of the EAS.
