IP networks have become a mission-critical part of just about every broadcaster's facility. Broadcasters need to monitor these networks with the same critical eye they use to monitor the rest of their video facility. This month, I will provide a high-level introduction into the topic of IP network monitoring.
Why monitor
Before jumping in, it might be good to look at some reasons a broadcaster might choose to monitor its IP networks. Those reasons include: establishing a baseline performance; detecting, isolating and addressing problems; identifying issues before failures occur; identifying failures that have already occurred; and, checking new IT components through acceptance testing and system commissioning. Let's look at each of these in turn:
- Establish baseline performance: One reason to monitor your IP networks is to establish baseline performance. Establishing a baseline lets you look back later to see how things behaved when the network was operating normally.
- Detect, isolate and address problems: When a network fails, you will be under pressure to get the network up and running as fast as possible. If you have already established a network monitoring strategy, you will be in a great position to quickly and efficiently identify the problem source and get things back up and running quickly.
- Identify issues before failures occur: You might think IP networks are an all-or-nothing infrastructure. Either the network works properly, or it fails dramatically. This is because users typically experience the network in this way. But, if you are monitoring the correct parameters, you can identify some problems before the network goes down. In my opinion, it is much more valuable to your company to be proactive rather than reactive and only fix problems after they have occurred.
- Identify that failures have already occurred: Interestingly, depending upon your network topology, it is possible to have a network failure and not know it. In fact, IP networking technology was developed by the U.S. Department of Defense to deal with the “smoking hole” scenario. It wanted a network that would keep operating, even in the face of a nuclear attack. Since IP traffic can re-route automatically in the case of a failure, how will you know that a component has failed in the core network? This can happen only by being aggressive about monitoring system alarms and looking at system logs.
- Acceptance testing and system commissioning — When we purchase a new piece of video gear, we usually check it out before we put it on-air. We may monitor the new equipment's behavior for awhile to ensure it is operating as advertised. Why would we not do the same thing with new IT infrastructure components?
How to monitor
Once you have decided that monitoring your IP network is important, and you have determined why monitoring is critical, the next obvious step is determining how to monitor your network. The easiest way to get started is to use monitoring capabilities you already have in your facility. Here are a few things to consider: router statistics, switch statistics, server statistics and workstation performance graphs. Again, let's take a look at each of these individually:
- Router statistics: Just about any router will provide you with overall bandwidth usage statistics. Many will also give you information on specific traffic types. Routers that connect your facility to the Internet will give you valuable information about the amount of traffic crossing your corporate boundary. Look for unusual activity or high-usage statistics. Some routing systems will allow you to view statistics over time. Do you see spikes of activity at unusual times of the day or heavy usage on weekends that cannot be readily explained? Since routers direct traffic between networks, do you see an unusual pattern of traffic from one network to the other that cannot be explained?
- Switch statistics: Routers provide important information at network boundaries, but they may not give you the information you need when it comes down to a specific device that may be causing trouble. Consumer-grade switches typically do not provide a lot of information. However, commercial-grade switches can provide excellent information that exists at the port level. They can provide detailed information on just how much and what sort of traffic is going to a particular computer. This can be very helpful when you are trying to isolate a particular problem. Note that you should look at managed switches, and you should check to see what sort of statistics are available; features vary depending upon manufacturer and model. (A note about switch statistics — in many cases, statistics are only available in real time.)
- Server statistics: Almost all servers can provide detailed logging information. In their raw form, these logs can be useful for pinpointing specific problems. Many log analysis programs are available, some of which are free, especially for *NIX servers.
- Workstation performance graphs: Servers and workstations alike typically provide network monitoring graphs in real time. On Windows and *NIX servers, the graphs can be quite detailed, providing information on incoming and outgoing traffic, filtered by type. These monitors can also show graphical representations of processor usage, which, when considered with network usage, may help isolate a specific process or program that is causing a problem and needs to be addressed.
Network probes
Another class of monitoring device is the network probe. While many commercial tools are available, I suggest you start with an excellent free resource — Wireshark. Wireshark is a free network sniffer or probe, which can record a series of packets and display them at a later time. It has an excellent manual, and there are many tutorials and other online resources available also.
As your familiarity with IP network monitoring grows, however, you may decide that you need to employ a full, commercial solution. There may be several reasons for this. For example, you may decide you need professional support. Or, you may require a feature or performance that is not available in free monitoring solutions. You may also decide that you need some professional video functionality that is only available in commercial solutions.
If you are moving video around in MPEG-2 on IT networks, then this may be a case where a professional monitoring tool would be useful. As you probably know, MPEG-2 can be encapsulated in UDP packets and sent over IP networks. Troubleshooting errors on IP networks that appear as glitches in MPEG-2 streams can be notoriously difficult to isolate. This is because errors can be introduced at many different levels. Is the problem present in the source video? Was the MPEG-2 stream properly configured? Is there a problem with the MPEG-2 encoder? Is there an issue with the IP transport? There are a number of places where errors could be introduced.
One tool is the FE Stream Analyzer from FE Engineering. This analyzer is specifically designed to inspect IPTV packets. The main screen gives a quick summary of critical, video-oriented measurements such as PCR jitter and video packet loss, while also displaying a number of parameters related to the compression system as well. (See Figure 1.) It will look into a multiplexed MPEG Transport Stream and allow you to pick out a particular service for analysis.
Broadcasters should be aware of another category of commercial solutions — network emulators. These devices allow you to create a clean-room network, and then degrade the network in a carefully controlled manner to determine how equipment performs under these degraded conditions.
Network emulation products allow you to select different types of network impairments and then adjust the severity of the impairments. This can be extremely useful when troubleshooting professional video equipment. Because MPEG-2 compression assigns different importance to different packets in the stream, the effect of a hit on an “I” frame will be much more pronounced than on a “B” or “P” frame. Thus, end-users may see differing effects of a particular network error depending upon where the error occurs. A network emulator will allow you to carefully control the behavior of the network, something you cannot do in the real world.
I encourage you to experiment with some of the free tools you already have in order to get familiar with the concept of network monitoring. I also encourage you to take advantage of the free trials offered by many network monitoring vendors. This can be a good way to become familiar with some of the advanced features and monitoring options that are available.
Finally, I cannot stress the importance of education enough. It is critical that professional video engineers maintain a level of expertise that allows them to understand and act upon the information made available through IP monitoring.
Brad Gilmer is president of Gilmer & Associates, executive director of the Video Services Forum and executive director of the Advanced Media Workflow Association.
Send questions and comments to: brad.gilmer@penton.com
