NBC's newsroom communication system
By Robert Streeter and Thomas Drewke
NBC’s production intercoms and associated communications systems successfully support the efforts of NBC producers and technical staff for news, sports and entertainment programming. But changing economic times, increased system complexity and competition in the broadcast market demand that communications systems and infrastructure cost less and perform better. Automation and digitization are obvious ways to achieve these goals. Thus, NBC has embarked on an aggressive plan to automate and digitize all mission-critical, infrastructure-related processes. While the NBC production intercom systems use voice-over-IP (VOIP) and other voice-over-data (VOD) transport technologies to lower operating costs, the need to lower design and maintenance costs could only be fulfilled through the creation of a system-specific supervisory system, which eventually will be linked to other databases.
This article details an NBC automation and digitization initiative that provides efficient system management and maintenance for its production intercom systems.
NBC’s control room 3A in New York. This newsroom’s communication system is fully networked with other facilities around the world through a trunked RTS/Telex system controlled by Trunk Master software. Photo by Chris Papas.
The article focuses specifically on the intercom interconnectivity among more than 15 different NBC entities and locations around the world. Each NBC location may contain a system as elaborate as a huge intercom matrix with hundreds of ports, or only a few remote intercom panels. Regardless of the size of the facility, for maximum benefit and efficiency, each communications system needs to be connected to all other systems within the NBC enterprise. This interconnectivity among intercom matrices is referred to as “trunking,” a word borrowed from telephone systems that used “trunk lines” to connect one phone switch to another.
A bit of history
In the old telephone long-distance model, when you wished to make a phone call from Chicago to New York, an operator (or later, a computer and a phone switch) would connect you to a physical copper path leading from a phone switch in Chicago to a phone switch in Pittsburgh to a phone switch in Philadelphia to a phone switch in New York. This was trunking. Major telecom systems have long since replaced this simple trunking method with a better one that digitizes and multiplexes phone conversations and passes them into massive data streams that flow along many possible paths to the required destination. Since there is no “trunk line” directly involved – but rather fiber optic cables carrying digitized conversations, video transport streams and other data flowing continuously – modern telecom systems achieve two enormous advantages over the old copper trunks: efficiency and reliability. The efficiency and the reliability are tied together in the same technology. When you pass conversations as data, you can transparently reroute conversations via hundreds of possible routes, bypassing failed nodes quickly and avoiding single points of failure. When many paths for data exist, conversations between New York and Chicago can travel via many alternative pathways, reducing the need to build new facilities and increasing the chance that a call will go through. A digital network is better than a single path.
The first RTS/Telex smart trunking controller, known as the Trunk Master.
NBC’s systems interconnect the many intercom matrices using a hybrid method combining the old analog telephone trunk method and the newer digital telephone trunking. If a user on the MSNBC intercom matrix in Redmond, WA, wants to communicate with someone in the NEWS intercom matrix at NBC headquarters in New York, some type of trunk controller is needed to set up a communications path from Redmond to New York. This process is similar to the old analog telephone trunk method because a controller creates a point-to-point connection between different intercom system users. But, over long distances, NBC digitizes the audio and transports it as part of a digital stream carrying multiple conversations as well as data and other communications.
Trunks and their master
An intercom-to-intercom communications path, called a trunk, is simply a four-wire (two-pair) audio link between a port on one intercom matrix and a port on another intercom matrix to provide communications between users of the two different matrices. The Trunk Master, a smart controller from RTS/Telex, manages these audio links. In 1994, the controller was the first of its kind to allow interconnections (trunking) among several different intercoms, providing distributed processing of intercom systems. Within NBC’s facility in New York, a single large intercom was replaced in favor of several smaller systems interconnected with copper audio trunks and controlled by this system. It was only later, beginning in about 1996, that the philosophy was extended to allow interconnection of matrices outside of NBC’s main headquarters. Today, the very same controller connects more than 16 intercom systems from Washington state to Washington D.C., as well as providing international connections to London. Temporary connections have supported various news events such as political primaries and conventions, sports events such as the Olympics at various locations – even a research vessel used for the Titanic reclamation expedition.
Although NBC has taken an important step by adopting digital multiplexers and VOIP equipment for the transport of trunk audio, it is important to understand that the controller itself does not know that compressed, multiplexed or any other kind of digital audio paths exist between intercom matrices. Its logic is based on the premise that there is a four-wire audio link between intercoms for each trunk, regardless of the technology used to achieve the audio link. It uses RS-232 or RS-485 asynchronous serial connections to deliver commands to and receive responses from each intercom matrix in the communications network to fulfill long-distance communications requests. If the intercom is distant, industry-standard converters convert this data to TCP/IP. Maintenance and supervision
In maintaining a system with as many as 16 matrices spread out geographically, the system designers encountered many challenges. The first and most obvious was that since the Trunk Master communicates with each intercom matrix in an out-of-band manner as described above, it cannot know if any one given trunk is actually connected and capable of transporting audio. The Trunk Master can only report when an intercom controller is unresponsive. This conveys no information about the continuity of audio trunks or the acceptable fulfillment of a communications request.
The second and less obvious challenge is the simple management of all the information about each intercom system. For example, consider these questions: How many trunks connect New York to Washington? How many trunks are actually working? Which port in the NEWS intercom at NBC in New York is attached to Trunk 105, and to where does this trunk connect? Which trunk is currently being used to connect a director at CNBC to a remote studio in London? Is the smart controller fulfilling each request as it is submitted from remote intercoms, or is it having resource allocation difficulties? And so on. As the overall system progressed in complexity, it became clear that the option of hooking up a tone generator in London and a meter in New York and manually testing each trunk from time to time was a poor one. Not only was there not enough time to do these tests but, even if time were found, it was not always possible to get engineers in each location to drop what they were doing and gather on a conference call.
Meeting the challenges
The solution to these two challenges grew out of discussions between the authors of this article, with contributions from many engineers from NBC, MSNBC and RTS/Telex. It was obvious that there had to be a way to take advantage of today’s automation technology and PC-based solutions to automate the management and supervision of the intercom trunking system. At first glance, one might think that simply adding intelligence to the existing system would solve the problem. But significant barriers precluded that solution. The most significant was the fact that the original Trunk Master was a task-built industrial controller with no capability to run a GUI. To leverage the databases of the Trunk Master and the intercom systems for future features, the engineers decided that the new system would require an external, supervisory PC and testing system.
The first breakthrough came when participants on this project identified a standard off-the-shelf product called the Auto-TIMS IIIR - Dataline Analyzer manufactured by Consultronics. The unit was capable of running automatic tests on the quality of a two- or four-wire voice circuit, data circuits or xDSL local loops. During the automated testing process, it can make a full suite of measurements and compare the results against pass/fail parameters. An RS-232 command port on the unit accommodates the commands sent to run tests, and the reported results. But, despite the capabilities of the Auto-TIMS unit, it still had a serious limitation. It would require several manually supervised units located at each end of trunk connections and/or at each intercom.
To overcome these limitations, the engineers developed a specification to link the testing capabilities of the Auto-TIMS unit and the intelligence of the intercom Trunk Master. A subsequent project undertaken between NBC and RTS/Telex resulted in a Trunk Master supervisory system.
Trunk Master supervisor
As built, the supervisor consists of a Pentium III PC running Microsoft Windows 2000, connected by serial port to the Trunk Master and the Auto-TIMS IIIR, and installed on NBC’s corporate Ethernet LAN. RTS/Telex provided the supervisor software according to NBC specifications. The software consists of modular sections within a main program that accomplish a list of tasks (discussed below). The Trunk Master operating software was also modified to support the functionality required by the new supervisory system.
The Trunk Master Supervisory System’s graphical interface shows the intercoms, their data connectivity status to the Trunk Master, the quantity of trunks between them, and the current state of each trunk.
NBC had several requirements for the new supervisor system. First, it needed to be a Win32 GUI program built to run on Windows 2000 and its successors, such as Windows XP. It also needed to collect all attached intercom configuration data from the controller and store it in a Microsoft Access-compatible database format (using the Microsoft Jet data engine). This information is readily available because the controller itself requires complete configuration data from each intercom to do its job. The supervisor simply piggybacks upon this existing data collection, consisting primarily of tables listing each port and trunk on each intercom, etc.
The software must be capable of testing all communications-network voice trunks using outboard automated test equipment (ATE) together with commands issued to each intercom matrix via the controller. The supervisor software must be capable of critical management functions including automated testing; recording and reporting the results of automated testing; and recording and reporting the activities of the Trunk Master as it fulfills trunking requests.
Other functions include displaying collected data in defined preformatted ways and notifying responsible parties when reported results are outside definable limits. These management functions involve the automated use of a sophisticated database, generating screen and output reports in real time.
In addition to screen and output reports, the supervisor software must be capable of graphically displaying the interconnections between all of NBC’s communications systems and indicating the quantity and quality of connections in use and available.
As a final developmental step, all of the current and historical information contained within the supervisor’s database must be integrated into a larger, Web-based database to allow any NBC employee or others with security clearance to view this information using a Web browser. It also allows multiple simultaneous users to examine near-real-time, dynamically updated screens showing the current status of intercom trunking throughout NBC.
As of August 2002, the first five elements listed above had been completed, and the system had been installed at NBC’s headquarters in New York. The recently introduced PC-based Trunk Master control by RTS/Telex will soon be integrated into the NBC systems and will provide enhanced features and reliability.
Using the supervisor
NBC uses the trunk master supervisory system on a daily basis to ensure the integrity of its trunked RTS production communications systems. It performs daily testing to verify continuity of all of the four-wire audio trunks that connect the 16 intercoms together. The system performs this test automatically at a user-specified time and interval, without any human intervention needed. It tests each trunk for audio continuity, loss and frequency. This is done by putting the trunk in a state known as “maintenance mode,” in which the supervisory system directs the controller to move any traffic on the target trunk to another trunk, and temporarily prevents the controller from using the target trunk for any new traffic. The system lists all trunks with their most recent test results in a trunk-summary table.
After the system tests the trunks, it deems them as either passing or failing based on user-specified parameters for acceptable loss. The system lists failed trunks in a daily alarm report, generates an e-mail to the engineering staff, send an alphanumeric page to the engineer on call, and automatically leaves the trunks in maintenance mode, essentially taking them out of service.
During subsequent daily testing, a previously failed trunk that has changed its status to a passing condition will be automatically returned to service. To complement the daily testing, the supervisory system can track all long-distance communication paths. The information is tracked in a requests table, which shows the near-real-time state of each trunk, identifies the intercom users that are attached at either end, and shows what action (such as talk or listen key presses) caused the request to occur.
The system can also filter the database to isolate traffic between particular intercoms or users. This is an especially useful tool because it allows engineers to troubleshoot complaints about inadequate or failed communications from a central location without having to contact their counterparts at other NBC facilities. In this way, a trunk that has failed since the last daily test can be manually put in maintenance mode, correcting the communications problem instantly. In addition, engineers can check descriptions of symptoms against actual system actions to further help them troubleshoot problems. The system also shows the status of all intercoms with respect to their data connectivity to the controller.
Intercoms that lose connectivity will trigger entries in the alarm report, generate alert e-mails and pages, and cause the system controller to initiate busy signals when trying to establish communications to or from that intercom. One of the most powerful benefits of the system, however, is a graphical interface that pictorially shows the intercoms, their data connectivity status to the controller, the quantity of trunks between them, and the current state of each trunk. This capability allows engineers to view the loading on the system and helps determine the quantity of trunk paths that are needed between intercoms to prevent a “busy” signal from occurring, even during peak periods. Many user-created displays show usage between two intercoms, groupings of several intercoms, or all intercoms and trunk paths in the entire system.
The display shows intercoms as circles, and trunks between intercoms as lines. Intercoms shown as blue circles are communicating with the controller, while intercoms shown as pink circles have lost their connectivity. The varying width of the trunk lines represents the total quantity of trunks between intercoms. The display uses three colors to show the state of each trunk: green indicates empty or available trunks, red indicates trunks currently in use, and yellow indicates recently used trunks temporarily awaiting use for an identical request. Summary boxes above each grouping of trunks show actual numerical values for this summary information. During busy times, these displays clearly show the dynamic performance of the trunking system.
The supervisory system efficiently maintains and supervises NBC’s trunked production intercom systems. Future integration of the Trunk Master supervisory system into NBC’s larger database systems will achieve additional efficiency by Web-enabling the existing solution. Although the scale of NBC’s communications systems is large, the solution achieved is scalable and can be applied to smaller enterprises with as few as two intercom systems.
Robert Streeter is a senior systems design engineer for NBC and is responsible for the worldwide interconnection of NBC’s production intercom systems.
Thomas Drewke is a video and communications systems engineer at MSNBC in Secaucus, NJ.
| Write us |
