The transmissions operation center is divided into two portions: incoming and outgoing, the latter of which is pictured here. This area serves as a final QC point as signals from all 19 networks are sent out of the building to the satellite.
How do you design and construct a major broadcast television facility when you have the unique opportunity of starting from scratch? Turner Entertainment Group, a division of Turner Broadcasting System, built a 193,000-square-foot broadcast facility at its Atlanta headquarters. The original facility had been filled to capacity with 19 television networks distributed to cable television operators throughout North and South America.
Just over two years ago, Turner Studios moved out of the old building and into a new structure on the 33-acre campus. It was during this period that plans developed to move Network Operations, the broadcast division, into an advanced facility of its own.
Ground broke on the project on March 13, 2000, clearing several parking areas to make way for Network Operations’ facility and an office building. Turner Construction and KPS Architects provided the bulk of the building work as plans were being drawn for the massive layout of the new facility. Perkins & Will interior designers and systems integrator AZCAR assisted in the preliminary design. MCSi, a systems integrator out of Atlanta that also worked on the new Turner Studios facility, was brought on later to help develop the final design, produce the drawing package, and assist with the wiring and installation. MCSi is still on-site, handling all equipment logistics for the final stages of construction.
The supervisor’s area of the TBS pod offers a view of the TBS Superstation and WTBS local analog/digital broadcast operations control rooms.
The new, all-digital network operations center offers room for network expansion and connectivity with its high-definition infrastructure. All facility wiring and routing is HD-capable and can easily accommodate HD equipment as it is added. Furthermore, the building is designed to employ file distribution as opposed to strictly audio/video distribution. This allows air material (programs, commercials, promos) to be delivered as files via fiber or satellite and enter the building as data instead of the traditional audio and video. Error-checking takes care of automatic resends should part of a file be corrupt, and programs are subsequently stored on a central server for easy access by network-distributed servers when needed.
As of September 2003, this facility will be home to the 19 Turner television network feeds. The first networks to move were TNT East and TNT West in December 2002, followed in March by TBS Superstation, local channels WTBS-17 analog, WTBS-20 digital and regional network Turner South. Of the 13 remaining networks, Cartoon Network East and West, Boomerang and Turner Classic Movies will move this May. The final nine networks, all Latin American, launch in September. With their associated equipment, the current 19 feeds will occupy 10 active control rooms. There is space in the new building to eventually accommodate up to 30 control rooms and 87 feeds.
The current 19 feeds originating from the network operations center are conveniently centralized on the first floor of the six-story building. The lower three, with computer flooring, were designed for technical equipment to support the broadcast operations.
The test control room replicates the layout and design of a functioning BOC. This room provides a designated area to test and evaluate hardware and software in a simulated broadcast and media operations environment.
The 10 active control rooms, or broadcast operations centers (BOCs), are comfortably isolated from one another and are divided among four separate “pods” with four control rooms each. In the event of a major equipment or software problem in one of the pods, the others will remain unaffected due to this isolation. Each pod features a spacious, glass-walled, central supervisor’s area for overseeing operations. Supervisors can also make changes to the individual automated playlists. The pods contain mostly computer-based equipment associated with Chyron Pro-Bel automation and its MAPP playout systems for machine control.
Though each pod features four control rooms, the number of active control rooms and quantity of feeds from each control room varies. For example, the international BOC will initially handle nine feeds between three operational control rooms, while an additional control room remains available for future channels. Meanwhile, the TBS BOC has three active control rooms, and a fourth is already earmarked for use later this year. Of these three active control rooms, TBS Superstation and Turner South each have their own control room with small producer stations for live programming, while feeds for the local WTBS analog and digital channels share the third control room.
Each BOC has full redundancy in case of failure, with an A chain and a B chain of equipment for every feed. In the event the A chain fails, a push of a button switches to the B chain and its duplicate complement of equipment. Likewise, if maintenance is needed on a piece of equipment within the A chain, the B chain is switched on without being noticed by viewers. A switching system was designed in-house by Turner engineers to accomplish these tasks.
Each network chain (A or B) contains a separate automation system synchronized with the other chain (38 systems in all, with a total of 168 associated computers for playout to server, ingestion and other purposes). Both systems independently run a playlist and are separately connected to their own Pinnacle MediaStream 900 server on separate SANs and Thomson Grass Valley switcher.
The media operations control center is the broadcast equivalent of an air traffic control center for the coordination of playlist changes and ingest operations.
The A chain features M2100 switchers; the B chain, for the sake of control room space, uses an M2100 with a smaller control panel, developed in partnership with the Turner operations and engineering departments. A Chyron Aprisa and Duet graphic system is part of each redundant chain in the domestic network control rooms, but single systems are shared between chains in the international rooms.
Two 50-inch Barco projection displays round out each control room to monitor the various inputs and what is happening on air. These are rear-projection monitors that use the Barco Hydra for displaying multiple pictures on each screen. The redundancy factor figures into this setup as well: If the left display burns out, one button shifts all monitoring capabilities to the right display, and vice versa.
Flexibility to alter the size of separate video windows serves as one advantage, while the ability to view a feed in HD means simply changing a card within the system. Information from the automation system (with A chain information coded red and B chain information coded blue) is also readable on the screens.
Some equipment is shared between the A and B chains. A 360 Systems DigiCart serves both chains for voiceovers and other audio work, and a Fibre Channel managing server is shared to move material between the redundant Pinnacle servers, if necessary. There is more sharing in the international BOC. Here, the Chyron Aprisas are shared and the B chain uses a simple 10x1 switcher. International live programming, while rare, is assigned to the A chain.
To perform voiceovers, or otherwise manipulate and mix audio in the control room, Snell & Wilcox IQ Modular equipment for audio de-embedding is installed. Throughout the facility, wherever possible, sound is embedded with the picture to avoid “lip-flap” that comes from processing signals. This is particularly useful during live events where satellite transmission, video effects, encoding, decoding and synchronizing all play a role in delaying video.
A dedicated test control room that features all the equipment and redundancy of an operational control room, was built to evaluate and test equipment before it is put online. Most often this involves software updates. With a fully operational test room, the staff avoids the risk of using new versions of software on the air and having them fail. They can also test equipment from other manufacturers to make certain it is compatible with the gear that is on-air.
A separate training control facility features fully operational domestic and international control rooms. Here, system faults can be simulated to train the staff on how to react to problems or troubles that they may encounter in the on-air control rooms. This is much like a flight simulator in the airline industry. Sessions can also be videotaped for future training. With the test and training rooms separated, conflicts between the two situations are avoided. Both can serve as emergency backup control rooms in a disaster.
The BOC pods take up a large portion of the first floor, but another area vital to the operation is also located here, notably the transmissions operation center (TOC). Viewable from the lobby of the Network Operations building, a massive area of curved glass surrounds the TOC, which is divided into two sections: incoming and outgoing.
The incoming section is generally dedicated to the reception of live feeds. As Turner Entertainment broadcasts numerous live events, especially sports, this area features three quality control stations for live incoming signals. Operators will then coordinate the satellite uplink feeds originating at the venue sites, control the downlinks at the Turner Teleport dish farm, and feed the incoming signals to the correct control room with a Thomson Grass Valley Trinix routing system.
Maintenance engineer Chuck Armitage adds a module to one of the numerous Snell & Wilcox IQ Modular systems throughout the facility.
Incoming TOC embeds the audio using Snell & Wilcox audio embedding modules into the live video as it is sent to its control room destination. There, it is de-embedded and mixed with additional audio before being re-embedded and sent on to the teleport.
The outgoing side features large Barco screens along the entire front wall with numerous windows assigned different feeds. This area provides operators with a final look at a signal before it is sent out of the building. A signal is sent to the satellite 23,000 miles up, and a return signal comes back to outgoing TOC. Scientific-Atlanta IRDs and Leitch frame synchronizers are used to receive and condition the satellite feeds.
A Snell & Wilcox RollCall network monitoring and control system is instrumental to this room, with the main terminal yet to be installed. This will allow virtually every piece of equipment throughout the facility to be monitored. Using this system, a signal loss due to a problem in any connected piece of gear can be easily pinpointed. In addition to the network monitoring system, this is also the location of the central alarm system that monitors the health (fire, power, HVAC, water) of the entire facility. The room is protected by an FM200 gaseous fire suppression system.
The remaining technical areas of the first floor include a BOC maintenance shop and two squeeze credit rooms. While most of the production and editing work is done at the adjacent Turner Studios facility, the addition of the squeeze credit rooms, supported by two Pinnacle Liquid blue systems, is vital for altering credits close to airtime. Promos often change prior to air, so this flexibility to make immediate revisions is needed in the network operations center.
More space is available for additional control rooms on the first floor (as well as the third floor), and audio delay suites may be added as well in the future, depending upon programming. Computer flooring throughout the first three floors allows for easy expansion. Additional feeds can be added to all existing control rooms when needed.
The second floor features a variety of rooms vital to 24-hour operation. A high-density storage library close to the size of a football field holds 20,000 commercials, a large number of promos and seven days worth of programming. The floor in this section was reinforced to carry the weight of the shelving.
Near the library is the short-form ingest area, comprised of 10 small rooms featuring Pro-Bel workstations, Sony videotape machines and Ikegami monitors. This area is part of the media operations center (MOC). Operators in these rooms type program information into a database and then ingest commercials and promos into the servers. These servers are controllable from the ingest stations to allow the operator to run a complete quality control check of ingested materials.
A central tape area is available for dubbing from one format to another and serves as “guard source VTRs,” providing a third level of redundancy for the BOCs. These machines are used to back up high-profile programs and are automation-controlled. As the A and B chain roll a program, the automation simultaneously rolls the VTR. While unlikely that both control room chains will fail simultaneously, a guard source VTR brings extra peace of mind for heavily promoted programs.
The incoming TOC is generally dedicated to the reception of live feeds. Operators here coordinate satellite uplinks and feed signals to the appropriate BOC with a Thomson Grass Valley Trinix router.
Syndicated programs that come into the facility via satellite are also videotaped in this area of MOC. Snell & Wilcox IQ Modular equipment conditions and decodes the signals in this area as they come into the building. Soon, these programs will all come in as files using the newly installed Pathfire system.
This section also features Snell & Wilcox audio shuffling modules. This is a unique product developed in partnership with the Network Operations’ engineering team to reconfigure various audio tracks associated with a program, depending on how it will be used. The majority of their programming reaches them in stereo English, yet they feed Latin America. This means they have to feed in Spanish, Portuguese and French.
Their Venus routers handle six channels of audio, and Pinnacle servers are programmed to store these six channels in their house standard configuration: Channel 1 is stereo English left; Channel 2 is stereo English right; Channel 3 is Spanish; Channel 4, Portuguese; Channel 5 is DVS to accommodate DVS-enhanced programming; and Channel 6 is French. When a program is ingested, audio de-embedders separate the audio and video, and audio shufflers rearrange the audio tracks as they are ingested into the server. They are then stored in memory as six designated audio paths. When needed for air, a network recalls the file and shuffles the audio back into a new configuration according to where it will be sent. For example, with the Brazilian feed, Portuguese language is the main audio associated with the video, possibly backed up by English.
Another portion of MOC handles automated long-form ingestion. This area features 18 Sony Flexicarts controlled by automation cache engines. Programming is loaded on an automated basis via a robotic arm that pulls two tapes from the bins and loads them into the VTRs. The programs are then ingested into server memory according to the cache engine schedule.
The MOC also holds the long-form-ingest area for the quality control check and ingestion of high profile programs, such as premiere movies or special made-for-television programs. These rooms operate much like the short-form rooms and feature similar equipment, but are used for programs that an operator wants to review closely. A prominent feature of the MOC is the media operations control center (MOCC). This unique glassed-in room is the broadcast equivalent of an air traffic control center, from which changes and ingest operations are coordinated.
A large number of computers reside in the MOCC and are part of the same automation systems in the BOC control rooms. Operators there are able to monitor everything and connect to sources in every control room. If certain commercials need to be eliminated immediately (like after a plane crash), an MOCC operator can alter the playlists from there. This room, along with the BOC supervisor areas, allows control room operators to concentrate on what is going out to air instead of making changes to the playlist. As the MOCC operators make changes to the individual network playlists, they can direct new commercials to be ingested in the short-form rooms. These changes show up on the supervisor’s automation as well as in the control room.
With 18 Flexicarts, up to 36 programs can be simultaneously streamed into server memory. Teranex digital processors, using one microprocessor per pixel of picture, clean up the material in real-time, reducing noise, fixing impairments, and making the video easier to be compressed for storage in the server. A confidence feed is automatically played back out of the server and onto monitor screens less than a second after being recorded. Automated alarms appear on the screens if anything fails, and operators can print out a report documenting the final results.
The TNT portion of the central equipment room is a designated area that allows for immediate action to any equipment problems.
The MOC and library occupy just one-half of the second floor. The other half is the central equipment room (CER). Kept at a cool 65 degrees, the CER houses 400 APW racks (with room for 100 more) and spans the length of a football field.
To address HVAC issues, an air conditioning slot was added to the ceiling along the front of each rack. This helps to bathe the front of the equipment with a curtain of cold air that falls from the ceiling. The equipment naturally sucks the cool air in from the front and the hot air is exhausted out the back or side to the interior of the rack and then up into a plenum in the ceiling, avoiding the addition of plenum cabling underneath the flooring. All racks feature rear access for maintenance through doors in the back.
The CER is laid out in network groups (TNT, TBS, etc.) to help maintenance quickly pinpoint and correct a problem. This room features two massive Cisco IP data routers connected to every source in the building for file distribution. These routers are redundant as well; if one goes down, the other instantly takes over. Seven Venus routers and nine Trinix routers handle audio and video distribution. Tektronix test equipment is also located throughout CER.
The majority of noisy or high power consumption equipment is housed in CER. Two separate 1500 KVA electrical generators back up power for the entire building. There are associated separate UPS systems for power redundancy. In addition, every rack contains two electrical strips. Each piece of equipment, where possible, is plugged into two separate power panels supplied by two different UPS systems. If one system were to be lost, the other will keep the equipment in operation.
Snell & Wilcox IQ Modular equipment occupies a large amount of rack space in CER. This gear ties everything in the facility together and distributes, conditions, manipulates and processes the signals at all stages. A massive facilitywide installation of more than 5000 modules in 3RU enclosures, these modules handle an enormously wide variety of applications.
Included in these applications are analog and digital audio and video distribution, analog-to-digital and digital-to-analog conversion, video decoding and encoding, frame synchronization, audio conversion, subframe remapping, digital “proc amp” control, gamut legalization, and standards conversion for international program exchange. Each enclosure features variable fan speed and input and output temperature monitoring while also allowing additional slots for expansion. The majority currently feature 10 or fewer populated slots, while up to 16 slots are available in each 3RU enclosure. This provides Network Operations with the means for extremely simple expansion of these functions in the future.
The network management system offers additional benefits. When a RollCall-enabled piece of equipment fails, the loss of signal will be noted by that piece of equipment on the management software. While most of the system’s electronics live in the CER, stations are set up at several locations within the facility. The main terminal is installed in outgoing TOC. This system is critical to keeping the networks on-air. Reducing the time needed to pinpoint a problem is crucial in a facility of this size. The system enables the 500 racks in the facility to react in a timely manner so the effects of a failure are minimal.
The CER also holds the central cache area, which is dedicated to storage. Two redundant EMC data storage systems feature 11TB of live server memory each. All 20,000 commercials and promos spend their lives in EMC, along with three days worth of programming. Avalon Management software controls the movement of spots onto and off of the memory and Pro-Bel automation manages ingestion. One might think this degree of redundancy backing up the main system is enough. However, in the event of a catastrophic failure, two redundant ASACA DVD jukeboxes, each holding the same 11TB of material as EMC, are available.
By June 2003, the network operations center will have Broadcast Inventory Management (BIM) online. BIM is the central cache holding everything featured in EMC and DVD jukebox memory. It delivers air material to a network playout server as a file when called upon by an automation system. BIM will allow the staff to truly ingest once and play out as often as necessary, across all networks. Once BIM is operational, a commercial can be ingested into the system from any short-form ingest station in MOC. Any network that then calls for that commercial will have it delivered to its network server at four times real time. In the event that a piece of equipment in the inventory management system fails, they can cache directly into the playout servers as they currently do.
The new network operations center is running efficiently with all of its advanced automation and redundancy systems. The equipment choices throughout the facility complement each other. With the new facility, Turner’s Network Operations is ready for just about anything that comes its way.
Ron Tarasoff is vice president of broadcast technology and engineering for Turner Entertainment Group.
