With a 5-year technology plan as their guide, CNN News Group staffers move the improvement process along one project at a time. The visionary behind the plan is Scott Teissler, CIO and CTO, who oversees both R&D and engineering. R&D is responsible for project management, while engineering is responsible for execution of the projects. Operations personnel in the various networks help define how projects are going to be implemented.
One such project that touches at the heart of the News Group involves video playback. In order to maintain their competitive position in broadcasting, CNN News Group is committed to making MPEG-2 the basis of their operation and to move completely away from videotape. Central to this goal is the implementation of server-based video playback throughout the organization.
The networks of CNN News Group are on the air around the world 24 hours a day, seven days a week. Yet the Atlanta, Georgia (USA) headquarters is in an almost constant state of facility redesign. Some compare it to changing the wheels on a moving locomotive. Very few of us will have the opportunity to experience firsthand the challenges of such a large and complex environment, but we can, courtesy of CNN, take a snapshot of this constantly morphing broadcast giant and use the lessons they have learned in our projects.
The server-based playback project started in mid-1998 with the realization that a lot of the complexities of getting a newscast on the air centered on playback operations and the need to shuffle videotapes back and forth, both within and between the various networks. A server-based playback world had the allure of enhancing the concept of sharing content across networks. But it also had the hard reality of changing the basic workflow so familiar to the networks’ employees.
Historically, CNN has developed networks as virtually stand-alone entities. The server-based playback project would create a more interdependent environment. The scope of this project was to eventually provide server-based playback access to CNN, CNN International, The Airport Channel, Headline News and CNN News Source.
The project manager is quick to point out that a significant portion of his team’s efforts involved working with employees whose day-to-day tasks would be affected by the changes. He credits their input and cooperation as being significant to the success of this project. In addition, while this and other projects flow from the Technology Plan, they are developed to help meet short and long-term business objectives — not just to introduce interesting technology to the company.
Once the decision was made to proceed with server-based playback, Chandler became overall project manager. He was joined on the project team by Tom Kingsley, director of engineering, CNN, who served as engineering project manager, and Bob Hesskamp, vice president of operations, CNN International, who had the role of operations project manager. These three acted as interface to a host of engineers, managers and operations people to accomplish the project. Because of the scope of this project and the fact that it would be implemented at the heart of the organization, the company acted as its own systems integrator.
The CNN project required 30-dual channel Leitch VR300 video servers. The servers were configured in two arrays containing 15 servers, each providing more than 60 hours of storage on each server bank.
The server-based playback project started by having potential vendors present their ideas for solutions. The primary criterion used to winnow the field down was reliability of hardware and software. In addition, the project team looked for buy-in from the vendors on the concept of shared storage with high port count need to service the many networks. The team wanted to use off-the-shelf equipment with only limited modifications. They were looking for the long-term flexibility of a mainstream product rather than a complete custom design.
Another factor considered during vendor selection was video quality because video would be compressed in all cases. Initially, JPEG compression would be used while MPEG-2 compression was being fully developed for the selected servers. Also ranking high in consideration of vendors was their ability to interface proposed solutions with the automation control systems already in use by the networks. After three or four months of research, testing, and site and factory visits, the design team selected Leitch Technology as the server vendor.
The next phase of the project would last almost nine months. During this time, engineers and operations staffers used specially written test software to subject the equipment to weeks of uninterrupted record and playback with no maintenance or human interaction. The servers were bombarded with commands and the resulting video was examined. One such test involved first defining the number of clips per hour that would have to be routinely delivered by the server. The software test then required the server to operate at 20 to 30 times that capacity.
During this testing period, CNN and Leitch engineers worked closely to examine the results of the tests and to make the minor software and hardware modifications necessary to meet reliability goals.
As work progressed on proving the core equipment to be used, various other facility issues came to the forefront, such as where to locate the servers. Since the current arrangement allowed each network to have its own terminal gear area, the decision came down to which one had enough space to accommodate the new gear. CNN and Headline News had about 3962 square meters of terminal gear space. This project would necessitate an expansion to about 5486 square meters, but this would ultimately shrink back to less than the original 3962 square meters. Space in the CNN terminal gear area was selected for the installation.
The CNN Headline News control room. The main technical operating position, shown on the right, uses a Grass Valley Group 4000 production switcher.
Cabling was another issue. CNN News Group networks are located in two multistory towers with a common atrium. Since the networks grew originally as separate entities, sufficient interconnect cabling to support shared server-based playback did not exist. Additional fiber cable was installed between the two buildings.
Meanwhile, testing and configuration continued with the Leitch servers. The design called for 30 dual-channel Leitch VR300 servers arranged in two 15-server arrays with 62 hours of storage on each half. Each array has 18 output ports and 12 input ports with RAID protected Fibre Channel storage. Any clip sent to the server is actually duplicated on both arrays. Sophisticated data management software is constantly checking to ensure that a complete mirrored environment is maintained.
Another important element of this project was writing and testing the software, which would allow the servers to interface productively with the Avstar Newsroom system used by editorial staffers to write, edit and build their newscasts. CNN Headline News was already using Avstar, and plans were under way for the other networks to implement its use as well. Leitch developed custom software for its VR300 servers to accommodate this requirement.
Always mindful of the need for a back-up plan, the design team developed an RS-422 controller, which is hardwired directly to the server and independent of any other data networks or computers. A control panel was built by DNF of California for each control room. The added hardware allows the control room staff to continue to operate the servers in the event of other systems failing.
As hardware testing and software development continued, the next step was selecting the first network to begin using the server. With its studio and control room renovation completed and the fact that its normal operation involved recorded rather than live playback to air, CNN Headline News was determined to be the best choice for first implementation.
In the fall of 1999, CNN Headline News began using server-based playback in a shadowing mode with their current setup. The new server worked so well that the network dropped its shadowing after only a few days. This quick success was due to the long months of testing the system had undergone, as well as the involvement of employees at all levels in the design and implementation process.
CNN Headline News Media operations area. Note the edit bays in the background.
With Headline News online, the CNN team could consider the next network to convert. CNN International (CNNI) seemed the best candidate. CNNI operates two simultaneous control rooms and often needs access to the same stories at the same time. Throughout the fall of 1999, CNNI’s operations staff worked on the conversion details involving not only accessing the server, but also implementing use of the Avstar Newsroom system and rearranging equipment in its control rooms. CNNI spent a significant amount of time getting input from employees on how the new design should be implemented in their operating environment. This hard work paid off in early 2000, when CNNI’s conversion to server-based playback was complete.
It is important to remember that for the CNN News Group, implementation of server-based playback was not occurring in isolation. During the time that this project was in process, the CNN newsroom was completely rebuilt. Headline News facilities, including control room, were relocated and rebuilt from top to bottom. In addition, a centralized routing system for video and audio was being installed along with a New Media Operations Area with 40 edit rooms that went online during the summer of 1999.
Timing and coordination of the various projects became an issue, as well as the application of resources to the projects. The same R&D and engineering resources had to be applied to all of these projects. One lesson learned from the server-based playback project was that for any project that touches so much of an organization, the time to complete is probably longer than first estimated.
With two networks now using the server, the project continues during 2000 to bring the other networks on line. Because of the effective close working relationship with the vendors on the project, any expansion needed in the future can be readily accomplished.
Overall, the CNN News Groups approach to facility design are:
- Work from a business-focused plan;
- Select flexible vendors who will buy into the design concept;
- Involve all levels of employees and test new systems exhaustively;
- Implement projects in a steady measured way, so that customers continue to receive the high-quality product to which they are accustomed.
and
Don Thompson is director of marketing for Leitch.
