An operator monitors HBO’s 32 internal channels on Barco VGA monitors, Panasonic plasma displays and Vivaldi monitor display hardware in the master control suite of the HBO Communications Center.
Since the HBO Communications Center was built in 1983 as a distribution center working out of a futuristic, custom-designed site, the Home Box Office (HBO) network has always forged new technological ground. That's because its business model demanded it: Over the years, the company has experienced incredible growth, expanding from four channels to a bi-coastal network of 28 standard-definition and four high-definition channels and on-demand distribution.
With such a large responsibility, the facility has to remain operational 24 hours a day, while providing responsiveness to industry change. In this case, 99.99 percent reliability is not good enough: HBO strives for 99.999 percent, which equates to no more than five minutes of downtime, per channel, per year.
When the network chose the technology that serves as the heart of the network's playout systems, it selected established players in both the broadcast and computer industries. This combination provides a high degree of system reliability and technology innovation.
The network considered moving to a server-based model back in the mid-1990s, as its tape-based systems began to reach their seventh year of online usage, but the cost of storage was prohibitive.
Multiple Grass Valley Open SAN systems from Thomson, consisting of two Profile servers each, hold about four days’ worth of content each for four HBO on-air channels. Each movie file is encoded as an MPEG-2 file and stored on hot-swappable Ciprico 146GB drive arrays.
When the time came to move from tape formats in SD and HD to a tapeless distribution system, HBO chose Grass Valley Profile XP Media Platform servers, Grass Valley Storage Area Network (SAN) systems and Venus routers from Thomson, combined with Sun Microsystems 6800 mid-level series servers and Hitachi 9980 storage systems.
The HBO Communications Center is a study in the tight integration of traditional broadcast and forward-looking, computer industry systems now available to support network operations.
Systems must be online 24 hours a day
The facility is protected by redundant power distribution that can keep it up and running for 18 days without outside fuel refills. During the massive East Coast blackout in 2003, for example, the facility continued to operate uninterrupted.
Two Sun Microsystems 6800 series mid-level servers, located on separate floors for redundancy, provide offline storage for approximately one year’s worth of content. Content moves between storage arrays at 350MB/s.
Beginning in 2002, HBO has so far transitioned eight of its channels to the new Sun-Thomson solution. The plan is to have the remaining channels playing from this architecture next year. When complete, HBO will move the same amount of content from the Sun architecture to six Grass Valley PVS 1100 Profile XP Media Platform servers as approximately 75 Digital Betacam VTRs.
HBO online storage starts with redundant Sun 6800 mid-level servers, on two separate floors, holding 50TB of storage each. The storage, provided by Hitachi, equals about 5000 hours, or one year's worth of content. Files are moved between storage arrays at 350MB/s.
These systems are linked to two Grass Valley Open SAN systems, consisting of two Profile servers each that serve the on-air channels directly. Each feature file is encoded as an MPEG-2 file and stored on hot-swappable Ciprico 146GB drive arrays.
Figure 1. Storage at HBO includes Sun 6800 servers for offline content (top) and Grass Valley Profile XP Media Platform servers (using RAID arrays) for programs airing that day. Click here to see an enlarged diagram.
Each SAN system is dedicated to four HBO channels, holding about four days' worth of content. Ensuring reliability, one SAN system serves as a backup to the other. Content is moved from the Sun server to the Grass Valley Profile servers via FTP file transfer at about 30Mb/s, with burst rates slightly higher. As a particular feature is readied for air, two copies are stored on the Profile servers about one day before the feature is to air. This process ensures that the content will air at the pre-determined time because the servers work in tandem. There is no human intervention as the FTP process moves about 1TB of data per server per day.
HBO wrote custom Java-based applications that enable the Profile servers to automatically duplicate a file and maintain replication after quality control. Written to be compatible with Thomson's Grass Valley ContentShare software inside the Profile servers, the programs were recognized by the Java community with a special Enterprise Design award. The award was presented by Sun CEO Scott McNeeley personally to HBO last year. As a final disaster-recovery option, HBO intends to create a third copy of a file and send it to the network's production facility in New York City, 50 miles away.
In addition to an advanced server infrastructure, the HBO facility includes a main master control (MC) suite and three control rooms. The MC suite monitors a total of 32 channels internally and is equipped with a full wall of Barco VGA monitors, four Panasonic plasma displays, a New Point Technology alarming system and Barco Vivaldi monitor display hardware.
There is one dedicated control room for HBO's 16 East and West Coast feeds, which incorporates both its SD and HD channels, a second control room for CineMax and its 12 East and West feeds, and a third “breakaway” room that handles all live events. During a live telecast, the West Coast feed is “broken away” from the delay (Profile) systems three to four hours prior to the event under automation.
Custom Java applications automatically encode incoming files and quickly distribute them to the proper channel — reducing the chance of human error.
HBO is developing a multi-tiered storage infrastructure using Java-based software that directs the content into two tiers. Tier 1 serves as online storage — content that's to be used immediately — while Tier 2 storage is used basically as an archive.
Most features now come into the facility on a Digital Betacam cassette, but the network is quickly moving toward a true file-to-file conversion approach that, in the future, will eliminate the use of videotape completely. As files are ingested into the system, approximately 15 Java applications help handle the material and reduce human error. Incoming files are automatically encoded and distributed to the proper channel with-in minutes. Interestingly, once a feature has reached its end-of-play cycle, it is not archived at all. When the feature's licensing agreement window has expired, a new file is overwritten on the drive to replace it.
All distribution of on-demand content comes from the HBO Communications Center in Hauppauge, NY, from which HBO distributes approximately 150 to 200 hours of content to cable systems across the United States. HBO has been working with encoding manufacturers and automation providers — as well as developing its own applications — to employ electronic workflow processes that will better support the growth expected in HBO's on-demand service.
Tiered storage
The best part about HBO's digital facility and its systems is that moving to the highly automated Sun/Hitachi and Profile servers and Open SAN systems has resulted in the development of electronic workflow architectures that increase efficiencies and allow HBO to put additional safeguards in place to continue its mission-critical distribution.
On-demand distribution
Michael Grotticelli regularly reports on the professional video and broadcast technology industries.
HBO:
Bob Zitter, exec. VP, CTO
Charles Cataldo,
sr. VP, broadcast op./eng.
Elmer Musser, Jr.,
VP, broadcast eng.
Kenneth Chin, TD, broadcast eng.
Barco:
CRT-based VGA monitors
Vivaldi monitors
Chyron Maxine CG
Ciprico 146GB drive arrays
Design team
Equipment list
Thomson:
Grass Valley Profile XP Media
Platform servers
Grass Valley OpenSAN
systems
Grass Valley Venus routers Hitachi 9980 storage systems New Point Technology alarms Panasonic plasma displays Sun Microsystems 6800
mid-level series servers
