In broadcasting as in life, things rarely go as planned. The proof is Rainbow Media’s VOOM HD Networks (www.rainbow-media. com), 15 HD channels featuring sports, movies and music programming that are carried on The DISH Network.
Initially, each VOOM channel featured “a single pay-per-view event that was repeated throughout the day,” says Steve Pontillo, Rainbow Media’s senior vice president of broadcast technology. However, it didn’t take long for 12 of VOOM’s HD channels to expand into fully scheduled broadcasts. For instance, VOOM’s movie channel went from playing one pay-per-view program daily to 12 different movies in the same time period.
Before this expansion happened, “the amount of video storage that was required for each of our channels was pretty minimal,” Pontillo says. “In fact, we had enough room on our Media Area Network [MAN] to store 15 days’ worth of content for each channel.”
However, after 12 of VOOM’s channels expanded their broadcast line-ups, things changed. “The requirement for storage really became unmanageable,” says Pontillo. “It got so bad, that we realized that we either had to reduce the amount of content we were offering on VOOM, or come up with a new storage solution that could handle all our extra HD programming.”
FRONT PORCH DIGITAL TO THE RESCUE!
Desperate for help, Rainbow Media turned to Front Porch Digital (www.fpdigital.com), their exclusive supplier for digital storage solutions since 2003. Front Porch’s answer was to deploy an HD-only version of DIVArchive, a unit capable of storing up to 28,000 hours of HD programming on digital data tapes. Rainbow Media was comfortable with this approach, and for good reason: they were already using an SD version of DIVArchive to store up to 100,000 hours of SD content.
This said, “There were a lot of two to four hour long conference calls between Rainbow Media, Front Porch and suppliers such as Grass Valley, Harris and systems integrator Communications Engineering Inc., when it came to planning how the HD-only DIVArchive was to be installed,” says Rainbow Media General Manager John Barbieri. “We knew that we were breaking new ground by installing such an ambitious content storage solution for HD. We really struggled to ensure that the new system wouldn’t just deal with our current HD storage problems, but also have enough headroom to allow us to expand our HD schedule further without running into limitations.”
A HYBRID SOLUTION
With HD video files being about three times larger than their SD equivalents, and with 50 Mbps of bandwidth being required to move a single HD program across VOOM’s MAN, it was clear that solving Rainbow Media’s HD problems would require some creative thinking.
A case in point: replicating the data tape-only approach of VOOM’s SD DIVArchive just wouldn’t work. “We already had bottlenecking problems in our system caused by moving SD content from data tape to Grass Valley video servers for playout,” Barbieri explains. “So using the same approach for HD content was simply out of the question. In fact, we were hoping to come up with a solution for HD that we could then implement in SD to resolve our existing bottleneck.”
Ironically, the problem wasn’t caused by the tape drives being too slow, but too quick. “Data tape drives are typically much faster than the video servers receiving the digital assets, presenting a bottleneck as tape drives are forced to wait for the video servers to process the data,” explains Brian Campanotti, Front Porch’s CTO. “By introducing fast managed disk storage in this communication path, data tape drives are able to run at full speed while the slower transfers to/from the video servers are accommodated from disk rather than directly from tape.”
At this point, savvy readers might ask why Rainbow Media was using data tape for SD archiving, when disk-based storage would allow quick, nonlinear access to archived files? The answer is cost: “Even with the continuing drop in hard disk drive costs, using HDDs would cost ten times as much as using data tape,” says Jeff Esposito, Front Porch’s director of professional services and deployment. Yet Rainbow Media already knew that going directly from tape to video server was causing a bottleneck on its SD system, so doing the same thing for HD was also out of the question. This is why Front Porch decided to take a hybrid approach to the problem, by adding ‘managed disk storage’ or nearline storage to its DIVArchive package.
“Specifically, any HD content that comes into Rainbow Media is first stored on an HDD-based managed disk storage system for 30 days,” says Campanotti. “After 30 days, it is transferred to data tape on one of eight LTO2 tape drives. The expandable data tape library can be expanded from its current capacity of 300 TB as Rainbow’s HD storage requirements increase.”
By using this hybrid approach, Front Porch has addressed Rainbow Media’s bottlenecking concerns quite effectively. With fresh content being stored on HDD, it can be quickly accessed as needed by VOOM’s Harris automation system. Then, by being archived after 30 days on tape storage, the content can be housed at the most affordable price point possible. Granted, accessing it from tape will take longer than from disk, but the compromise is a good balance between cost and performance. Not surprisingly, Rainbow Media is now applying this same solution to its SD DIVArchive.
A final point: data tape storage is “easily scalable” for future expansion, Esposito says. In other words, Rainbow Media can simply add more LTO2 systems to handle extra content as needed. Meanwhile, as the company’s other HD networks expand their broadcast schedules, the managed disk storage system can also be expanded to store this additional content for 30 days.
CONSIDERATIONS
One of the most striking aspects of Rainbow Media’s HD archiving system is that it is not integrated with the company’s SD program stream. In fact, the two are “totally independent,” says Pontillo.
The reason for the separation is “data throughput,” he explains. “We were concerned that mixing SD and HD programming on the same system could cause further bottlenecks and delays that could jeopardize our national services. To prevent this, we decided to keep SD on one archiving platform, and HD on another.”
In making this choice, Rainbow Media is now running “what may well be the largest dedicated HD archive in North America,” says Mike Knaisch, president of Front Porch Digital. “We expect such separate HD systems to become the norm as time progresses. Given the significantly larger file sizes required for HD versus SD, running separate HD-only archives makes sense. Besides, as HD establishes itself in the marketplace, the need for SD storage may well diminish over time.”
Another striking aspect is the HDD/tape mix described earlier. For broadcasters wondering if such a hybrid solution would make sense for their operations, they should start by deciding how long their archived material needs to be available for quick access.
Take news: most news stories get aired a few times, then head into the archive and are never seen again except perhaps during year-end wraps or as part of Whatever Happened To... look-back features. In such an operation, it doesn’t make sense to keep content for 30 days on an HDD system. Instead, only ongoing stories need to be kept readily accessible on disk—events like Hurricane Katrina or the infamous Janet Jackson breast flash—while the rest can be sent to cheaper data tape immediately.
In contrast, broadcasters such as VOOM who repeat programming on a scheduled basis have a real need for 30-day HDD storage. The same can be said for networks that run certain series in virtual perpetuity, as is the case with the various Star Trek series. In fact, it might be wise for Paramount to package Star Trek on a series of external HDDs, in line with this truth.
THE FINAL FRONTIER
Make no mistake: “Rainbow Media built our HD archive out of sheer necessity,” says Pontillo. Yet, by doing so, the company has solved real problems for itself, particularly by using a mix of HDDs and data tape. Add the fact that DIVArchive can create low-resolution Windows Media proxies of everything it stores, and Rainbow Media not only has a fast way to make offline edits of its archived content, but also a quick-to-deploy backup programming source should something happen to its main video servers. Currently Rainbow is only doing this on its SD DIVArchive system, Barbieri says, and “we intend to move in that direction for HD.”
Clearly, Rainbow Media has found a way to cope with VOOM’s rapidly expanding broadcast schedules both today and in the future. Broadcasters who are wondering about their own HD archiving needs, and just how much those needs may grow in years to come, would be wise to take note.
James Careless covers the television industry. He can be reached at jamesc@tjtdesign.com.
