Do it smaller, cheaper, lighter, faster, and by the way, in HD. That was David Mazza's assignment for this year's Summer Olympics in Beijing. As senior vice president, engineering-Olympics, for the NBC network, he and his staff were given a challenge for this year's games.
Do more — but quicker
Costs are a concern for every broadcaster, and originating the Olympics is no different. The invisible tribute to Mazza's team was the network's ability to produce more content, in HD plus 2200 hours of streaming programs, all accomplished by a production staff primarily remaining in the United States. This feat is largely due to the team's skill in pushing technology to new capabilities.
Key to his success was NBC's work-at-home initiative. The objective was to continue producing high-quality programming for increasing numbers of channels, but without the accompanying increased costs. Meeting this goal involved achieving several objectives: producing more HD and streaming content, expanding the work-at-home initiative, and doing so with a file-based workflow.
Step one required reconfiguring NBC's highly successful containerized core broadcast center. Beginning in 1999, the key components of the network's BOC were mounted in what the network calls RIBs, or racks in a box. Each RIB consists of two rows of 10 racks, mounted back-to-back with a work space between them. This allows engineers to mount and wire the equipment in the racks while still in the United States. After the wiring is completed, each RIB is moved into a shipping container and transported to the Olympic broadcast site. Once on-site, the RIBs are interconnected with other venues and equipment via ceiling cabling. The result greatly shortens build-out time.
Although some of the 2006 Winter Olympic Games were broadcast in HD, this year's goal was to provide U.S. audiences with a full complement of HD images in addition to several thousands of hours of streaming content. For the Torino broadcasts, the network built an overlay of HD on top of an existing SD infrastructure. Given the amount of HD content needed, Mazza needed to retool the BOC's central infrastructure.
Despite the requirement for HD, Mazza actually reduced the number of RIBs needed.
“We shaved off five RIBs. We had 13 RIBs in Torino, and now we only have eight,” he said. “Considering that we went from SD to HD, it is amazing that we were able to shave off that many RIBs.”
Part of this success stems from newer routing technology.
“The original SD router that we built in 1999 took up 10 racks, a whole half of a RIB,” he said. “That router was about 320 × 320 and only switched SD. Now we have an 800 × 800 HD router all contained in one rack. The new router reduced the required rack space while simultaneously almost tripling the number of crosspoints, all at six times the data rate.” Another important change was to move the RIBs closer to each other because NBC was pushing the limit of HD over cable.
“We needed to get the RIBs close enough so that we could make the connections on copper,” Mazza said. “Our goal was to reduce the setup time by 25 percent, which meant finishing in six weeks. We made it.”
Home sweet home
Another transformation in the network's Olympics workflow was to further build on its substantial and successful work-at-home effort. Beginning with the Winter Olympics in Torino, NBC experimented with doing actual content assembly in the United States. The process involved sending material back to New York so U.S.-based editors and staff could assemble final content. Not having to transport, house and support hundreds of production people at a remote location could result in significant cost savings. Mazza's support of this concept really shined in Beijing.
The work-at-home production was spread across eight U.S.-based control rooms. Five were at NBC headquarters at 30 Rock, one in New Jersey, one in Florida for Telemundo, and one at SoHo in lower Manhattan for the foreign language feeds. The work was divided into three types of programming: the off-tube factory, where voice-overs took place; the streaming factory, where new media streaming content originated; and the highlights factory, where SD and HD clip-based material was assembled.
While the Torino games required only 14 program feeds back to the United States, more than 100 separate program feeds were in place this year. Forty of these feeds were Web streams. The remaining feeds were a combination of HD and SD signals. NBC streamed more than 2200 hours of content for the Summer Olympics.
A 6000 mile remote
Because of the long distance between Beijing and the United States, any remote production solution had to meet three criteria. First, the content had to be resolution-independent. That's because the network needed to deliver content at rates ranging from 50Kb/s for streaming feeds to 50Mb/s for HD. Second, the solution also had to be location-independent. While the content originated in Beijing, production needed to occur in New York. Finally, the solution had to support a file-based workflow.
So what is the major challenge in producing content 6000mi remote?
“The first is physics,” Mazza says. “It requires about 240ms to loop the signal via fiber between Beijing and New York. This places unique requirements on any file-based content passing through routers.”
A key component in the work-at-home solution was provided by several Omneon products. (See Figure 1.) Twenty-two Omneon MediaDeck servers supporting 44 channels of ingest were based in Beijing, handling both IMX and XDCAM HD files. These servers also automatically generated matching low-resolution proxies for transmission to the United States. Both high-resolution and proxy files were then stored on a Beijing-based MediaGrid server. Eight channels of MediaDeck servers and another MediaGrid server were located in New York.
Once the content and proxies were generated, the proxies and some HD content needed to be sent back to New York. For this task, NBC used an Omneon ProCast CDN file transport engine. The ProCast CDN is designed to quickly move large files over long distances and yet remain unaffected by path length.
“TCP/IP sends a small blast of data, and then very quickly after transmission, it listens and says, ‘Did you get that?’ And if it doesn't get a response, it assumes that you didn't get it, so it sends it again,” Mazza explains. “However, with 240ms of delay, this could just result in chatter. ProCast overcomes this issue through the use of several transfer protocols.”
ProCast CDN's protocols permit file transfer speeds that can be orders of magnitude greater than FTP, especially over long distances like that from Beijing to New York. Two of the network's OC3 (150Mb/s) links back to New York were controlled by the ProCast CDN to speed file transfer.
Once the proxy files appeared in New York, shot pickers used a Blue Order MediaArchive DAM to browse, view and edit the content. The shot pickers were primarily responsible for shot selection for Web-based content. They would pick the shots, insert graphics and then drop it into a work folder that directed the file where to go. They could also send the content to specific locations. Some content might be targeted for an Avid editor based in New York, or even an editor based in China. The shots could also be fed directly to the Web system or to a MediaDeck for playout.
A different workflow was needed for HD content. Using proxies, a U.S.-based editor would develop an EDL and send it back to Beijing, where it was received by a MOG Solutions server. The server then retrieved the matching high-resolution images from the MediaGrid and conformed them into a single file, which was returned to New York. Those files could be further processed at 30 Rock or sent directly to the Anystream system for Web, VOD or mobile playout.
NBC rewinds
For the summer games, NBC developed a sports feed called “rewind.” Rewinds are actually live streams coupled with recorded metadata. The combination of streaming and content-specific metadata allowed the Web viewer to navigate through the content based on the metadata. For instance, a viewer could interactively select any video clip based on the statistics. If the viewer wanted to see a goal by a player that took place 11 minutes into the game, all he had to do was click on the appropriate metadata. The stream jumps to that point and plays out. NBC calls it user-driven highlights. The network produced approximately 3300 highlight clips a day to feed multiple new media platforms.
This production platform was essentially an automated distribution system, capable of flipping the file to whatever format the recipient needed. A three-minute clip of “goals of the day” could be sent to the Anystream encoder, which might make 50 copies of content in multiple file formats, all based on the destinations' needs.
“This was a huge part of what I like to call a very long-distance, file-based workload,” Mazza said.
Something old, something new
Mazza is a bit of a belt-and-suspenders kind of guy. In Torino, he said that the Olympics isn't a place to experiment with new technology because there is no room for mistakes.
“It's true. We say that,” Mazza said. “We couldn't run with that same level of risk for all those new things on the broadcast side because there's just too much risk.”
However, when it came to trying new things, Mazza says, “We ran a whole bunch of new things that had the risk meter essentially pegged. We couldn't have done that without both new technology and trying some new ideas.”
He adds, “The key is balance. You ask how many people are going to see the stream, how visible it is. If somebody tunes in to the Web at eight in the morning and they were thinking there was going be a certain clip and there was a slightly different clip, or we were trying to make 100 clips and we only made 90, most people won't know the difference. So we keep that in mind in determining how risky we're willing to be. But, we certainly pushed the envelope big time with the digital media arena.”
The continuing challenge
When asked why he keeps taking on this Olympic assignment, Mazza laughs, saying, “I find each game is a totally new challenge. I couldn't have told you two years ago that we were going to be launching all these new media projects. Sure, I knew something was coming, but we didn't know it would be this big. Our production staff and executives say, ‘Hey, we need to get to this point.’ They don't tell us how to get there; they just say, ‘Here's the starting point, here's the budget — you need to get to this point.’ It's up to us to figure it out.”
He adds, “With the Olympic broadcasts, there are always going to be new challenges. And as much as we try to manage the risk and say we don't want to try new things because a show is too high-profile, or there's too much to lose, we always end up doing it. For me and the team, it's kind of a love-hate relationship. We're preaching conservative working products, nothing new, nothing earth-shattering. Yet we really are doing earth-shattering things each time. That's kind of exciting, and it can even be scary at times.”
Mazzo laughs, saying, “But it's a good challenge.”
