After you've done something six, going on seven, times, you would think that the process would become easier. For David Mazza, senior vice president of engineering for the NBC Olympics, that's true — and not. “When I first took this job in Atlanta in 1994, I was a freelancer. We had budgets, and I thought this would be easy,” he said.
On the eve of the 2006 Winter Olympics in Torino, Italy, I sat down with Mazza to find out how the game had changed. It's his responsibility to show millions of American viewers the competition in living color and, this time, in HD. This challenge shows just how far Torino is from Atlanta technologically.
With NBC forking more than $614 million dollars for the 2006 Olympics and the pressure to deliver in all HD, one would expect Mazza to be a bit frazzled just days before the opening ceremonies. However, during our interview, he was as cool about the events as the snow caps on the nearby mountains. Cool, but not careless. When this kind of money is on the line, there's no room for error, hence his belt-and-suspenders approach to the use of technology. But more about that later.
Something tried, something new
Just three years ago, NBC committed $2.2 billion for the 2010 and 2012 Olympic broadcasts. These kind of numbers echo the pressure to be more efficient at everything. Sometimes that means you have to try new things, unproven things.
Each year, changes are made to the NBC Olympics broadcast facility. Sometimes this means new camera technology, tape machines, routers and such. However, one point Mazza strongly emphasizes: He doesn't like being first with anything.
“We're not trying to be new,” he said. Covering sports is always tough, you get only one take, and with the Olympics, the risks are even bigger.
“This isn't the place to experiment,” he said. And, buying new for newness sake is a sure path to disaster. Mazza and his crew must walk a path between the new (and unproven) and older (but well known and reliable) solutions.
Mazza faces a second equipment challenge: The games require millions of dollars in new equipment for a 17-day broadcast. “Rule number one is not to burn up capital that you're only going to use for three months,” Mazza said. “If I can't reuse it, I have to rent it. All our purchases are wrapped up in this kind of philosophy. This also means I won't buy something that's perfectly mature, because the downline customer wants something newer.”
The challenges of HD
While there have been some HD images from previous games, the 2006 Winter Olympics represented the first time that virtually everything brought to U.S. viewers was available in HD. This begins with most productions being originated and produced in HD, and only at the last minute downconverting the HD images to SD for OTA transmission. Oh, did I mention the entire production was handled in PAL?
So, what were some of the challenges in doing the Olympics in HD? Initially, it seemed straightforward. The needed equipment and budgets were there, so where was the problem? Turns out, the devil was in the details.
First, the system needed to be upgraded for HD. To avoid throwing away the whole SD infrastructure and building an entirely new 1.6Gb/s HD one, the team decided to overlay the new HD on top of the old SD infrastructure.
“This allows the existing SD equipment to be used for monitoring HD,” Mazza said. “And actual HD signals were plumbed only where required. The new Sony HD equipment and router were key in this effort's success.”
But there was another imp stoking the problem fires. “Technically, what consumed a lot of our pre-games time was solving the aspect ratio issue,” Mazza said. While the capture, graphics and production was handled in HD, most of the audience still watched in SD. This meant using center cuts, shooting in safe areas and preventing aspect-ratio production mistakes by way of some automated manner. A key part of the solution included using Sony HDCAM's SD output to automatically generate a 4:3 center cut.
Embracing an HD workflow
“You can't upset how people do things,” Mazza said about his decision to keep actual workflow similar to that used in previous games. “With less than a week to train folks, this isn't the time to introduce entirely new workflow practices.
“I'm also a firm believer that you can't just impose workflow on your people, your producers and creative guys. Sure, you may have to get people to change the way they do their work, but you can't impose workflow without listening to what they're trying to do.”
Another factor he took into consideration: “The NLE conversion has taken about 10 years. Because we are such a short-term event, our ability to exploit some NLE changes are harder for us to do,” Mazza said.
Graphics solutions
Graphics embraced a new HD workflow the most. This was headed by long-time director of graphics engineering and operations for NBC Olympics, Phil Paully. “One lesson from Athens was that we routed no video between an Avid edit session and a graphics session,” Paully said.
Echoing Mazza's point of listening to the users resulted in a simple, but effective, new procedure for storing images. “At Athens we came up with the idea of using a single graphics folder per day,” Paully said. “We have folders for each day, one per day for the 10 days leading up to the games and one folder for each of the 17 days of the games. This means everyone, including all the audio guys, knows where to drop their work.”
An Avid-centric workflow
The Olympics graphics department is responsible for all the eye candy that is so important to an exciting visual Olympics presentation — moving transitions, bumpers, transitions and openings. To make this happen, Paully's department relied heavily on Avid technology, primarily Avid Adrenalins. A total of 47 Adrenalins formed the backbone of the graphics and editing platform.
Avid's just-introduced ISIS (which stands for infinitely scalable intelligent storage) system was a new addition to this year's Olympics. Operating with Avid's DN×HD 120Mb/s signal, the SAN provides a highly scalable, self-balancing and distributed architecture, which was just what Mazza and Paully were looking for to implement HD production.
Because HD files are three times larger than SD files, the staff needed twice as much storage as in Athens just to stay even. And because graphics was the biggest storage hog, the team decided that Avid's ISIS would give them the biggest bang for their HD buck.
Inside the ISIS SAN
Images are stored in 8-bit 1920 × 1080 sizes at 120Mb/s. The NBC Olympic installation relied on a pair of ISIS SANS. Based on an 8TB, 4RU structure, an ISIS can be scaled up to 64TB of storage. A 40TB ISIS was is used in graphics, which provided about 380 hours of storage. A second 24TB ISIS supported the videotape room and three NBC edit suites.
The ISIS architecture supported Mazza's belt-and-suspenders approach. Multiple levels of redundancy exist on each server. Each of the four bays of drives that make up the SAN has its own backup drive.
Six drives form the basic storage module, with a seventh RAID spare drive. Should any problem develop, the spare drive automatically kicks in, and the system immediately begins rebuilding itself in the background. No images or time is lost.
Each ISIS is equipped with double-power supplies, dual-power feeds with auto switchover. Any type of system alert triggered sends an e-mail to Paully.
Outlying seats connected into the ISIS system. Users could point over to the graphics ISIS and select the desired elements, which were then transferred via GigE to the seat's local storage system for use.
On-site NBC network users “Access Hollywood” and Telemundo also had access to the ISIS SANS. While both operated in SD, the same storage served both HD and SD production needs. The SAN doesn't care whether the images are SD or HD; it's just data. A third ISIS SAN located in New Jersey produced the curling programming.
The benefits are more than just added space. “We also win on the dollar side by combining users rather than having separate storage systems, separate media managers, separate index servers and various bits combined together,” Mazza said. “The risk may go up slightly, but that's a balance we try to keep in check.”
Fully digital production
NBC creates the Olympic graphics on Avid Deko/Thunders, with all images saved as QuickTime files. QuickTime Movies become the basic file exchange format for Olympic graphics. “The term still store is obsolete,” Paully said. “There are no more stills; everything moves.”
The graphics stations were Macs running Final Cut Pro for ingest. Over-the-shoulder shots were managed by After Effects. All graphics and edit suites were the same so operators didn't have to train on multiple setups. All suites have identically loaded and configured software. Each edit station was equipped with 6GB of RAM, dual processors complete with HD I/O. Two graphics Macs, with 3TB of external storage, connected to the local SAN.
Venue ingest was handled by six Avid workstations connected to the primary graphics ISIS SAN. These stations processed the live venue feeds into the ISIS. At the end of each day, every venue created a highlight reel. The reel was then transmitted to the NBC facility and ingested into the ISIS. With six venues and six ingest stations, multiple feeds could be handled simultaneously.
The typical NBC Olympic edit suite consisted of two Avid Adrenaline media composers with DN×HD I/O. Miranda processors were used to embed and de-embed the audio from the SDI stream.
Each production suite had two Sony HDCAM VTRs, a router, HD LCD monitors, a Tektronix WFM and Dolby decoder, as well as NHT and Genelec speakers for surround monitoring. Two edit suites were also equipped with Panasonic DVCPRO HD decks.
Flexible graphics production
The Olympic broadcasts place unique requirements on the graphics department. Not only are there thousands of graphic elements, but it's impossible to plan for every contingency.
“Things change, so you need people pushing buttons,” Paully said. “When the director says ‘drop cut seven and insert cut six,’ automation can't do that. Sports is a seat-of-the-pants operation. For instance, we don't have a graphic for someone breaking a leg or falling on the ice or sky jump. If that happens, we need a custom graphic. You have to be able to change on the fly.”
Transferring moving graphics was another issue that had to be solved. The Avid Deko graphics platform provided the solution through Make DekoMovie. “Every graphic that we make is a movie,” Paully said. “We don't make still graphics any more. Backgrounds move. Heads animate. There's always motion in a graphic.”
Any information an artist creates, such as clips or CGs, is done with Make DekoMovie. Once an artist saves it as a movie, the graphic can be shipped or dragged and dropped and rendered on any Avid desktop.
“Our workflow had to change because we couldn't tie up 14 edit rooms trying to time graphic changes to video,” Paully said. “Now we have four operators. They can pre-build images and let the Avid render it at the desktop. We'll do a QuickTime MXF transfer and simply drop it onto the timeline. We proved this works at Athens.”
Paully created a flexible and fast graphics and editing platform at a cost he could afford. “If it wasn't for the PC and Mac platforms, I don't know how I'd afford to do graphics. They've allowed us to go from using custom, proprietary hardware that cost $350,000 per room to $40,000,” he said.
Proxies and metadata
The workflow at the Summer Games in Athens relied on MXF proxies generated by Sony eVTRs. Because that feature hasn't been implemented yet on the HDCAMs, which were employed in Torino, a different solution was needed.
Forty MOG Solutions proxy encoders were used to generate the 16:9 MXF proxies. The real-time MPEG-4 proxies were generated for each venue feed to support desktop browsing at both Olympic suites and back in the United States.
The proxies were handed off to a Blue Order MAM system, which, among other chores, added scene transitions information and generated a storyboard. In addition, during live events, operators manually inserted dynamic metadata such as race results.
Editors could access the proxies within seconds after the ingest started. The proxies were searchable locally and via IP through Internet interfaces to multiple remote locations. The system allowed graphic artists and editors to quickly find the footage they needed as well as identify the tape reels containing the master footage, all without tying up a lot of network bandwidth.
Tape is not dead
At this Olympics, NBC recorded almost 20,000 hours of tape, which would be tough to put on spinning disks. Any disk solution would have required a lot of drives, creating its own set of problems. “Besides,” Mazza said, “those HDCAM tapes are a lot cheaper than a disk drive. Also, there is the added benefit of distributed risk. If one tape machine breaks, there are still 149 machines working.”
Tape is still the most portable of mediums. And, according to Bill Lorenz, project manager of edit systems for NBC Olympics, it's very reliable. “You have to have a piece of tape to go anywhere you want to go. You aren't going to trust any disc system with your entire show. If you can, you'll put a piece of tape down at the same time you record to hard drives. You can take tape anywhere; it's a wonderful way to operate,” he said.
Last, but not least: audio
Audio is a huge component of any HD broadcast, and NBC didn't short-change this effort either. Most venues created six discrete audio channels. Once they were returned to the NBC broadcast facility for post production, the discrete channels were encoded into two-channel Dolby Pro Logic II. This turned out to be key to moving some 250 HD video signals around the facility with accompanying surround audio.
By re-encoding the original six channels into two Dolby Pro Logic channels, the NBC infrastructure handled the full mix and effects sound. The studios still monitored discrete surround.
However, to maximize the benefits of capturing in six discrete channels, Dolby modified the encoding software slightly to spread the center image somewhat for decoding in viewer's homes. The result was a wider aural image that reinforced the HD images.
At NBC, the graphics department handles all the sweetening for bumpers, promos and teasers. Therefore, along with the HD upgrade, Paully and his crew needed a different audio mixer. He selected Digidesign Pro Tools driven by an IKON digital console control surface.
“This Mac-based console interfaces seamlessly with the Mac-based edit stations,” Paully said. “Although the console also comes in a PC flavor, the majority of our audio engineers prefer the Mac version.”
Ship it home
Once the Olympic content was packaged for transmission to the United States, it was sent via geographically and technologically divergent paths. The SD and HD signals even took different paths. In other words, there was no single point of failure in any of the three paths — belt, suspenders and a second belt approach.
The HD feeds were converted by ShibuSoku converters. The SD feeds were converted by Snell & Wilcox converters. TANDBERG Television transmission encoders were the last devices in the NBC Olympic center to see the images before they were relayed back to the United States.
That's a wrap
Back in 2002, when I interviewed David Mazza at the Salt Lake Winter Olympics (“An Olympic success,” March 2002), he compared the Olympic production to doing 16 Super Bowl games — all at once. When I asked the same question again this year, he said, “The Olympics is a giant remote. We still do everything in a temporary fashion; we just do it in a much bigger way. We have a lot more risk. We're on more nights, and we have more stuff and people. But, at the end of the day, we'll do whatever we have to do to stay on the air.”
So is it worth all the effort? Mazza quickly says “yes.” The only disappointing side, he sighs, is that despite all the effort of his network and crew to produce pristine HD images, 90 percent of his audience never saw them. Most Olympic viewers were still watching in 4:3 and analog NTSC.
Oh well, there's always 2008 in Beijing. Stay tuned.
