Ol' Blue Eyes springs back to life through advanced image technology
NEW YORK
It took the late Frank Sinatra to elevate the planet's most advanced imaging technology to death-defying levels on the stage of the world's largest indoor theater. An army of talent and technologies-bridging film, video, special effects, computing, music, dance and advanced stagecraft-united in October to present "Sinatra: His Voice. His World. His Way," a stage production that gives new meaning to the concept of convergence. Sinatra's return to life at Radio City Music Hall in New York was a prelude for what is expected to become a multimedia attraction in venues throughout the world.
For a brief time, the technology disappeared. No one thought about the fact that 60,000 frames of film and video had been meticulously manipulated to bring Sinatra back to life as he sang along with live musicians backed by a 40-piece orchestra and accompanied by the Radio City Rockettes and other performers. Sixteen high-definition video projections displaying 3D images on a shifting montage of 40-foot-high movable panels and blocks all became a seamless blur as the show progressed.
In the end, it was all about the man and his music. That's the way the bright young staff of Guava, the New York City-based visual effects house, wanted it. Less than a year old, the fledgling boutique had scored a coup, landing the massive job of bringing Sinatra back to life. Twenty staff members and a bevy of freelancers tackled the production job, which was on the scale of a full feature film but due in one-third of the time.
The origins of reborn Sinatra consisted mainly of black-and-white 35mm film made during a 1957-58 ABC television series and color videotape from later productions. In the case of the film, Sinatra knew well the poor quality of '50s-era television kinescopes and, at his personal expense, rented a 35mm camera to attach on the television camera used for his singing performances. In all, Sinatra recorded about 60 songs on 35mm film that season, retaining ownership of all the footage.
Unfortunately, the original negatives are missing and all that can be found today of those filmed performances are copies once used as workprints. The workprint footage, on about 150 reels, was used to make frame-by-frame 2K digital files for the Radio City production. Extensively handled over the years, the film frames were covered with scratches, stains and editor's grease pencil marks. In some places there was adhesive tape across Sinatra's face.
EARLY ROTOSCOPICS
The '70s video, which made up about 25 percent of the stage production, was thought to have been originally recorded on two-inch quad tape. No one was quite sure of its origins, since the recordings had been dubbed several generations before reaching the engineering staff at Guava on Digital Betacam.
The first phase of Guava's work involved isolating Sinatra from his original backgrounds. A process called rotoscoping was used to cut Sinatra out of the original frame. The main rotoscope software tool used by Guava was Commotion by Pinnacle Systems. The technique, however, is far from new.
Developed in 1917, the rotoscope, a patented, mechanical device, predates computer technology. Its inventor was animator Max Fleischer, creator of Betty Boop, who developed the machine to project individual live action frames of film onto a drawing board. By tracing the projected images, the animator could quickly produce lifelike drawings.
At Guava, the rotoscoping process and such related techniques as keying, garbage masking and compositing were aided by Apple Computer's Shake 3 compositing and effects software. Next in the workflow came film restoration tools-in this case da Vinci's Revival software, integrated with Discreet flame and inferno visual effects systems.
"After isolating Frank, we had to restore him selectively. Not the whole frame, but just the parts of the frame that include Frank," said Ari Zohar Klingman, Guava's technical supervisor on the project.
"Revival and its render farm, Powerhouse, gave us a big advantage because it could read the Discreet storage system," he said. "That was important because after restoration the footage was going into the flame and inferno systems for color correction and digital special effects.
"Revival's toolset is very strong," he continued. "We are talking about light dirt, dark dirt, scratches, grease pencils, chemical stains, water marks-we had the full gamut. Revival's automated filter was very good and fast. The integration between the automatic and manual filters is strong. 'Automatic' got about 50 percent of the problems."
The vintage video presented different problems. The tube-based cameras that recorded Sinatra's performances would often leave burn-in marks of bright lights behind him. Parts of his movement would occasionally disappear into the burns. In some cases, there were 20 to 30 missing frames that had to be totally rebuilt from scratch.
Rotoscoping the video was more difficult because there were shifts in color separation.
"The best-case scenario for that (process) is a nice clean well-defined edge that you can easily find and trace. The video footage didn't have that at any time," said Alex Catchpoole, a Guava supervising visual effects artist on the project. "We needed to come up with solutions."
One way was to isolate a color image of Sinatra, rotoscope as well as possible, and then, instead of cutting him onto black as had been done with the film, make the edges monochrome and push them slightly out of focus. This, and a host of other visual tricks, solved the video isolation problems.
The biggest challenge with the video, said Klingman, was "the simple fact that it was 60 fields per second and NTSC resolution. The blow-ups that you see on stage are the up-converts to 24p. Early on we knew we were going to be faced with this challenge and the desire was to try to get the material to 24p as soon as possible."
Some experimentation with video upconversion provided a better than expected solution, said Klingman.
"We used the Snell & Wilcox Alchemist (Motion Compensated Standards Converter). It did a fantastic job. It took 60-field NTSC video footage... and we used it to up-convert to 23.98 PsF HD. That worked splendidly. We were very impressed with the quality of the upconversion. Also, because we went from basically a 30-frame interlace format to a 24-frame progressive format, we reduced the frame count that we had to paint by 20 percent."
Alchemist's motion compensation also improved the look of the video, Klingman noted.
"The result is actually what you see... several generations down of probably quad-originated two-camera video running through the Alchemist in a non-pristine state and projected at 40-by-40 feet at least," said Klingman.
Guava's staff singled out Apple Computer, Pinnacle Systems and da Vinci for exceptional support during the project. Klingman said his company made heavy use of Apple's G4 Macintosh computers for the project. Guava's finished high-definition segments, consisting of about 60,000 restored frames in all, were handed off to the Radio City production crew on Panasonic D-5 HD videotape format. The segments were then loaded, uncompressed, into a Doremi Labs V1-UHD disk recorder for cueing, sizing, and projection into the live show.
