Certain technologies just limp along for years until they crystalize—and then suddenly its potential just explodes in your face. That has happened with virtual television and film production, which really came of age during the pandemic.
Some form of virtual production has been used for a long time, with a good example being 13 years ago on James Cameron’s film “Avatar.” However, that lavish and experimental film was far from a mainstream production.
Today, virtual production is being used on television programs and saving real money. Smaller crews, budget constraints and pandemic restrictions have been a driving force behind the surge of the technology. For example, the FX series “Snowfall,” a show set in Los Angeles during the crack cocaine epidemic of the 1980s, saves about $50,000 per episode using virtual technology.
Since the term has been used so loosely over the years and there are different definitions of virtual production, we’ll start with a modern description, so we are all on the same page.
Virtual Production Defined
“Today, virtual production is the use of real-time technology in production. It’s normal production—all the things that filmmakers already do—with the added ability to look at their visual effects or animation in real time,” said David Moran, director of industry relations at Epic Games, the maker of Unreal Engine 3D creation software.
“Before now, all those things were done in post and took a long time,” Moran told a 2022 NAB Show panel. “Today, we can do them with live action and normal production techniques.”
Virtual sets replace green screens and can use 2D playback of photographic footage on giant LED set walls, or 3D tracking that moves with the camera so the image changes as it does on a real set. A big difference today is the use of high-resolution LED walls to project the background of scenes, like those used on FX’s “Snowfall.”
“Computer graphics at first were very slow,” Moran said. “It took a long time to do renders, like four hours per frame. It was not good for visualizing what you were doing. Virtual production started a long time ago—Robert Zemeckis used it on ‘The Polar Express,’ a 2004 film. But then it was just the director who could see it in his own virtual camera. The latest phase of virtual production is the one where you can put the visual effects on a LED wall in front of the entire crew. Everybody can now work as they normally do on the set.”
A.J. Wedding is the co-founder of Orbital Studios in Los Angeles, a virtual production studio with satellite operations nationwide. He is also the virtual production supervisor for “Snowfall,” which uses a 62x14-foot LED wall for a penthouse set alone.
“A lot of the things that were not possible yesterday are possible today in virtual production,” said Wedding. “With pixel pitch being much tighter, you can get the camera closer to the LED wall. We are doing a lot of things that even just a few months ago were not possible. The technology is getting that much better very quickly.”
Christian Herrera, a director of photography who has worked on Season 5 of “Snowfall,” said scenes can be shot closer on LED set walls and the wall itself becomes a light source.
“The technology is still in a very nascent stage,” he added. “So we try to work with the LED panel and camera manufacturers and say, ‘These are the issues we’re having. How can you solve them? Can you change the pixel pitch? Can you give me panels at 120 hertz?’ And we do a lot of tests about color. Color is a big problem because camera sensors don’t necessarily see the same colors that your eye sees. That’s going to be a huge improvement that’s still coming.
“But having LED walls, instead of a green screen, makes the set a lot more realistic,” Herrera continued. “The reaction of the actors to these LED walls where they can see a playback image, allows them to get more involved, just as having a real set on a real location. They perform in the moment.
Video Plates in Action
Herrera said he remembered a scene where “we had a nighttime video plate and the camera looked at a table where the moon was reflected on it and then it moved up to the actors. The moon was what was lighting the shot from the LED wall and it was kind of awesome.”
Webbing added: “The workflow in television production moving from preproduction to post is tough. We were able to develop a system that’s easier for TV schedules by using video plates that we track in two-and-a-half dimensions. We basically layer it—something we can do very quickly.”
Herrera said, “Making the video plates is a collaboration. They must work on the narrative for the script that we have. So it can get tricky going through which kind of plates and what time of the day do we need to capture. Say we do 2,500 plates on the streets of L.A. You have a lot of information and detail in the buildings, on the streets and in the moving cars. We capture all those plates with a high ISO camera. You can see all these moving elements that you cannot get on a green screen.
“Also, for a lot of these plates, we have to remove buildings that wouldn’t be there in 1985, the era of the show,” he continued. “We had to have period cars driving by. If you think about having to shoot that, in reality, you’ve got to lock up the street and have the cars going by. It’s a nightmare to do that.”
Moran said after the making of the 2009 film, “Avatar,” there was an explosion of interest in virtual production. But, he said at the time, all the productions were very expensive, and the industry failed to make the technology palatable for ordinary productions. Then, over time, virtual production evolved. It became their mission to take the real-time technology and bring it to a broader group of production types.
“We were careful to not hype it too much,” Moran said. “Now we’re listening and learning. We have put together a lot of training programs—one is called the ‘Unreal Fellowship’—you don’t go to class, you learn on the set; it’s an apprenticeship on the set. Also, standards are very important and must be developed.”
Virtual production, when it is fully implemented, will bring another seachange to television and film production. “If you’re going to do full virtual production, you’re going to have to shift not just the workflow, but also your budget,” said Webbing.
“A lot of people say ‘Oh, my God, virtual production is so expensive.’ But you’re taking money from other buckets and putting them into this bucket. In the end, it is actually money saved. Using this technology, instead of doing multiple locations, is a lot easier and higher quality than any other way of doing production.”
This article has been updated.
