MUNICH—When I was in my “formative years” (whatever they are), my parents, relatives, teachers, neighbors and others passed on many gems of wisdom. One of these was “believe nothing you hear, and only one half of what you see.” Beyond the obvious gossip and tall-tale factors, this seemed to be solid advice back in the 1950s when audio tape recording was coming into its own and the media was full of stories about how easy it was—with nothing more than a pair of scissors and some splicing tape—to totally rearrange and alter the meaning of recorded conversations.
Lytro light field motion imaging camera There is a now a technology that can totally undermine even that promised 50 percent of believability: light field imaging. Germany’s Fraunhofer Institute has been rather quietly touting it at trade shows for the past several years, but it wasn’t until the 2016 NAB and IBC Shows that it got a lot of notice, perhaps due to the many other high-tech entries that the large research organization always brings along. Also, with all the buzz about UHD, HDR, HFR, ECG and other “disruptive” technologies, light field imaging seems to have gotten short shrift.
However, this is one “light” that shouldn’t be hidden under a bushel. The technology offers some really remarkable advantages in image capture, and brings with it a large share of “wow factor” as well.
So what exactly is light field technology all about?
GOING BEYOND CONVENTIONAL IMAGE CAPTURE
In a nutshell, it’s a technology that adds another quantity or dimension to image capture. In conventional imaging, be it television, still camera, or cinema rig, the imaging device (or film), in addition to recording light intensities, really only captures two quantities: the height and width of the object or objects upon which the camera lens is focused. A light field camera builds on this by defining the direction from which the light rays reflected off the object or objects being captured originated. In other words, another dimension—depth—is also recorded.
Jon Karafin You may say “so what’s new?—3D does this and it’s been around for years.” True, 3D is a form of light field imaging; however, it’s a “low sampling” manifestation, as the 3D images are created from only two optical samples (left/right eye). Today’s light field technology goes way beyond this, capturing many, many more such visual samples that are representative of an equally large number of viewpoints, thus generating a very large amount of information about a scene.
“Capturing light field is like photographing a digital holographic image of the world,” said Jon Karafin, head of light field video at Lytro, a key player among the dozen or so entities involved with light field imaging research and development. “It is the true merger between the virtual and real where you can have the benefits of a virtual production set while shooting in a real-world environment. With light field data, you have unprecedented creative control during post-production as well as unfounded capabilities for visual effects.”
Pete Ludé Pete Ludé, chief innovation officer of RealD, another company that’s deeply involved in advanced imaging technology, enumerated some additional advantages associated with light field cinematography.
“There are two important practical benefits,” said Ludé. “One is that since you are able to capture the light rays from all directions, it allows you to refocus and readjust depth of field in postproduction. It also allows you to reset the stereo base on a stereoscopic image capture and it allows you to do depth mapping, which sort of segues to the second practical benefit, which is that you don’t need to use green screens to do mattes and masks.
“Encoded within the image itself is the volumetric space that allows you, later in postproduction, to know where to set the background plane without having to light behind the actors. You could just shoot [the scene] anywhere and the computers will later know where to insert the background plates without any further action.”
Light field brings with it another benefit too—very selective focus by artificially creating an extremely shallow depth of field.
“Today you can shoot at f2.4 or f16 and get different depths of field,” said Ludé. “By using light field, you can mathematically calculate what the depth of field would be with an f0.2 lens—which is physically impossible to construct, but mathematically you could treat the light rays as if that were possible.”
Also, due to the multiple “point of view” capture inherent in the system, it’s just as easy to create a 3D product as a 2D version.
Siegfried Foessel Siegfried Foessel, head of Fraunhofer’s department of moving picture technologies, also offered his own take on the magic of light field imaging.
“You can now capture multiple elements of a light field, which allows the recombining of individual light rays for rendering new images, and that is the basis of light field,” said Foessel, noting that the Fraunhofer has been very interested in the development of light field imaging, even to the point of creating a special study group as part of the organization’s Digital Media Alliance for exploring “computational imaging” some five years ago.
ENABLING THE TECHNOLOGY
While the concept of light field imaging has existed for more than a century, not much was done with it until fairly recently due to the size and other physical limitations of devices necessary for its practical implementation. That has all changed with the arrival of both smaller imaging devices with higher resolutions, and very high-speed processing engines.
Lytro has used today’s advanced solidstate imager technology to construct a highly specialized camera with the image quality necessary for high-quality cinema productions. It’s been used for several projects, including the creation of a short production (“Life”) that shows off some of the remarkable capabilities achievable with light field photography, and which has been exhibited in connection with light field presentations at both the NAB Show and IBC this year. This demo film included seamlessly intercut footage from both the Lytro camera and ARRI’s ALEXA.
The Fraunhofer experimental light field imaging device consists of nine small HD cameras mounted in a 3 x 3 matrix. So are there any drawbacks to this revolutionary image capture technology? Perhaps the greatest one right now is the physical size of the camera. It’s constructed around a very special 755-megapixel sensor that’s backed up by an equally specialized micro lens array (think compound insect’s eye), which spreads light rays from many directions over the surface of the large imager and is quite ponderous.
“The camera is approximately six feet long and weighs hundreds of pounds,” said Karafin. “It is not designed to be handheld, but rather mounts directly to a crane or dolly. We provide a production truck to transport the system to and from the production location to further enable the technology to be leveraged by any studio or project.”
However, it must be kept in mind that this is a first-generation device. Recall the enormity of the original three-strip Technicolor motion picture camera and the size and weight of first-gen color television cameras and how these steadily shrank with advancing technology.
Karafin noted that “the next generation of the technology is designed to be more commensurate with an industry standard definition of a traditional cinema camera.”
A DIFFERENT LIGHT FIELD METHODOLOGY
Not all light field research is being done with a single sensor and micro lens array though. Fraunhofer’s Foessel described a somewhat different approach taken by his organization using multiple small cameras with individual lenses.
“For the movie production itself we saw that the approach with a micro lens array was a bit too limited, because you are restricted in viewpoint renderings to the dimensions of either the sensor, if you have no main lens, or to the entrance pupil of the main lens,” said Foessel. “So you cannot calculate so much special effects based on our micro lens array sensor, especially if it has no main lens, and that is the reason why we continued with camera arrays. With the camera array, you can reposition each camera differently and make different arrangements to capture the light field in a wider space, and this allows us now to make virtual camera movements and to make more special effects [such as] looking behind objects and so on.”
In describing the multiple camera array, Foessel was quick to note that Fraunhofer has not abandoned its work with micro lenses.
“We are still working on projects with micro lens arrays for new ways to capture images,” said Foessel. “One of these is for capturing images in smartphones, mainly to get additional depth information.”
THE FUTURE OF LIGHT FIELD
So where is this technology going? Will it eventually replace conventional imaging technology for movie production completely? Will it be possible to construct a television studio or movie sound stage with the walls ringed with light field camera arrays, allowing a director to get 360-degree shots with no movable cameras (or camera operators)?
Foessel believes that this could happen, but says it will take time.
“I think it will be probably in conjunction with development of the cloud, because you need a lot of computation power to do this, and the cloud is now exponentially growing and more and more of the broadcasting and movie production is moving to the cloud,” he said. “So I see this happening within a five to 10-year time frame.”
Stay tuned. In the meantime, if you’re invited for a demo of light field imaging and processing, by all means take advantage of the opportunity. You’ll be witnessing something that’s really cutting edge!
