Karl Paulsen Of the numerous elements involved with digital file-based workflows, three seem to be evolving at a phenomenal rate: storage, archiving and transcoding.
Some believe that storage and archival activities should be merged onto the same device or platform. Others view these as distinctly different processes, involving media that can be succinctly distinguished based upon accessibility, need and purpose.
Granted, when looking at the accessibility side, one must consider both capacity and speed as well as which types of media and formats fit which set of activities. For example, hard disk drives (HDD) reach data rates of around 500 MBps with solid state drives (SSD) upwards of between 400-900 GBps. The latter SSD figures employ PCIe SLC flash memory arrays offering 2 GBps read bandwidth and 1.4 GBps write bandwidth, 30 microsecond latency and employ eight PCI Express Lanes.
These are unprecedented factors that were previously unavailable even with the high-end digital data storage service providers; yet these may not be necessary or even applicable to your workflows.
Looking at the archive side, the familiar LTO-5 product provides 1.5 TB of uncompressed storage per cartridge with a maximum transfer speed of 140 MBps (uncompressed). These figures double when using LTO-native 2:1 compression, although not recommended or advantageous for most video media applications.
Fig. 1: Traditional Production Workflow: Discrete systems—coupled together via multiple sets of hardware, software and media components—producing various forms and formats for application-specific workflows
(Click To Enlarge)
Generation 6 (LTO-6) Ultrium tape increased storage to 2.5 TB uncompressed (6.25 compressed) with up to 160 MBps transfer rate, the equivalent of about 1.4 TB of data transferred per hour. Vendors are offering a “pre-purchase” option for LTO-6, with an expected availability (at the time of writing) sometime in December 2012.
Traditional workflows often consist of discrete systems (Fig. 1) that add time, risk and complexity to production processes. Today, opportunities exist for direct-to-LTO-5 tape recording. Users can capture 55–60 hours of video, depending upon overhead. The actual storage hours are affected by tape drive starts and stops, a direct offshoot related to the number of individually recorded segments per tape.
LTO overhead factors increase when using LTFS as the file system must partition each LTO cassette, thus consuming a bit more space in which to put file index pointers and metadata specific to the file system. The remaining processes of transcoding, storing and playout complete the workflow—each step of which adds hours to getting the work completed.
For a couple of years, users have been able to record direct to LTO-5 tape through “VTR-like” platforms that mount one or more linear data tapes. The platform may also include integral hard drives or can attach to external drives for editing purposes.
Fig. 2: New Production Workflow: Multifunctional systems—sets of hardware, software and media components—condensing
production services, while increasing flexibility and reducing costs
(Click To Enlarge)
Users can now simultaneously archive raw content while caching the same content to integral or external drives. These workflows allow content to be immediately archived and protected, while the data wrangler or director of photography reviews and makes selections directly from disk cache. Rejected takes can be eliminated from the HDD without the risk of losing the original content.
The hard drive recording can then be transcoded into proxy and transmitted to the cloud or a local wireless network for tablet viewing by anyone connected to the network and running appropriate, secure applications.
GAINING MOMENTUM
The concept of integrating nonlinear capture, linear archive and immediate transcode— all from a similar platform—is gaining momentum in the production community. Getting the content stored, protected and approved as close to the shooting time as possible improves location budgets and speeds up many previously disjointed processes that occurred only after the crews or talent left, the sets were struck and the ability to reshoot is lost. Fig. 2 shows a current approach to digital workflows using multifunctional components providing simultaneous functions.
Another step available in new storage solutions is the replication of content to ruggedized, secure and transportable disk packages. Ruggedized and transportable drive sets have been around for at least five years; however these disk packs now include RAID capabilities and secure USB keys, thus promoting both the transport and protection of sensitive content. When using a portable RAID system, bandwidth (i.e., data throughput) is an important issue, so these transportable drive packs should be configured for “audio/visual” uses and not necessary for just simple “backup” operations.
Ideally, a secure, transportable storage system should allow for real-time recording direct from the camera or at least from an HD-SDI connection through a server/codec. Packages should allow for various RAID levels including 0, 1, 3, 5, 10 and JBOD. Hotswappable components (dual power supplies, removable drives, etc.); multiple I/O configurations (eSATA, Thunderbolt, USB 3.0, etc.); and a graphical user interface for direct data management and metadata entry are advisable options.
The last issue deals with transcoding. If, during the production process, workflows allowed for proxy creation; conversion to a mezzanine format (e.g., to produce an XDCAM version from a ProRes native format); and generated a branded image with integral burned in timecode, one would have a complete package that was essentially ready for post production very shortly after the director calls “That’s a wrap.” Such ideas are now a reality, made possible by established manufacturers as well as the private development of production service organizations specializing in digital file-based workflows.
Karl Paulsen (CPBE) is a SMPTE Fellow and CTO at Diversified Systems. Read more about these and other storage topics in his book “Moving Media Storage Technologies”. Contact Karl atkpaulsen@divsystems.com.
