Karl Paulsen
To meet an ever-expanding data storage agenda, the industry continues to create higher capacity hard drives to support larger pools of storage. Beyond those tangible solutions, familiar companies from Microsoft to Cisco to Amazon sell storage space for users to put their information into cloud-based repositories of an unknown scale. Entire industry sectors, not just broadcast, media and entertainment, encourage drive manufacturers to throw fuel on this uncontrollable fire.
Today’s 1 TB drive at less than $60 is a perfect example. Marketing often exemplifies drive capacity in terms of the number of photos or hours of movies that can be stored on its drive (usually with no other specifications and presumably not at full uncompressed HD data rates). As capacities are extrapolated to the 3, 4 and more TB drives, one can quickly see how this continual quest for data storage is being satisfied, at least near term.
CAPTURING DATA
For news and electronic field production, most video and audio is captured to either solid-state or spinning media, predominantly optical, with tape continually evaporating from existence.
For ENG/EFP, portability, capacity and transfer speeds are prime drivers for storage systems. For motion picture production, the capture medium is generally a portable disk drive set, solid-state memory or a linear tape format of some flavor. Integrity, security and transportability are this market sector’s key importance drivers.
How these two industries store their content, for the short and the long term, continues to evolve as technological advances address each of their driving factors.
Over the years, high-end tape formats have boiled down to a select few, i.e., Sony’s SRW for HD-video capture or the Linear Tape Open format known better as LTO for mezzanine level JPEG2000 or DPX, and as a digital intermediate format per SMPTE 268M-2003. Major studios and program content producers are gradually shifting to the proposition of accepting digital linear data tape as their “master-of-choice.”
File-based workflows bring us a plethora of video formats. In turn, they have spawned a number of acquisition methods that depend on a growing variety of media types. This diagram conveys a sampling of devices and their associated storage mediums. For non-real-time production activities, the LTO solution (with or without the LTFS option) is a suitable solution for carrying original content from stage to first-level system import. Alternatively, for field production, memory card-based formats (e.g., P2, SxS, QXD, SDHC, etc.) make an argument against the optic-disc alternatives. Essentially, the choice becomes a house-format preference bounded by choices in video compression formats that are supported on that specific medium or platform.
For real-time operations, a dedicated professional tape-based transport with sufficient bandwidth to record 1080p60 (or greater) content still seems to be accepted. Drive packs (portable RAID arrays or OEM packages of small JBOD stacks) provide the ability to replay on set and to rapidly dump files from the field state to archive state.
For portability, tape ensures that traditional familiarity and robustness remains— so long as the media and the transports are both supported. For other than disk-based solutions, the SRW platform remains for those high-end, high-bandwidth motion picture productions. For news and general field production, choices vary depending on the video encoding format, whether or not you need to transfer the content to another storage platform, or if you have unlimited budgets that let you keep everything captured on original media for eternity.
The real-time emphasis is important as it further defines the workflow and structure for the methodologies onto which content is first captured, transferred or utilized in the production process.
DIFFERENT RULES APPLY
News has an entirely different set of time constraints compared to commercial or motion picture production. The rules apply differently to the environment for which they serve. LTO for news would be an impractical solution, yet LTO makes great sense for the retention of content as it moves from camera-based media to long-term archiving or mass portable storage.
The evolution of solid-state storage (non-volatile memory or “NVM”) has helped manage both the risks and limitations typically associated with linear tape. NVM resolves the linear approach of videotape by providing random access to file-based content stored as video.
NVM is robust, less likely to be erased or damaged by external forces, and has a direct transfer (copy) rate that exceeds most other solutions. NVM has become affordable to the point where the advantages of size, robustness and flexibility for field capture and transfer for editing (or to archive) becomes practical for most high-resolution formats.
From a total cost of ownership perspective, the expense of media replacement, mechanical transport longevity, updating and maintenance and the associated licensing costs for archiving allows NVM to become a default method for field image capture and first phase initial storage.
A drawback to NVM-based solutions for broadcast production and post comes from the general need to transfer original field content from NVM media to a secondary medium; similar to the days of videotape-based ENG and EFP when tape was a controlled commodity.
Videotape took up real estate, was generally improperly cared for and was used over and over again until it degraded or wore out.
NVM can initially be more costly and a little harder to manage unless you rigidly adhere to a universal metadata structure that is embedded into the files. Because NVM is physically smaller, it may easily be prone to loss. Hoarding of NVM-cards is simpler than tape, although easier to track. NVM is also plagued with a similar record/playback issue, that is, you need an expensive interface to read or write the data, at least for most professional NVM-card solutions.
Many vendor solutions and workflows are available that let users turn the NVM-media around quickly. Depending upon the manufacturer of the camcorder or field recorder and the post-production systems, solutions can begin right at the point of image capture.
Simpler solutions involve collecting timecode, shot data and device identifications directly into the file. MXF provides a standardized approach to the carriage of that information, but lacks the commonality or specifics of what (metadata) is contained in that data set.
Some of the more formidable solutions involve GPS data to identify time and exact locations or linkages to scripting on secondary tablet-based producer interfaces allowing annotation and scene selection in real time. Automatic simultaneous generation of proxies, which are radioed to a laptop or returned to the station (via Wi-Fi, microwave or satellite) let producers, journalists or editors get a jump-start on the post process well ahead of the actual high-resolution content transfer process.
The latter approach requires a commonality of the entire camera, editorial system and newsroom computer system. Ultimately this provides the organization with an integrated end-to-end workflow that can turn the NVM-media cards around in record time; as well as speed up the story-generation process immensely. This truly becomes a real “file-based workflow” that covers the concepts and essentials up to the point of archiving—making better use of NVM or other disc-based forms of media.
With this background and overview of how we got to this point, next month we’ll dive into the methodologies for moving content from the capture state to the archive state. The subject of archive will address both the caching or near-term storage of content and the issues associated with the long-term preservation of assets including media cataloging processes and partial or complete restoration of content when those workflows are necessary.
Karl Paulsen, CPBE and SMPTE Fellow, is the CTO at Diversified Systems. Read more about other storage topics in his current book “Moving Media Storage Technologies”. Contact Karl atkpaulsen@divsystems.com.
