Few things can create data bits more rapidly than a digital video camera. And storing those bits can be like drinking from a fire hose. Storing and recalling even highly compressed video involves data rates of millions of bits per second. And, even in error-free channels that don't need error-correction overhead, the video, audio and, increasingly, the metadata about both are dense media that must be compressed for practical storage.
High flow rates
Video storage serves many purposes, including original recording, play to air, production, archive, search, on-demand usage and delayed usage by consumers. Each purpose spawns its own set of technical and economic requirements, costs and approaches. Archived video, for instance, can involve video at several quality levels. But there is always a very high-quality version that is archived so it can be used in the future for other purposes. At one end of the quality scale is HDTV at full-bandwidth 4:4:4 sampling, which is appropriate for high-value content that may be repurposed for broadcast, theatrical or other uses in the future. This video (SMPTE 372M-2002: Television — Dual Link 292M Interface for 1920 × 1080 Picture Raster) requires a total data rate of 1.8Gb/s for the picture alone. Sixteen tracks of uncompressed AES audio add about 2.5 to 4 percent overhead, and metadata add another, smaller amount. Thus, today's highest-quality uncompressed signals (1920×1080p30) flow at a rate of approximately 2Gb/s. Moreover, 60p systems are developing for future use. Such systems will roughly double the data rate to 4Gb/s, assuming similar audio and other overhead.
Slowing the flow
Sony and Panasonic have developed high-quality compressed recording formats that allow lower data rates. Sony's HDCAM-SR, which uses MPEG-4 Studio Profile compression, records at a data rate of 600+Mb/s. Panasonic's HD-D5, which uses a proprietary DCT-based compression process, records at 360Mb/s. But even at those lower data rates, the firehose creates huge puddles of bits. In the 4:4:4 mode, HDCAM-SR records 3.2TB/hour (uncompressed, the same signal would require a whopping 6.7TB/hour — not exactly what you normally have laying around for long-term archive). Using HD-D5 compression would lower the total storage to 1.3TB/hour for 4:2:2.
Storage gets easier for lower-speed encoding rates. For instance, DV recording (25Mb/s) requires 90GB/hour, which is just 1.3 percent of the uncompressed 4:4:4 example above. Clearly, DV is much more friendly to storage. Dropping to a lower data rate, which might be practical soon using the recently standardized MPEG-4 Part 10 AVC codec (H.264), would allow recording standard-definition video at about 2Mb/s, with a total storage requirement of 7.2GB/hour. This is three orders of magnitude less than uncompressed 4:4:4 video. At that rate, laptop editors might hold 20 hours of high-quality video, and on-line video servers could hold 500 hours at modest cost.
Weighing the variables
We can consider the problem of storing video as a simple equation with variables such as data rate, cost of storage (E/bit), transfer rate and maximum capacity. Different media will offer different solutions to the equation. For instance, DVD RAM cannot sustain the data rate needed for uncompressed standard-definition video, let alone HDTV.
Nonetheless, at high compression rates, even standard DVD can play HDTV at perhaps 20Mb/s to 30Mb/s. But, since write speed is an issue for DVD, a buffer is usually required to allow high-data-rate recording. DVD offers the advantage of portability, low cost and small size, which magnetic drives cannot match. On the other hand, magnetic disk drives have significantly higher writing speed and can handle the sustained throughput necessary to support even uncompressed video storage (where striping or other techniques may be necessary). Magnetic disks are inexpensive; 300GB drives in consumer computers is not uncommon. Using just-a-bunch-of-disks (JBOD) approaches, storing large amounts of video with suitable redundancy planning can be very affordable indeed.
Something old, something new
Videotape is certainly not yet dead. It is cheap, ubiquitously available, portable and highly reliable, not to mention quite familiar. But, when compared to optical and other techniques, it suffers from limited shelf life and long seek times that make nonlinear read/write impractical.
Last year, in reaction to the limitations of tape, Sony introduced the XDCAM series of optical recorders for news gathering. XDCAM system uses blue-laser media, a rewritable 5-inch disk in a protected cartridge with a storage capacity of 23.3GB per disc. Sony says that the heat-resistant medium is designed to record, erase and re-record more than 1000 times and to read the written data more than 1 million times. The XDCAMs can record either MPEG or DV; this flexibility facilitates interfacing XDCAMs with widely available nonlinear news-editing systems.
Panasonic countered at NAB last year with the P2 recording system, which uses non-volatile memory based on compact flash memory embedded on PCMCIA cards. The P2 camcorder has no moving parts, uses little power and has extremely high write and read speeds. Currently, the memory cards have limited recording capacity. But, as memory capacity increases, so too will recording time. Eventually, it may be a viable solution for in-camera HDTV acquisition. Panasonic has also worked with third-party vendors, allowing its system play nicely with nonlinear editing systems. Both Sony and Panasonic expect to be delivering their respective products in quantity this spring and summer.
We should emphasize that these two new recording techniques from Sony and Panasonic are simply two additional acquisition media; they do not replace videotape. Tape still has longer recording capability and higher data rates as well. One Sony field digital tape recorder (the SRW-1) can record the output of two full-bandwidth cameras on one tape simultaneously. Two-camera shoots with one tape are an interesting possibility, as is the ability to create 3-D images with one recorder.
Progress
The progress of storage technologies is a continuum with no defined target point. High density, low cost, high reliability, and fast read and write times are all needed for high-quality video applications. DVD and, especially, blue and blue/violet laser approaches, are rapidly advancing on the turf of moving-tape and rotating-disk approaches. In the last five years, we have moved beyond CD-ROM jukeboxes to DVD robots. In the past year, some facilities have deployed large-scale implementations that use a hybrid of approaches to achieve a total system architecture that is innovative and highly reliable.
Moore's Law predicts increasing capacity and decreasing cost of storage. But, due to limitations imposed by the laws of physics, the curve that plots Moore's Law will eventually flatten out. As the spatial density of storage on hard disks increases, the size of an individual memory “cell” approaches the limit of physical processes required to record and play back information. At some point, quantum effects take over and bits become probability equations, and producers would likely not accept a “reasonable probability” of successful recording. Nonetheless, modern engineering has advanced recording techniques much further and much faster than we could have predicted a few years ago.
One network's storage solution
Turner Entertainment Networks has implemented a video transport and storage system that uses a hierarchy to move content seamlessly from tape and electronic delivery to air and archive. It is based on Pinnacle Systems servers, DVD robots, EMC high-bandwidth storage and Storage-Tek tape-archive robots. The concept is simple and elegant. Content is delivered electronically, if possible, to edge servers (for instance Pathfire or Media DVX).
Under the control of Turner's custom broadcast-inventory-management (BIM) software, or the company's Pro-Bel automation system, ingest servers ingest the content. BIM stores the content on redundant EMC storage arrays and then transfers the media to air servers, DVD archive robots for spot library or StorageTek PowderHorn archive tape robots. When content is needed for air, the automation system(s) send(s) a request to BIM, which moves a copy of the content to the air and back-up servers. When content is ingested directly to the air servers, it is scavenged back to BIM and its hierarchical storage for long-term archive.
Figure 1. New optical storage techniques such as holography are being developed to offer long-term stability and reliability for video archiving. Click here to see an enlarged diagram.
Turner uses a variety of storage approaches, depending on how long the content will be stored, how it came in and when it will be used. It uses the most appropriate technology for each situation and manages the multiple instances to ensure that all content is available at all times. Users care only that the content is protected and available at any time. The software and hardware system manages the rest of the process entirely in the background.
Holographic storage
Other, newer video-storage options are moving towards practicality. InPhase Technologies has developed an approach to optical storage that uses holography to record extremely high-density data on small cartridges. (See Figure 1.) By 2006, the company plans on delivering 200GB of storage on removable media with a write speed of 20MB/s and a sustained transfer speed of 160Mb/s. Such total capacity is not sufficient for studio or field recorders. But, in an archive system, it might offer long-term stability and reliability and be attractive indeed. In principal, data recording does not require rotating media. Thus, a very simple mechanism could be developed to record data on media the size of a credit card. Other vendors are developing optical techniques that will move well beyond hard-disk recording in the next generation of storage products.
Shrinkage
Storing bits is like operating any complex filing system. The size of the container must fit the application. And the more files you have, the bigger your cabinet gets. But, as we have seen, the files and cabinets are continually shrinking.
John Luff is senior vice president of business development at AZCAR.
