Data storage and its management directly affect workflow efficiency and the bottom line. Demands for storage continue to increase, even as the technological complexities create more headaches. A storage strategy is critical because, of course, without efficiently organizing, storing and monitoring data, we don't have a finished product.
With regard to media files within a production environment, the old adage, “You can never have enough storage,” is true enough. However, with technical advancements in storage hardware, along with an array of options, there is no single approach to storage needs.
For example, at Digital Jungle we are constantly integrating multiple platforms within our facility. Moreover, many of our digital intermediate (DI) projects are in a true 4K workflow with deliverables in 2K digital cinema, HD, IMAX and 35mm film. Any single 4K long-form project or feature with all of these deliverable requirements generally requires massive storage (100TB or more).
Making it work
A traditional and popular method to link all of the workstations together would involve an online centralized SAN storage system. SANs are complex and can quickly become unruly without precise calculation. Many companies call for an offline-shared environment with one or two stations being able to give off the final product. This requires guaranteed data at all times and a large infrastructure.
A typical SAN solution requires additional headroom storage beyond actual storage use. When calculating your data requirements for the SAN, take the number of stations and multiply that by the number of video streams each station is going to play back simultaneously for the offline systems. Then add your online stations to the data requirements. (Each one of your online stations is only going to play back one stream of video at a time, so calculate at the highest possible rate to insure reliability.)
Smart and dumb storage
There is also a different line of attack. To efficiently deal with storage, you can split your storage inventory as either smart or dumb. Some drives are made to act as smart online storage. They know which files and sequences are identical, eliminating wasteful duplication. No matter how many timelines or versions are created, these storage systems are smart enough to store only what is absolutely necessary.
Dumb storage, on the other hand, can be used for offline RAID fiber storage. It is relatively cost-effective and has a very fast 4Gig throughput. (We can transfer 10 to 12 DPX files per second at 4K resolution to or from our RAID system.)
It's called dumb storage because it simply stores precisely what it's given. This storage hardware acts as an archive. It stores not only the media, but the entire project as well. With a PC server and RAID storage, you can off-load projects to a client's FireWire drive or store media and data for future use.
Using a fiber switch to connect multiple RAID storage systems to smart storage allows broadcasters to effectively share footage and data across all platforms. Adjusting the ratio between smart and dumb storage is a constant balancing act — based on current project needs and projecting storage needs that are in the queue.
While having all projects online all the time is ideal, it is not very cost-efficient. For now, a hybrid storage scenario works well within most workflows. And here, it's important to remember that it's not only the amount of storage that matters, but also the type and how it's managed that is crucial.
Last but not least, we must always remember to properly budget disk space for all of our projects. First, what are you going to edit with? No matter what format you start on, you have to choose a codec that is going to fit well within the budget.
At the moment, 4K is the biggest hog of disk space, which is why most editors choose to convert it into a more manageable format like HD.
If you shoot SD DigiBeta, you might consider DV as your offline codec. Either way, you need to know the amount of media that has been shot in order to have enough disk space while editing.
Once you know your compression, find out how many megabytes per second it takes to sustain that particular compression. Take the megabytes per second, and then multiply by 3600. You now have the approximate baseline total of disk space you are going to consume for the project.
An easy way to get the baseline total is to use the AJA Data Rate Calculator Application Version 2. (Visit http://tinyurl.com/45x3fm to download the TAR file.) This will quickly show you how much data will be consumed per hour. Remember, this will not tell you the complete storage requirement, but it is a good quick reference tool that is easy to use.
Dennis Ho is president of Digital Jungle Post Production.
