Media storage is the one component of your system that you want to set and forget. Most of us take our local drives and network volumes for granted. We expect them to just work. When they don't, we quickly become aware that there's a problem as work grinds to a halt. Having fast and reliable media storage is the cornerstone of modern editing, graphics, animation and color correction. For a lot of us, videotape masters and deliverables are long a thing of the past, so a successful storage strategy is integral to staying in business.
TECHNOLOGY PILLARS
Storage technology has advanced over the years, with drives gaining capacity and speed and entirely new storage mechanisms being developed. There are four pillars to consider: solid-state storage, RAIDs, interconnection and shared storage.
Echoing the evolutionary trends of other digital technologies, SSD (solid-state drive) storage devices have been gaining capacity while coming down in price. These days it's quite common to have computers—especially laptops—with internal SSDs. As this trend progresses, the same will become true of local external drives. The majority of editors today are using spinning disk drives for local storage, but that's changing as editors are called on to deal with higher resolutions and frame rates.
RAID 5 writes data across all hard drives in the array and a parity block for each data block. If one physical hard drive fails, the data from the failed hard drive can be rebuilt onto a replacement hard drive. While the files stored on a RAID 5 array remain intact should one hard drive fail, data can be lost if a second hard drive fails before the RAID is rebuilt with the replacement drive. Image by en:User:Cburnett via Wikimedia Commons
If you want the best performance with high capacity, then RAID (redundant array of independent disks) is the way to go. Once expensive and exotic, there are plenty of configurations available today, ranging from a chassis with two drives up to eight and more. Promise Technology, CalDigit, LaCie, G-Technology and OWC are leading RAID storage vendors.
RAID data virtualization combines two or more physical disk drives so they act as a single volume that may be optimized for speed, performance or redundancy. While there are several different RAID levels, you'll typically encounter RAID 0, RAID 1 or RAID 5.
RAID 0, also called "striping," combines two physical drives into a single virtual volume to achieve faster performance. RAID 0 does not provide data redundancy to offset disk failures, so the system is able to use the total capacity of both drives for storage, but if one drive dies, then you've lost all of your media. RAID 0 is optimized for increased throughput and provides the best performance of any RAID level.
RAID 1, also called "mirroring," records duplicate data onto both drives in a two-drive array. If you lose a drive, you have a backup; however, capacity is divided in half. (Half of the capacity is used to store your data and half is used for a duplicate copy.) RAID 1 does not offer the performance boost of striping.
The most common configuration with larger arrays of drives is RAID 5. This level writes parity data across all of the drives so that if one drive fails, the parity data can be used to rebuild the media information when a replacement drive is installed. Nothing is lost. You gain performance and protection, but at the cost of some storage capacity. For example, in a four-drive array, you give up approximately 25 percent of the raw capacity to parity data.
Interconnection technologies have likewise evolved over the years. Fast interconnection between a computer and external storage used to require a separate PCI card be installed in the computer for protocols like SCSI. As more of these protocols were built into the computer itself, the cost has come down, since you don't need to purchase extra hardware. FireWire (a favorite among Mac users for years) is largely gone, supplanted by newer built-in protocols like Thunderbolt 2 or 3, as well as USB 3.0 or USB-C. There is still support for eSATA, too, which may require a peripheral, such as a Thunderbolt dock with an eSATA port.
The last of our pillars is shared media storage, often called SAN (storage area network) or NAS (network attached storage). I won't go into specifics about the differences between SAN and NAS systems, but the simple differentiator is that SAN solutions generally connect to computers using Fibre Channel, while NAS uses Ethernet. Until recently, SAN systems were considered to be faster than NAS, but those waters are muddy today, with NAS configurations being used in installations working with 4K and higher video resolutions. SAN and NAS installations are the best way for multiple editors and designers to access the same media from separate workstations.
HOW LEADING PROVIDERS SEE THE TREND
The film and TV world was first introduced to shared storage through Avid Technology. That company's storage solutions have evolved from the original Avid MediaShare, introduced in 1995, through Unity and ISIS, to the current iterations of NEXIS and NEXIS | Pro. Storage solutions today have to be open to working with many creative applications. Avid's storage, which initially was designed to work with Media Composer and NewsCutter, has been tuned today to also work with Pro Tools, Adobe Premiere Pro, Apple Final Cut Pro X and Blackmagic Design DaVinci Resolve.
Avid MediaCentral now features the MediaCentral | Cloud UX (user experience), a task-oriented graphical user interface that runs on virtually any operating system or mobile device, and is available to everyone connected to the platform, no matter where they are.
Avid storage is still a common fixture in large post facilities and broadcast operations, but use of shared storage solutions is definitely moving "downmarket." Smaller shops with two or more edit stations can benefit from common access to projects and media without the need to "sneakernet" external drives between computers. Thanks to the advancement of simpler systems that can handle the needs of media pros at a lower price point, shared storage is an increasing part of the small and mid-sized business landscape. Avid, too, has recognized this trend, introducing lower-cost NEXIS bundles to attract SMB customers.
"We continue to see strong demand for digital media storage as the industry tries to keep up with viewers' ever-increasing desire for high quality, engaging content," says David Colantuoni, senior director of product management at Avid. "But tech budgets aren't increasing, forcing our customers to look for more flexible, cost-effective media-optimized storage solutions that help them keep costs down. That is one of the reasons Avid recently announced our strategic partnership with Microsoft Azure—to enable the migration of our Media Central Platform into the cloud in order to help our customers become more agile, flexible and collaborative."
The market that Avid effectively created has spawned many competitors at all size and price levels. Naturally the continuing advances, coupled with the oft-misunderstood benefits of "the cloud," can be confusing to customers. Leading storage vendors are attempting to clarify these issues for users in the face of increasingly challenging technology requirements.
EditShare grew from a small creative shop into a leading worldwide storage provider. According to Andy Liebman, EditShare founder and CEO, "Judging from 'requirements documents' we get every day from customers seeking out new shared storage systems, the adoption of UHD/4K in both acquisition and postproduction is now in full swing. Everybody is talking about multicamera editing with data rates between 100 MB/s and 2,000 MB/s per stream—in DNxHR, ProRes, DPX and other high-quality formats. This is a mind-boggling amount of data that has to move in and out of storage in real time—many times what we were dealing with a few years ago in the era of uncompressed HD.
"To meet this new UHD/4K challenge, you need blindingly fast local storage like our high-performance, fault-tolerant XStream EFS scale-out NAS," Liebman continues. "As we have been saying for many years, shared on-premises storage isn't going away anytime soon. Even if you weren't concerned about the security aspects of the public cloud, you couldn't get your editing done just with cloud storage. It's not fast enough for editing this type of content.
"Of course, we realize customers are totally in love with the idea of accessing their content anywhere, and they associate this concept with 'the cloud,'" Liebman says. "And so that's why we promote using our EditShare Flow MAM to transcode all your local UHD/4K content into low-resolution proxies and then expose everything to the internet via our secure On-Premise Private Cloud—so your team can access, view and edit content remotely. It's the best of both worlds: local high-performance storage and remote access that originates on-premise too."
While SSDs can help, just throwing flash drives at the problem won't deliver the performance content creators need for the most demanding projects. For example, OpenDrives manufactures high-performance, purpose-built shared storage products (from all-SSD to all-HDD) for uncompressed 4K, 6K, 8K, HDR and HFR media and entertainment workflows. OpenDrives is used by motion picture studios, television networks, broadcasters, online content creators and advertising agencies. Recently OpenDrives was the production and postproduction storage for Deadpool, Only the Brave and David Fincher’s Mindhunter.
Jeff Brue, OpenDrives co-founder, explains, “At OpenDrives, we created a platform that enables true real-time collaboration among editors, VFX artists, colorists and directors. Our high-throughput/low-latency products allow artists to work in native uncompressed formats. OpenDrives’ application-aware operating system accelerates project load times through intelligent and adaptive pre-fetching for critical functionality like Adobe and DaVinci database files. OpenDrives also significantly increases the number of concurrent users. This empowers content creators to work on increasingly complex projects and seamlessly adopt new features like the shared projects functionality in Adobe Premiere Pro—something that just isn't possible with traditional data storage hardware and software."
For many, the middle approach—a hybrid SSD/spinning disk unit—is an appealing tactic. This technology combines a set of SSDs together with a set of spinning drives in a single rack-mounted chassis. Facilis Technology's TerraBlock shared storage line, for example, includes hybrid product options.
"As the need for storage continues to grow, facility managers are realizing that there are big differences between storage systems when it comes to performance at 4K and beyond, especially with multistream and file sequence workflows," says James McKenna, vice president, marketing and pre-sales engineering. "One of the reasons for Facilis' continued success is that we've found the sweet spot for performance, scalability, flexibility and cost. Our customers tell us we provide the greatest ROI in storage for postproduction, and it's something we're very proud of."
NEW TECHNOLOGY OFFERS NEW OPPORTUNITIES
Avid, EditShare and Facilis are established players in the world of shared storage. But as the number of creative shops has increased with the diversification of media outlets, the storage market has heated up. The launch of Apple's Final Cut Pro X really demonstrated the sector's renewed vigor. LumaForge popped onto the scene to provide easy-to-set-up storage solutions optimized for FCP X and Adobe Premiere Pro.
"Shared storage used to be primarily the domain of dedicated post houses and facilities," explains Patrick Southern, chief workflow officer for LumaForge. "Now YouTubers, in-house content departments, DITs and on-set editorial teams are all using shared storage. We fully believe in NAS and high-speed Ethernet as the future for 4K, 8K, VR and beyond. SSDs are still very expensive compared to spinning disk storage, but they are also much faster. Our Jellyfish line of storage uses both SSD and RAM caching to significantly increase performance. However, as the price of SSDs continues to drop, full SSD systems will become a more practical solution for smaller teams."
Southern continues, "Many people ask us about direct connections via Thunderbolt 3 to our systems. Currently the cable lengths for full-speed Thunderbolt 3 are very limited. Unless you are sitting at the same table, connecting to shared storage via Thunderbolt 3 is impractical. Small teams are discovering that 'sneakernet' is inefficient for multi-editor projects. Now that there are affordable, high-speed shared storage options available, there are fewer reasons to buy a plethora of hard drives."
File sharing via QNAP
As with all electronics in the modern era, Asian hardware manufacturers have made shared storage products accessible at nearly any price point. "I observe a radical departure from conventional postproduction as we have known it in the past," says Bob Zelin, independent engineer at Rescue 1 Inc. "Today, a modern post or graphics facility is simply a bunch of computers tied to a 10G [Ethernet] network switch, which connects to some sort of shared storage system. That's it. And with the recent introduction of Taiwanese network attached storage products from companies like QNAP, Synology, Netgear and Thecus, the price to accomplish this is literally a fraction of what it was just a few years ago. All of these products work flawlessly with modern editing and graphics software from companies like Blackmagic, Avid, Adobe, Apple and Maxon."
These advances are all great news for users. Systems perform better, cost less, and are easier to install and maintain than ever before. Naturally, expectations are going to continue to increase—what was good for HD is taxed for 4K and will be outdated as higher resolutions become the norm. While we have not achieved the golden triad of good, fast and cheap, modern storage solutions are about as close as we can possibly come.
This story first appeared on TVT's sister publication Creative Planet Network.
