Introduction
Video workflows are very different than the enterprise applications that IT organizations are accustomed to deploying and supporting. Large video files and demands for fast access and real-time performance result in requirements for highly scalable storage systems with enormous bandwidth, consistently low latency and the ability to effectively support highly specialized video applications.
Attempting to force-fit traditional enterprise storage into a video workflow introduces the risk that the IT organization will have dissatisfied users unable to work trouble-free with the tools of their trade. Further, IT may face support challenges resulting from attempting to deploy storage where it simply was not designed to go.
This paper is intended to introduce IT staff to the unique nature of video applications and workflows, and to illustrate the unique requirements of shared storage for video workflows.
The Strange World Of VideoWorkflows
In the enterprise world you may be accustomed to focusing on applications, such as CRM, ERP, and email, and core elements such as databases and virtualization technologies. Though unstructured content is gaining in importance, in the enterprise world data is often block-oriented, and the relevant performance measure is frequently transactions per second.
The world of video is different. In the video world the focus is on workflows, where various specialized video applications are utilized almost like stations of an assembly line to process and distribute video content. In such an environment, it is critical to quickly move large video files between different systems in the workflow. Thus the key performance metrics are around latency and bandwidth, not transactions per second.
Years ago, video workflows were based on analog media: moving images were captured on film or analog video tape, and clips were physically spliced together to create new film or video materials in final form (e.g., TV shows, movies, commercials, etc.). In recent years, film and video have gone digital, and media production workflows have increasingly become file-based. As with many IT applications, the video processing infrastructure may initially use islands of Direct Attached Storage (DAS), but with workflows requiring that video files be shared across different applications and by different users, the ideal approach is to have high-performance shared storage at the heart of file-based video workflows. Figure 1 shows a generic video infrastructure to support a file-based workflow, in this case for a broadcaster.
Fig. 1: Example Video Infrastructure For A File-Based Workflow
As shown in Figure 1, there may be many elements in a file-based video workflow, and the key applications may include:
• Ingest, which refers to taking raw video content that may have been shot with a TV camera, and converting it into video files that can be digitally processed.
• Editing, where video clips are cut, enhanced, sequenced, mixed with audio, and further processed into a segment of final video that someone would want to watch, like a TV show or movie. Key applications here are products like Avid MediaComposer, AutoDesk Smoke, Apple Final Cut Pro, and Adobe Creative Suite, all of which typically run on very powerful workstations.
• Transcoding, where video files are transformed from one format to another for editing or viewing on different types of displays (e.g., TV, iPad, iPhone).
• Playout, where video files are distributed over satellite, cable, or the Internet for viewing by the intended audience.
The unique storage requirements for video workflows
There are unique, challenging requirements for shared storage to support a video workflow:
Predictable, real-time performance
Many video productions must run at 24, 25 or 30 frames per second, and storage must be able to reliably support the consistent bandwidth required to deliver these frames to the requesting application without excessive latency. There’s no extra credit for running faster, but running slower so that frames are “dropped” is unacceptable.
High bandwidth
Moving around many large video files can generate enormous requirements for bandwidth. Storage systems for video workflows are often required to support gigabytes per second of bandwidth.
Media application support
Media applications are different, and the storage system that works well with Microsoft Exchange or SAP ERP may flounder with video workflows and ingest, video editing, transcoding, or playout applications.
Effectively support hundreds of terabytes of large files
General-purpose IT storage is often optimized for transaction processing performance with small files, in environments where storage capacity is measured in terabytes or perhaps tens of terabytes. In contrast, throughput-oriented video applications use large files and storage capacities measured in hundreds of terabytes or even multiple petabytes. This different type of workload can put unique strains on shared storage.
Continuous operations
If storage is down, the workflow grinds to a halt. In the best case this means that a lot of skilled people have to take a long break, but in the worst case it means the TV broadcast becomes white noise, the news program can’t go on, or the video that thousands or millions wanted to watch online is not available. Shared storage for video workflows is mission critical: if the data is unavailable, revenue is lost.
Easy scaling
The ongoing rollout of HD and now 3D content is just the latest in a continuing series of advancements in video technology. Though each advance provides a better viewing experience, each advance also results in yet another type of video that needs to be produced, with larger video files to process and store. As these continuing advancements are adopted, requirements for bandwidth are continually growing within a video workflow, such that it is important that the storage system’s bandwidth can easily be increased without disrupting users or applications.
Ability to edit growing files
Some workflows have very specific, and very critical, functionality requirements. For example, when broadcasting live events such as sporting events, the ability to create a highlights package while the event is still ongoing is critical, and this translates into a requirement for the storage system to be able to edit a file while it is still being written.
Table 1: In Contrast: Enterprise storage vs. video storage
Enterprise integration and other basics
In addition to the above unique requirements for video workflows, the storage system must also meet some of the basic requirements of enterprise storage: • Comply with enterprise requirements for security • Integrate with enterprise management frameworks • Easy to manage • Simple to deploy Cost-effective
Vendor expertise
The unique nature of video applications and workflows makes it particularly important to select not only a product that meets the technical requirements, but also to select a vendor who understands this unique environment, knows the application partners, and has demonstrated success across a variety of demanding video workflows.
Storage Options For Media Workflows
Direct Attached Storage (DAS)
DAS can provide good performance for individual application silos, and offers a low “price per gig” compared to networked storage. However, as in enterprise environments, DAS does not effectively scale, can be subject to downtime, and becomes complex to manage when data must be shared. Though DAS is sometimes used to support an individual application in a video workflow, it is not a practical choice for scalable shared storage or fluid workflows.
Fibre Channel (FC) SANs
FC SANs with an integrated SAN file system can provide the levels of performance required by video workflows, at least initially, and are a viable option as centralized storage in a video workflow. Unfortunately, the predominant file systems used with FC SANs suffer from performance degradation due to file system fragmentation; SANs can be problematic to scale; and often most importantly, FC SANs are complex and expensive to deploy and manage.
Traditional NAS systems
Traditional NAS systems consist of two controllers (or “NAS heads”) in a failover configuration. The controller hardware defines the performance of the system and usually becomes a bottleneck for bandwidth: once a controller is “maxed out,” increasing performance requires either a “forklift upgrade” or a manual re-distribution of the workload across multiple systems. This fundamental limitation in scalability often makes traditional NAS systems a poor fit for demanding video workflows.
Clustered NAS systems
To address the scalability limits of traditional NAS systems, vendors have developed clustered file systems which can effectively join many controllers together and present them as a single image. If additional performance or capacity is required, then additional controllers can simply be added to the configuration. The scalability of clustered NAS systems makes them a viable candidate for video workflows.
Like their predecessors, clustered NAS systems are accessed via standard NFS and CIFS protocols. These NAS protocols can present a performance limitation that is insufficient for some video workflows, and make it very difficult to ensure consistently low latency. In addition, many clustered NAS systems are plagued with an inability to maintain consistent performance as the file system fragments.
Omneon MediaGrid
Omneon MediaGrid, a shared storage system by Harmonic, builds on the clustered NAS approach with a system designed specifically to meet the demanding requirements of video workflows. Like clustered NAS systems, MediaGrid uses a clustered file system to scale performance while maintaining a single system image, and can be accessed with standard CIFS and NFS protocols. However, MediaGrid goes a step further by being able to deliver higher performance via a File System Driver, which provides power users with true direct parallel access to multiple storage nodes through multiple Ethernet connections.
MediaGrid:
• Is built specifically for the unique and demanding requirements of video workflows
• Delivers higher bandwidth and superior price/performance compared to clustered NAS systems
• Maintains consistent performance over time, with no degradation due to fragmentation
• Consistently maintains low latency, with a tuneable file system and client-side caching enabling near wire-speed performance
• Scales non-disruptively to tens of GB/s of bandwidth and petabytes of capacity
• Is based on pervasive and simple Ethernet topology, providing the performance of SAN with the simplicity of NAS
• Is built for high availability, with no downtime required for hardware failures, software upgrades, or maintenance activities
• Has been tested, optimized, and proven with dozens of the leading video applications
• Is offered by Harmonic Inc., a leading provider of video infrastructure solutions with extensive experience in many of the world’s most demanding video workflows
About Harmonic
Harmonic provides infrastructure enabling content and service providers to efficiently create, prepare, and deliver differentiated video services for television and new media platforms. For additional information see www.harmonicinc.com.
