Today’s media companies are increasingly being put under the pressure of higher performance demands, specifically the demands of 4K, 8K and even higher-resolution video formats, as well as the cost of housing ever-growing content archives. Adding to this strain, media companies have to manage and leverage growing data sets faster than ever before, to both quickly satisfy new business requirements and adapt to an increasingly competitive industry overall.
To manage these growing performance requirements of production applications—from rendering and editing to transcoding—and the requirement to store constantly increasing volumes of data, the traditional response has been for studios to build multiple tiers of data storage. With this approach, each tier is designed to address the performance and cost requirements of some set of applications in the pipeline. In fact, we as an industry have been building these storage pyramids so long that storage hierarchies have become entrenched as the doctrinaire way to manage data for media.
However, this approach also adds unnecessary complexities and costs for media companies. As these performance demands only grow for media companies, so do these complexities, costs and overall roadblocks that stand in the way of further digital transformation.
So what are the steps that media companies can take? It starts with truly understanding the shortcomings of sticking with the status quo for managing data, identifying the new technologies and approaches that open the door for a different way of thinking, and mapping these investments back to the ever-changing path of media and broadcast’s future.
THE PYRAMID CREATES COMPLEXITY
Having multiple tiers of storage inherently adds complexity in several ways. Most obviously, someone, or more likely some script, MAM or data manager, must migrate assets from one tier to another as they move from creation, through production and into the organization’s archive. While global namespace overlays, automounter scripts and storage management software make these moves more or less transparent to users, the resulting infrastructures are relatively fragile. Even worse, that fragility makes the studio overly dependent on the one or two system administrators who really understand how they work and can patch them one more time when something breaks. This dependency only causes inefficiencies and bottlenecks in the overall media workflow.
In many media organizations that complexity is amplified by their use of specialized storage solutions at some stages of their pipeline. While media-specific storage solutions like SAN file systems have historically provided better performance, they add complexity with client agents. Similarly, application-specific storage may support proprietary APIs that may improve collaboration, but using dedicated storage for specific applications just increases the number of storage systems, data movement scripts and independent pools of free space to manage.
The pyramidal storage hierarchy only really makes sense when the cost differential between the tiers is large enough to offset the cost and complexity of migrating data between them, and the slower tier is still fast enough to support those applications that access data there. But as users’ storage performance requirements and the demand for online content rise, and the performance of traditional disk storage actually falls, the old reliable storage pyramid is starting to crumble for media organizations.
CHALLENGING THE PYRAMID
We’ve seen some people advocating for a “two-tier” approach to data storage, with active and permanent tiers. While two tiers are certainly simpler than four or five, data still has to be moved from the active tier to permanent tiers and back as needed. While media asset and storage management systems claim to make “intelligent” decisions when placing data, some data will inevitably be misclassified to the permanent tier and will need to be retrieved on demand for users, slowing their work.
Recent changes in media workflows and storage technologies challenge the underlying premise of tiering in the first place—that the total cost of multiple tiers will be lower. A new generation of all-flash storage systems revolutionizes flash economics by bringing the total cost of storing all of a media organization’s online assets, from the most active to archive in a single, simple-to-manage namespace, below the cost of tiered storage.
By combining the performance demands of an organization’s I/O-intensive applications with the capacity demands of their archives, these systems can manage lower cost flash to satisfy the cost, performance, resiliency and endurance requirements of the most demanding media organizations.
Storing active and archival data on fast, all-flash storage not only eliminates all the complexity of moving data between tiers but also makes all that archival data available for further reuse. For example, rather than relying on decades-old metadata, organizations can unleash “GPU farms” to use artificial intelligence and recognize artists, logos or other images in their libraries for further efficiencies.
By switching to all-flash storage infrastructure, media organizations can deploy a single system for everything from special effects rendering to feeding their streaming services and archiving their most valuable media assets in a platform that delivers unmatched benefits.
ROADMAPPING THE FUTURE OF MEDIA
With growing competition as major broadcasters expand their offerings to the streaming world, rising demand for a wide range of content (especially as consumers spend more time at home amid the COVID-19 pandemic) and media organizations looking to drive the bottom line and produce a strong ROI throughout these increased investments to satisfy such demands, the need for efficient and effective infrastructure is at an all-time high.
While recent months may have served as an accelerator for the demand for high-quality, data-intensive applications and content, this demand will only grow as competition in the space does as well. This means no stone must go unturned for media companies as they both evaluate and reimagine their workflows for the modern media and entertainment industry.
As such, media organizations should explore the next class of storage systems. If an all-flash storage model doesn’t cost more than a multitiered pyramid and yet still gives you increased performance, why would you continue to invest in the multitiered approach?
Howard Marks is technologist extraordinary and plenipotentiary at VAST Data.
