The accelerating adoption of digital, file-based infrastructures continues in the broadcast world, forcing the need for more storage capacity and effective storage management. Digital storage silos — independently serving post-production, on-air playout, graphics and newsroom systems — are becoming a common solution to fit these unique workflows, despite the obvious disadvantages. Effective unification of these distinct digital silos is necessary if a facility's storage and content management capabilities are to facilitate next-generation, file-based collaborative workflows.
To put this into context, consider a traditional network file server. It would be unacceptable for the IT department to mandate that all word processing documents be stored on one server, spreadsheets on another, GIF images on a third and so on. But in a broadcast environment, this is pervasive and considered an acceptable practice. It is time to look at server storage differently.
The file-based environment
A file-based broadcast model is comprised of three necessary layers (as shown in Figure 1 on page 70):
- digital broadcast devices, including video servers and newsroom and editing systems;
- physical storage infrastructure, including a storage management system, high-speed networks and a mix of disk storage arrays, data tape or optical libraries; and
- control systems, which are the user-facing applications, such as broadcast automation, media asset management (MAM) or less expensive content management solutions that help users to facilitate content identification, use, reuse and collaboration.
These layers work in tandem to facilitate the end-to-end, file-based digital content workflow. At the heart of this concept is the handling and management infrastructure or, simply, the storage management system. These systems can include direct connectivity to various broadcast devices and provide intelligent physical storage management and abstraction. Advanced solutions can also facilitate digital content repurposing via time-code-based partial restore, high- and low-bit-rate content transcoding for tapeless interoperability between systems, as well as automatic site-to-site content distribution and replication for disaster recovery.
Most broadcast storage infrastructures feature four distinct tiers: online, near-line, archive and offline. The major benefit of this tiered storage model is that storage cost decreases significantly as content migrates from online to near-line, from nearline to archive and, finally, from archive to offline storage. (See Figure 2.) Each tier of storage also provides certain workflow advantages and disadvantages, which must also be taken into careful consideration during the planning stage.
The storage management system is ultimately responsible for the effective management of these storage tiers. There are currently two types of storage management systems often confused. They are hierarchical storage management (HSM) and media storage management (MSM) systems. Both share similar names but offer fundamentally different functionality because of their origins.
HSM migration
HSM systems were developed to address storage infrastructure management in traditional IT environments. HSM systems generally manage the migration of any type of file to and from different storage tiers using file-based rules or migration policies. (See Figure 3 on page 76.)
Once the administrator defines these policies, the HSM software manages migration between storage tiers automatically with little or no interaction. These HSM migration policies can specify the movement of a file between storage tiers when it has not been accessed for a certain period of time, has a specific file extension or meets other general parameters.
Watermarking is another migration policy typical to HSM systems. With it, the free capacity on near-line storage is monitored. When it reaches a certain threshold, seldom-used files are transparently moved to archive storage, freeing space on the near-line tier. These HSM policies typically mirror simple business practices where files become less relevant as they age.
The HSM software typically runs on the same single server that hosts the network shared drive, which it monitors for files that should be migrated to different tiers based on configured policies. For example, an HSM application can run on a network server monitoring a shared drive (near-line tier) and migrate content to a data tape library (archive tier) as additional space is required on the network drive or a particular file has not been accessed for several months.
HSM disadvantages
The monolithic nature of the HSM environment guarantees significant downtime for upgrades and greater susceptibility to catastrophic software and server failure. Also, these systems are intended for cost-effective management of large volumes of data and don't fit dynamic content workflows common in broadcast applications. HSM systems simply move files to less expensive tiers as the configured migration policies are satisfied.
In order to perform content migration, HSM systems use stub files to trick applications into thinking the original file is still present on a particular tier of storage. This stub file is typically a small fragment of the original file, containing pointer information to where the actual file has been moved to in the less expensive storage tiers. If the HSM system migrates files to these other tiers of storage without leaving a stub file behind, applications would simply believe the file no longer existed and have no way of retrieving it.
When the HSM system receives an access attempt on a stub file, it will ask the requesting application to wait while it retrieves the remainder of the file from other storage tiers. The operating system on the client computer considers the entire contents of the HSM system to be online, unaware that the bulk of the files may reside on other storage tiers.
This fundamental HSM mechanism can be problematic in a broadcast environment. Leaving stub files on online storage (video servers, newsroom systems, etc) is simply not possible. If a broadcast automation system believed a particular commercial existed on a video server because it found a stub file in its online storage, on-air disaster would likely follow. Automatic file migration or movement to other storage tiers can also cause unexpected results when, for example, the next commercial to play on-air is migrated by the HSM policy engine because it has not been accessed recently.
For these reasons, typical HSM systems do not directly interface to online storage. They instead rely on introducing yet another layer of control to copy or move content from online to near-line storage where the HSM system can then take over the simple migration process. Not only does the additional control layer add complexity to the overall solution, but it mandates that its provider develop many proprietary broadcast device interfaces to support the necessary workflow. Because this development is not typically the primary focus of the provider, it can also limit the customer to a non-ideal and somewhat convoluted workflow.
Unfortunately, these are not the only potential pitfalls in selecting an HSM solution. A side effect of the storage abstraction provided by the use of stub files is that control systems have limited visibility into where files are actually stored at any point in time. Also, HSM systems typically provide limited mechanisms for request prioritization. The combination of these factors eliminate the potential for deterministic system behavior, which is an absolute necessity in broadcast applications.
Another significant limitation is that HSM systems deal with files independently because of their IT-centric approach. They are not capable of grouping related component files together to form compound objects.
The ability to group content is a requirement when dealing with a media asset, which might consist of a header file, several audio files, a VBI file, a digital video file and more. These independent files need to be treated as one single element as they flow through the storage infrastructure.
Imagine a situation where a movie is required for playout to air. Its audio and VBI files are in near-line storage, but its video and header files sit on different pieces of media in the archive library. This can present a potentially disastrous situation.
Although rarely supported, offline storage can present additional issues because of the randomness of files contained on each piece of archive media. Administrators would be constantly shuffling archive media in and out of the library as users requested access to their files.
Considering these significant issues, HSM systems are condemned to provide limited value and leave us far from the goal of having a truly unified broadcast and media storage infrastructure. Thankfully, there is an alternative.
MSM basics
HSM and MSM systems are similar in that they can migrate files from one storage tier to another. But this is where the similarity ends.
MSM systems, sometimes referred to as archive management systems, are becoming less focused on media archiving and more focused on active, file-based digital workflows, distribution, content exchange and collaboration. The term archive management no longer does these advanced systems justice.
Fundamentally, MSM solutions interface directly into broadcast devices (online storage), manage all tiers of storage and provide a unified and intelligent view of the storage infrastructure to various broadcast control devices. MSM systems rely on copy rather than move operations (no stub files) to migrate content into the storage infrastructure from the online tier. These systems are less focused on expensive storage tier capacity maximization and more focused on the complex collaborative and accessibility requirements of a broadcast facility.
MSM variations
To confuse the issue further, within this class of MSM solutions, features and capabilities vary widely. In general, MSM systems are server-based software solutions that reside between the so-called media network — which connects various broadcast devices — and the storage network — which connects the near-line and archive storage tiers. (See Figure 4 on page 78.)
MSM systems take responsibility for the broadcast asset directly from online storage through near-line, archive, offline and back. This eliminates the need to have an additional proprietary control layer act as an intermediary between online and near-line storage to facilitate storage management.
Some MSM solutions can provide a truly unified asset storage infrastructure. They can span a mix of broadcast devices, including on-air video servers, post-production editing platforms and digital newsroom systems.
In addition to the obvious cost and workflow benefits of this single storage infrastructure, advanced MSM systems can also offer in-path content transcoding, allowing tapeless interoperability between these distinct silos. Add time-code-based partial file restoration to this mix, and the true benefits of a collaborative, file-based broadcast workflow become obvious.
MSM migration
Advanced migration, or content lifecycle policies, provided by MSM solutions fit better into broadcast environments than the harsh rules defined in HSM applications. Not only can decisions be made based on media-centric parameters, but these MSM applications also provide robust programmatic interfaces (APIs) for control and management by higher level business systems, such as traffic, inventory systems and automation.
For example, with a single command, traffic can directly instruct an advanced MSM system to copy a newly ingested, high-revenue movie into the archive library, make two copies of it for protection and generate a frame-accurate Windows Media proxy copy for Web access. An editor can then access the proxy generated by the MSM system from his or her desktop using the MAM interface and select the shots to be used in creating movie promos.
This shot list can be sent to the MSM system via the same API to partially restore the segments based on mark-in and mark-out time code values defined in the edit decision list (EDL). The MSM system then extracts the matching segments from the original high-bit-rate movie, transcodes them as necessary and digitally delivers them to the editing system for creation of the promos. Once the promos have been completed, the same workflow (and API) brings this newly created content from the editing platform through the MSM system and to the on-air video servers for playout to air.
Broadcast control systems can provide intelligent contextual management of content as it migrates through the MSM system. By monitoring relevant broadcast-centric factors, such as rights management information, operator needs, on-air playlist demands and intangible content relevance, migration can be driven by complex decisions rather than simple machine logic.
Prioritization is a key feature of MSM systems. For resource assignments, it factors in the importance of content and how quickly it is required. This intelligent migration management and inherent support for request prioritization allows effective use of offline storage, providing near limitless and inexpensive storage expansion, as well as support for off-site content replication.
Because MSM systems are focused on direct interface to broadcast devices, they can handle either single-file assets or compound objects comprised of a few or even hundreds of component element files. As these assets migrate throughout the storage infrastructure, they are handled as a group, ensuring the complete asset is available as required.
MSM solutions provide all of the necessary broadcast-centric functionality, with the added benefit of easy, low-cost expansion as the broadcast operation evolves. With a distributed architecture, advanced MSM systems offer incremental scalability through the addition of movement engines, or actors. These actors are used to provide additional system bandwidth and redundancy at any point without necessitating downtime. This redundancy and scalability simply is not possible with HSM systems — as well as some less-advanced MSM systems — because of their monolithic architecture.
What this means
This is by no means an exhaustive list of the factors that should play part in this complex decision-making process. It is important to partner with a solution provider who not only fully understands the business, but who focuses exclusively on the complex, file-based storage management needs of global broadcasters.
Effective unification of the distinct digital storage silos is necessary if a facility is to benefit from next-generation, file-based collaborative workflows. Implementation of an advanced MSM system, built specifically to meet a broadcaster's needs, gives users the power and flexibility to get the most out of their assets and achieve a unified media storage infrastructure.
Brian Campanotti is chief technology officer for Front Porch Digital and has been involved with the development of MSM systems for nearly 10 years.
