The transformation to file-based digital workflows in broadcasting has resulted in large, growing volumes of files that need to be stored and protected for long periods of time. These files are created and processed by various applications in the workflow chain, and it is common for organizations to store the files in siloed storage infrastructures, whereby a separate storage system is used for each application or geographic location. (See Figure 1.)
Silos require additional investments for backup and replication products to protect against data loss. They also need tiered storage to enable movement of data to the most cost-effective storage devices over time. This siloed approach creates significant complexity, and increases the capital and operating costs of storage systems. Furthermore, using silos to store large amounts of data for long periods of time results in the following issues:
- Insufficient data protection
The solutionTraditional backup/restore is not suitable for large data volumes where the time-to-backup becomes prohibitively long. In the event of disaster, restoration can take weeks, months or even years. In addition, backup does not protect against alteration or corruption. - Downtime and unavailable dataGiven long retention times, faults will occur over the lifetime of the data. Silos are prone to single points of failure and require manual intervention to recover from a fault, increasing downtime that results in lost productivity.
- Inadequate scalability
The technologyThe lifetime of files exceeds that of the storage hardware on which they reside. Silos require manual migrations from one hardware generation to the next, limiting an organization's ability to take advantage of dropping hardware costs.
Bycast StorageGRID 8 virtualization software for large-scale digital archives allows organizations to improve operational efficiency and avoid the pitfalls of siloed systems by consolidating and virtualizing storage across sites and applications.
As shown in Figure 2, this solution manages heterogeneous storage devices, ranging from high-performance disk to tape, across multiple locations to create a unified file archive. It insulates applications from changes to the hardware, including equipment failure and obsolescence. The solution also facilitates the creation of fault-tolerant systems that can scale to tens of petabytes and hundreds of sites. Stored data are managed in accordance with user-configurable Information Lifecycle Management (ILM) policies that govern the automatic placement of content at geographic locations and storage tiers to meet the business needs of the organization. The solution simplifies the management of massive storage systems through automation, while ensuring the availability of data even during system downtime.
StorageGRID is an object-based grid storage system that stores unstructured data such as images, audio, video and documents. An object is any container of data and may have metadata associated with it. In the most common use case, an object is a file, and its metadata are the attributes of that file (type, name, path, etc).
Within this solution, objects are uniquely identifiable and stored in accordance with storage policies based on their metadata. These policies determine how many copies of the objects are stored, geographic placement of objects and storage tiers on which objects reside over time. Applications store and retrieve data from the grid via gateways that provide file or object level access over standard interfaces such as CIFS, NFS and HTTP. The solution can be deployed within one site or across multiple facilities, and objects stored at one site may be accessed at all sites.
Stored files are assigned unique digital fingerprints, which are used to verify integrity of data at the time of access, replication and movement between storage devices. Proactive checks performed in the background protect the integrity of inactive content. When a file is stored, the solution examines its metadata and enforces applicable storage policies. This allows dissimilar file types to be managed differently. For example, intermediate files associated with one project can be stored directly to tape, whereas models used in 3-D renderings may be kept on disk and replicated to a second site where a render farm resides.
Data protection and preservation is provided by enabling content to be replicated to multiple sites, and monitoring the integrity of data to guard against corruption. If a device or even an entire site is lost, it will automatically rebuild data from other replicas and validate its integrity. If corruption is detected, data is automatically rebuilt from a known good copy.
By abstracting the hardware, the solution ensures that users are unaffected as new storage devices are added and old ones are retired. Migration from obsolete hardware to newer generations is automated and transparent. When a storage device is designated to be retired, its contents are automatically moved to newer devices without manual intervention, or changing file paths. Files remain where users and applications expect to find them.
StorageGRID addresses the shortcomings of the silos in three ways. First, it offers increased data protection. With configurable replication policies, the solution automatically stores redundant copies at distinct geographic locations. These, together with automated rebuild, eliminate the need for error-prone backup/restore procedures. Furthermore, comprehensive integrity checks protect data from corruption or alteration, thereby ensuring authenticity of data.
Second, it minimizes downtime. The grid architecture eliminates single point of failure. In the event of failures, or even when an entire site goes down, failover is automated, and operations continue. Automated disaster recovery protects against site, storage, or system failure and fully restores data to its original state.
Third, the system provides unparalleled scalability. The solution enables capacity and footprint expansion as needed, when needed. In addition, it allows organizations to take full advantage of decreasing hardware costs, increasing densities and new technologies through seamless hardware refreshes that do not require manual migration. Organizations can increase storage capacity, while reducing the footprint within the data center.
Summary
Challenges of long-term file storage can be minimized by choosing an archive system that consolidates and shares storage resources across sites and applications, rather than managing individual storage silos. With StorageGRID, organizations can use commodity hardware to create scalable digital archives that preserve and protect data, while reducing capital and operational costs of long-term storage. The solution also ensures consistent, reliable access to data for decades by delivering data protection, business continuity, automated disaster recoveryand multisite access in heterogeneous storage environments.
Annette Saliken is director of communications and David Slik is chief architect for Bycast.
