While HD television, 3-D films, Web 2.0 user-generated content, file-based acquisition from new digital film and video cameras, and the coming Ultra HD have opened new creative possibilities and greatly enhanced the end-user experience, they have also created vast amounts of digital data that need to be stored, accessed and retained.
According to the analyst firms Coughlin Associates and the Clipper Group, HD video generates more than 500GB/hour of data and quintuples the storage needs over SD. Additionally, 3-D films require twice as much storage as HD, and Ultra HD is predicted to require 20 times storage of HD. The result is that between 2009 and 2015, the media and entertainment industry will see a 10-fold increase in the required digital storage capacity per year. By 2015, more than 46 exabytes of digital storage will be needed for digital archiving and content conversion and preservation.
For many years, professional videotape was a mainstay of the media and entertainment industry, valued for its portability and low cost. And while production needs often rely on disk file-based storage, long-term storage represents a different challenge. Today, many users find that data tape is a cost-effective way to maintain large amounts of data for long periods.
Linear Tape-Open (LTO)
The advancements in Linear Tape-Open (LTO) technology have made it an ideal long-term data retention solution for video applications. Tape has a potential 30-year shelf life. The LTO standard is supported by more than 3.7 million drives, and 150 million LTO Ultrium cartridges have been shipped since the technology was first launched in 2000. Each LTO-5 cartridge has a native capacity of 1.6TB, making it easy to scale storage needs simply by adding more LTO tapes.
Finally, an LTO-5 solution can reduce physical storage requirements. A standard wall rack (6ft by 6ft or 1.8m) with 14 shelves can store approximately 1000 cartridges — or the equivalent to 120,000 hours of 25Mb/s SD or 30,000 hours of 100Mb/s HD content.
Linear Tape File System (LTFS)
Linear Tape File System (LTFS) specification is a self-describing tape format using the LTO Ultrium 5 open standard. LTO-5 data tapes can be exchanged between systems that understand the LTFS format, and software systems that understand the format can provide users with a file system view of the media.
The LTFS format specification defines the organization of data and metadata on the tape. Files are stored in a hierarchical directory structure. LTFS is enabled by the dual partitioning capability of LTO-5 technology. LTFS technology provides file system access at the operating system level, using one partition to hold the file content data and the other to hold the associated file metadata.
This allows LTFS LTO-5 tapes to be self-describing. Data tapes written in LTFS format can be used independently of any external database or storage system. This enables a standard file system view of the data stored in the tape media. Because each tape has its own file system, files stored on the LTFS-formatted media can be viewed and accessed the same way as other files stored on any other portable storage media, such as disk, USB drives and flash drives.
Advantages of LTFS
With LTFS, tape systems can be easily indexed for fast and easy retrieval while ensuring data portability and data interchangeability. Users can interchange content across different operating systems, software applications and physical locations. An LTFS solution combines the economy, robustness, high density and low power consumption of tape, with much of the functionality and usability of a hard drive.
One of the clear advantages LTFS has over the proprietary archive formats used by traditional back-up and archive solutions is that it preserves the file system structure of the source file system and leaves the files as identical copies of the source files. That may seem like an obvious thing to do, but most proprietary archive systems encapsulate the source files into objects. This has the advantage of encapsulating a group of files with their associated metadata, but it has the disadvantage of requiring an extra layer of software to access, extract and interpret what is in the archive. Any additional required software beyond the basic and ubiquitous standards provided by IT manufacturers could reduce the chances of being able to access the archived content in the future. With LTFS, the files will always be accessible from common platforms at the file system level without requiring another application to extract them.
Folders are a natural way to encapsulate groups of files and their metadata using just a file system. Most digital media professionals are already arranging their disk storage this way such that all sources shot on a particular day in production or an entire project in post-production can be found in a unique directory tree. Giving the highest level folder a meaningful name can facilitate finding a past project or footage from a particular shoot easily.
Because LTFS brings a disk-like file system to LTO data tape, it allows digital media professionals to use a simple drag and drop process. This keeps folders and files in the same hierarchy as on the source disk or digital media. At any time in the future, users can restore a single file or an entire directory tree with a simple drag and drop.
As compared to standard tape implementation, LTFS provides digital media professionals with a several benefits:
- Faster access to data: When a tape is mounted, the stored tape files and directories can appear on the desktop in the same way as a disk directory listing.
- Simple drag and drop file operations: LTFS increases ease of use, allowing users to simply drag and drop files to and from tape.
- Compatibility across customer environments: LTFS uses a non-proprietary data format file system that is independent of specific software applications. LTFS can address long-term archive strategies by creating a self-describing tape that is not dependent on the software and hardware environment.
- Increased data mobility: LTFS allows users to easily share content with any LTFS-enabled system using open source software to increase data mobility.
Limitations
There are, however, limitations to LTFS. LTFS cannot be used by itself in a network or multi-user environments because the open code or “base level” deployment requires that the LTO drive be dedicated to a single computer. Also, LTFS has inherent problems that result in long waits when used with file management tools like Windows Explorer or Finder. List views are generally fine, but icon and other graphical views cause these programs to undertake painfully long seeks to read each file.
In addition, restoring more than one file at a time with Explorer or Finder will be inefficient if sorted by anything other than archive date. This is because such operations will attempt to restore in sort order rather than the order of the files on the tape. The result is that restores can literally take days.
Finally, LTFS should not be used like random access disk because, due to the inherent nature of tape, accesses must occur over a linear arrangement of data. If an application tries to update a file, it will be rewritten to the end of tape, resulting in long delays when restoring such files.
Going beyond the standard LTFS implementation
It is clear that LTFS works better when leveraged by third-party software. Some manufacturers are planning layers of software above the standard LTFS implementation to address its limitations. None of these limitations exist when LTFS volumes are accessed, for example, from an archive appliance web-based user interface.
Other benefits of embedding LTFS in an archive appliance the Cache-A system include:
- File manager utility: file moving that avoids issues with Finder/Explorer;
- URL encoding: allows archiving of file names with illegal characters;
- Networked: makes LTFS volumes available across a LAN;
- Easy formatting of media: no shell command line is required;
- Easy mount and unmount: no shell command line and no locked eject due to folders in use; and
- No client-side software: every computer can instantly work with it.
An LTFS solution offers several important operational benefits. To the user, the tape both looks and behaves like a disk, and all tape operations are handled transparently in the background by the archive appliance. An archive appliance will be able to deliver a solution that is:
- Self-contained: platform neutral to stand the test of time;
- Interchangeable: easy to distribute and interchange assets; and
- Extensible: scalable to meet future needs.
As a complete archive solution for digital media professionals, LTFS can simplify operations and improve manageability while meeting a facility’s long-term data retention requirements. Self-contained, easy to deploy with no assembly or installation required, LTFS appliances know where every file lives on each tape and can organize and restore the content for efficient linear access. Users should still use hard disk caching for file transfers. That will protect them from the latencies and access issues associated with linear tape.
Conclusion
Combining LTO-5 data tape with LTFS creates a powerful IT solution for the media and entertainment industry. LTO-5 based products using LTFS make an ideal archiving and interchange solution, enhancing both production and post-production workflows, as well as improving asset management and interchange.
Because LTFS makes the contents of the tape compatible and accessible to any facility with an LTO-5 tape drive, it provides an interchangeable and future-proof content storage format on secure LTO-5 media. Third-party archive solutions that leverage the LTFS format can provide easy-to-use archive solutions based on open standards that help ensure content will be searchable and accessible many years in the future.
—Mark Ostlund is vice president of sales & marketing at Cache-A.
