The term “mission-critical” has always seemed a bit dramatic, almost as if it were a phrase from one of those save-the-world Arnold Schwarzenneger adventure movies. The mission-critical concept has been around a long time, and the phrase is gaining popularity. But what, exactly, does it mean?
Mission-critical refers to the operations that are critical to an organization’s ability to carry out its mission. In other words, mission-critical operations are those operations that are essential to an organization’s ability to perform its intended function. A mission-critical facility is one that guarantees it will continue to operate, regardless of external conditions. Examples of such “hardened” facilities include hospitals, “911” call centers, emergency-management agencies, banking systems, credit-card companies, telco networks and express-delivery operations.
Different facilities require different levels of guaranteed support based on two main factors. First and foremost is the cost of failure. There are tangible and intangible costs to bear for failure, such as consequential loss damage or loss of reputation in the marketplace, and short-term or long-term loss of revenue. The second factor is availability. Most facilities have peak-service periods when the corresponding level of protection must also be at maximum readiness. The idea that the facility’s resources will be operational when it needs them means that the facility will be available during the time frame (hours per day and days per week) that best suits its clients’ needs.
While the concept of the mission-critical facility is not new to the broadcast industry, it is an emerging concept that station owners and engineers are just now starting to embrace. When many stations were being built, designing into them the types of systems and infrastructure necessary to support a 24-hour-a-day, seven-day-a-week fail-safe environment was beyond imagination. In most markets, the need simply did not justify the cost.
But things have changed. Today, more than 70 percent of Americans rely on television for their daily news, and the average amount of time they spend watching is seven hours daily. The public need is clearly there. Add to that the failure factors previously mentioned (not to mention frustration and embarrassment), and soon the balance is tipped in favor of providing something more than the traditional backup generator.
The buildings that house most TV stations were not designed specifically for that purpose. More often than not, the station construction was a retrofit in an existing building originally intended for something else, and systems such as power distribution and cooling were modified to fit the new station’s requirements.
Even in cases where stations were housed in structures and facilities intended for TV, few were designed to be mission-critical. Equipment was analog, and, in general, consumed more power per device than today’s equipment. And because studio lighting was exclusively incandescent, generating a great deal of heat, it was impractical to include more than a flash camera with a single light on a backup power system. Air conditioning was usually excluded as well, leaving the traditional station with little chance of staying on the air in a prolonged power outage.
New requirements
The PC has been pervading the office environment for many years. In the technical professions, the change has been slower, but inevitable. Devices built around a PC platform gain improved reliability, and broadcast-specific frames offering hot-swappable power supplies and other features have been built around PC-based devices to form character generators, still-store devices, video file servers, and more, and to control devices for critical systems. Still, these PC-based devices are more sensitive to their environments than traditional broadcast devices. When planning a new facility, care must be taken to provide the necessary space, air conditioning and power conditioning for a safe and reliable environment.
For example, chronic device failures, freezes or crashes can be totally eliminated by the insertion of an uninterruptible power supply (UPS) between the power utility and the device in question. The online UPS is fast becoming the engineer’s best friend because of its ability to condition power as well as provide reliable service. As the demand for electrical power threatens to outpace the supply, it is likely that the quality of that supply will worsen. Broadcasters are finding a greater need to be energy self-sufficient, especially where energy quality is a concern.
There are several ways a studio can conserve power as well. Exclusive use of fluorescent lighting would consume significantly less power than its incandescent counterpart. A video file server can replace many VTRs in a master-control or newsroom edit environment. Modern routing switchers and production switchers continue to grow smaller and consume less power. Flat-panel display technology is advancing, prices are declining and they continue to replace CRTs. If a station implements these energy-saving strategies without expanding its functions, it will decrease the amount of space, power and cooling it requires.
UPS technology has improved steadily, becoming smaller and more feature rich with each new generation. Improvements in UPS, backup-generator and energy-storage technologies allow the station’s entire critical load to be made failsafe within a workable footprint. A well-designed power system can also include dual utility feeders from separate power grids. These feeders are path diverse and completely independent, and each is capable of handling the entire load of the facility.
New technology, old facility
There are two ways to upgrade a facility: the traditional “replace-it-as-it-breaks” method, and the “tear-it-out-and-start-over” method. Each has advantages and disadvantages. In time, the former usually leads to the latter. But, short of a new facility, a facility upgrade is the next best thing. Upgrading a master control to digital, for example, can present opportunities to fix some of the known problems within a facility, and to provide some backup if desired.
When planning an upgrade, it becomes evident that much of the new equipment, including its associated power and heat load, can be physically located further away from the operator(s). Ideally, the only things that need to be near the operators are control panels and monitoring systems. Putting all of the station’s core equipment together makes more sense today than ever.
This approach allows a control room to go just about anywhere, with content storage and data paths delivering audio and video someplace else. If the station does not already have an equipment room, an upgrade is the perfect opportunity to consider installing an equipment room, where the environment can be tailored to the equipment, and power and cooling can be well managed. If the budget permits, the backup-power capabilities of a UPS and standby generator can be most easily implemented when designing an equipment room, since the critical load will be concentrated within one area. When sizing the system, determine the extent of the entire load and overrate the capacity of the system by 30 percent.
New technology, new facility
For a new facility, technology placement and integration are only part of the equation. Site selection is the first step, and is of paramount importance. In this decision, proximity to utilities, fuel and telecommunications networks, and space for terrestrial and satellite links, are obvious factors. Less obvious are considerations for disaster avoidance, including such factors as flood plane vs. building elevation, ground water, geological and meteorological disturbances, and locations of natural and manmade barriers.
The needs of the station must be addressed in a master plan. Ideally, this plan should describe every aspect of use of space as well as the build program. Everything must be considered, from departmental adjacencies to levels of redundancy. Some questions to ask when deciding the level of protection required include:
- Is the facility supporting a product that must be available without interruption?
- What is the business cost of downtime?
- Are short-term interruptions acceptable?
- What level of risk is acceptable?
- How does reliability or the lack thereof set your business apart from its competition?
- What are the market trends and your anticipated growth?
- How much redundancy is enough? Where N is the number of unprotected systems online, would your business require N + 1, N + 2, or even 2N?
Integrating the technology becomes much easier after resolving these issues. The foundation for reliability is laid during the building’s planning phase. Master control, newsroom, studios and edit suites are generally arranged so that they surround the equipment room. Depending on the level of redundancy decided upon, several equipment rooms may be employed in a distributive-processing approach.
For example, the studio router can be in one equipment room, while the master-control router is in another. Using this approach, each router can be smaller and can share its room with a few backup destinations. The routers can share sources through tie-line management or through the use of time-domain multiplexing employed by some of the newer routing switchers. In another example, a storage-area network can be clustered across several rooms. The rooms don’t have to be far apart. The idea is to distribute processing devices, but not so far apart that they can’t work as one.
Compartmentalizing is the ultimate goal. True mission-critical facilities are designed so that a catastrophic event in one area does not shut the whole facility down.
Vision of a mission-critical station
The facility needs to take its location into account. What are the environmental threats that may need to be overcome? How critical is critical? If it is deemed that downtime is unacceptable, the next question to ask is, “For how long?” The planning of the facility takes into account the duration required by the user. Twenty-four-hour, 48-hour or 72-hour durations are not uncommon, and some users demand more. Fuel capacity, emergency water supplies, physical security and even food rations can be included in the deployment plans.
One thing to keep in mind when planning for the future is growth. While new devices consume less power than their predecessors, the trend is to take advantage of the newfound space and power with new equipment. More capabilities add to the bottom line by increasing revenue or by decreasing production costs so that even if the square footage remains fixed, the overall consumption will increase. The needs and expectations of the public are growing. The ever-increasing demand for information, coupled with growing viewer reliance on TV, make service interruptions increasingly unacceptable. The demands on resources are growing as well. Electricity consumption in the United States has increased to the point where it now accounts for over 37 percent of the total energy consumed worldwide. This is up from 25 percent in 1975. Given the problems in California, and how those problems are beginning to expand to other regions, one can only assume that the current trend will get worse before it gets better. The broadcaster must determine the requirements placed on his new facility, for both short- and long-term. The priority level of the final product is a major consideration when determining what the facility must be able to withstand.
Paul Kast is technical director of Carlson Broadcast & Media. His e-mail address is Pkast@carlsonsolutions.com.
