One of the keys to moving large files on a network is to move the data in large blocks without a lot of header information in between.
As the computer world moved from mainframe computing to the distributed world of desktop systems, there were a number of technologies available for interconnecting equipment. Ethernet, High-Performance Parallel Interface (HIPPI), SCSI, ATM and a number of other legacy technologies existed or were under development. As computers became more powerful, the need for high-bandwidth connections increased. We have seen the development of Ethernet in its 10, 100 and now Gigabit flavors; HIPPI has continued to evolve; SCSI has grown to include Ultra-SCSI and other variants; and ATM has seen broad deployment as part of the telco infrastructure. Each of these has benefits and limitations.
Fibre Channel started out as a way to move high traffic volumes between systems without the overhead associated with Ethernet and without some of the cable headaches associated with HIPPI and SCSI. To a large degree, Fibre Channel has succeeded.
One of the keys to moving large files on a network is to move the data in large blocks without a lot of header information in between. While individual packet payload size is 2048 bytes, Fibre Channel allows the designer to string a large number of payload packets together into a sequence for delivery. A sequence may be as large as 4GB. Furthermore, Fibre Channel recognizes sequences at the hardware level. This means that large sequences can be delivered to a device without requiring a lot of processing power to read and interpret header information on the packets.
For many years, storage devices were an integral part of the server itself. Typically, servers were connected to disk drives using IDE or SCSI interfaces. The problem with this approach is that it is limited by the constraints of either IDE or SCSI (limited cable distance and limited number of available drives), and it does not allow efficient sharing of high-bandwidth data such as video.
Fibre Channel leverages existing technology. It allows manufacturers to continue to use SCSI software commands, but replace the limited IDE or SCSI physical layer with a new architecture.
Some fifteen years later, how is it going? Things look good for Fibre Channel these days. It has been adopted as part of the core infrastructure for server-to-server transfer by a number of vendors in the broadcast television market. While interoperability problems exist, it is possible to put together Fibre Channel systems using off-the-shelf components. To a large extent, Fibre Channel does deliver the performance it promised in the early days. If systems are designed as point-to-point, performance is even better.
Before we get into a discussion of Fibre Channel storage, let's review some basics about Fibre Channel. First, Fibre Channel allows three common topologies; point-to-point, arbitrated loop and switched. Second, it allows designers to employ two very common protocols; Internet Protocol (IP) and SCSI. This allows them to easily migrate existing products that are either IP- or SCSI-based to Fibre Channel. Finally, unlike ATM, Fibre Channel block sizes are large, making it a good match with the very large file sizes typically found in video. Characteristics of Fibre Channel are listed in Table 1.
What is Fibre Channel?
Fibre channel is a two-way (duplex) communication channel that can be used to interconnect a wide variety of computers and storage subsystems. Figure 1 (on page 40) shows an example of this type of system. Each computer or storage subsystem is a node that has both a transmitter and receiver. Data can be transmitted and received across a Fibre Channel link, which can be a copper wire (up to 25 meters), a short-wave optical fiber (up to 500 meters) or a long-wave optical fiber (up to 10 kilometers). There are three main classes of service available with Fibre Channel. Class 1 is a dedicated connection for point-to-point operations. Class 2 provides a connectionless operation that requires a confirmation from the receiving node. Class 3 is a connectionless service that requires no confirmation. This is the class typically used for storage subsystems. Fibre Channel can support a number of software protocols, of which the two most important are SCSI and IP.
Fibre Channel frame structure
Data is sent across the Fibre Channel fabric as payload contained in Fibre Channel frames. The 2K payload is surrounded by a header and footer, which help direct the frame through the network and correct errors that may have occurred in transmission.
After the Start of Frame, there is a Frame Header. The header contains information about where the frame came from, where it is going, and other information that helps the data in the payload to be correctly organized at the receiving end. Then comes the payload, the data to be transferred across the network. After the payload, there is a four-byte CRC error check and finally a four-byte End of Frame marker. (See Figure 2.)
Fibre Channel layer structure
Fibre Channel is designed in a layered structure. These layers, illustrated in Figure 3, are defined as FC-0 through FC-4. Much like the ISO layer model, they specify different functional components of the overall Fibre Channel technology. FC-0 defines the physical link to be used to connect the components. This includes physical measurements of connectors and fibers along with electrical parameters.
FC-1 defines the way data is encoded and decoded (commonly called the transmission protocol). It also defines how errors are handled. The information transmitted over a fiber is encoded eight bits at a time. When data is sent across a fiber optic cable, a scrambling algorithm is used to make sure the data is DC balanced, meaning that the signal being transmitted does not have a long string of ones or zeros. Two bits are added to the packet during the scrambling, making for a total of 10-bit encoding.
The Signaling Protocol (FC-2) layer serves as the transport mechanism of Fibre Channel. FC-2 defines the framing rules for the data to be transferred between ports, the different mechanisms for controlling the three service classes and the means of managing the sequence of a data transfer.
The FC-3 level of the FC standard is intended to provide the common services required for advanced features. These features include combining multiple ports to aggregate bandwidth, the ability for more than one port on a device to respond to the same address, and multicasting, which allows one device to send to multiple destinations on the network.
FC-4 defines the application interfaces that can be used over Fibre Channel. While a number of interfaces are listed, the predominant ones are SCSI and IP. SCSI is primarily used to connect disk devices to servers across a network. IP is used for FTP and many other commonly used IP applications.
Topologies
Fibre Channel fabric can be configured in a number of different ways depending on the requirements and performance required across the network.
Point-to-point is the simplest and least expensive topology to implement. It also is self-explanatory. In an equipment pair, the Fibre Channel Gigabit Linking Modules (GLMs) are connected back to back. No hubs or other control devices are needed. Costs are low, installation is simple, the bandwidth on the network is well defined, and control and interoperability issues are limited; so resolving technical issues is a breeze.
The next step up in Fibre Channel topology is the Fibre Channel arbitrated loop (FC-AL). Figure 4 illustrates a single FC-AL. FC-AL has several advantages. As with point-to-point, it is low cost and external hardware is not required. In small configurations it is simple, and for that reason it is easy to troubleshoot. It is also expandable, with up to 126 devices per loop. Single-loop FC-AL does have some problems though. First, it is prone to failure. Since it is a single loop, a break anywhere in this loop crashes the entire network. Second, in a single-loop configuration, Fibre Channel does not support simultaneous communications. This can seriously limit bandwidth on the network.
Broadcasters will find that most vendors employ a dual-loop configuration. The dual-loop FC-AL eliminates the single-loop failure mechanism. If one of the loops fails, the other assumes the load. A dual-loop FC-AL also allows simultaneous communications between devices, greatly increasing bandwidth available. While the cost of dual-loop topology may be greater, for most applications the security and performance increases are worth the increased costs.
The third common Fibre Channel topology is switched fabric. If you are familiar with switched Ethernet networks, you understand the basic premise behind switched Fibre Channel.
Switched fabric works by connecting full-bandwidth pipes between any two devices that wish to communicate. This allows many devices to communicate at the same time, it increases the effective bandwidth available for each device dramatically, and it provides fault tolerance in large networks. Of course, all of this comes with a large price tag — anywhere from $20,000 to $100,000 and beyond depending on the complexity of the switch. Also, if you are looking for switches with full redundancy and SNMP monitoring, your choices at this point are limited.
Fibre Channel is one of the best ways to move large amounts of data within a facility. Its ability to put together a large number of packets into a single sequence for delivery is one of its key strengths. This allows Fibre Channel to deliver high actual throughput with a small amount of overhead.
Brad Gilmer is executive director of the AAF Association and president of Gilmer & Associates, a broadcast consulting firm.
Fibre Channel at a glance
- Data rates up to 100 MB/s for single loop and 200 MB/s for dual loop
- Distances of up to 10km over fiber optic cabling, although there are advances that extend this to over 100 miles, and up to 47 miles over coax
- Protocol independence — support for ATM, SCSI, IPI-3, IEEE 802, SBCS, HIPPI and IP
- May employ point-to-point, loop and switched fabric
- Requires SCSI-like bus arbitration with non-simultaneous I/O unless dual loops are employed
- May be used as either a storage interface or a network topology
- Has a limit of 127 devices in a FC-AL, and 16 million devices in switched fabric
- Uses absolute addressing
- Performance remains constant as distance increases
A Fibre Channel primer
Let us stop and explain the options for a Fibre Channel arbitrated loop. Hubs are devices that attach multiple nodes to a Fibre Channel-arbitrated loop, with all nodes sharing the 100MB/s bandwidth. Hubs on the market today provide the connection of between five and 12 nodes. The hub allows devices to be added to or taken off the loop without impacting loop operations. A major advantage of a hub is that if any one link fails, that link is removed and the other links continue to operate.
Switches provide multiple connections as well — typically eight or 16. Switches are more expensive than hubs, but can operate multiple concurrent transmissions at 100MB/s. As an example, one server can be communicating through a switch to a disk subsystem at 100MB/s, while another server is communicating with a tape library through the same switch at 100MB/s.
Another component is a Fibre Channel router. This is a device that can communicate with an SCSI cable on one side and a Fibre Channel link on the other side. It is useful if you want to attach a tape library that doesn't have a Fibre Channel controller to a Fibre Channel loop.
