Fibre Channel vs FCoE vs iSCSI: Which to Choose for Performance

For anyone who needs to understand the different protocols available for sharing block storage over a network, the choices can be confusing. So, let's look at the differences between them, and why a person or organization would choose one over the other in a particular situation.

What is Networked Storage and Why is it Used?

Back in the early days of computing, data was often stored on media such as punch cards, magnetic tape, and floppy disks. There were various convenience and reliability issues with all of these. When the hard drive was developed and became popular, different connection types evolved. But in most cases, this involved one computer system; the storage space and the data stored on it was basically locked onto the computer it was physically connected to. The downsides to this arrangement are the difficulty of accessing the data from elsewhere, in addition to the frequent waste of disk space.

The invention of methods of sharing this storage between computer systems helped solve these challenges. Networked Storage enables the allocation of disk space in blocks to multiple computers, making the use of expensive disk space much more efficient. It also can enable, for example, a cluster of database servers to have access to the same data simultaneously.

There are also methods of sharing files across a network, which are called NAS (Network Attached Storage). This is a totally different strategy that has its own protocols such as NFS, SMB, and AFP, and is beyond the scope of this article. Here, we are focusing on "block" storage, which is chunks of disk space that appear to the computer's operating system to be the same as a locally attached hard drive.

So, how does all this work?

What is SCSI?

SCSI is the grandparent of them all. Let's jump back in time again to the days before Networked Storage. There were a variety of methods in use to connect hard drives to computers, such as SASI, SMD, ESDI, ST-506, etc.  Most of these were proprietary, i.e. developed by a manufacturer such as Seagate or IBM, with little or no interoperability with other vendor's equipment.

The  SCSI (Small Computer System Interface) protocol evolved from SASI (Shugart Associates System Interface), and became an ANSI Standard in 1986. This enabled multiple manufacturers to develop products that would work together, instead of buyers having to "lock in" with one vendor. This in turn fueled its growth in popularity.

A typical SCSI (usually pronounced "skuz-ee") subsystem implementation involved one or more controllers (circuit boards with cable connectors), one or more copper wire ribbon cables with multiple hard drive connectors, and a set of hard drives. Some of the primary features of SCSI are the relatively simple command set that is used to control the hard drives, and the amount of devices that can be attached to a controller (16 or more).  One drawback was the limited cable length supported, but this was rarely an issue with hard drives within a single computer system.

While these physical aspects of a SCSI subsystem are rarely used in computers today, the SCSI protocol itself has found longevity as a basis for newer protocols such as SAS (Serial Attached SCSI) and FC (Fibre Channel), the latter of which we will examine next.

What is Fibre Channel (FC)?

The Fibre Channel, abbreviated simply "FC", was developed in the late 1980s and became an American National Standards Institute (ANSI) Standard in 1994. As the name suggests, it was designed to be based on relatively new multi-mode fiber optic cabling as the physical transport, at least in longer distance scenarios, to overcome some of the limitations of the SCSI physical layer.  FC can actually run on copper cables over short distances, but the distance limitations on fiber optics are much more generous.

Because the SCSI protocol was very popular and robust, it was implemented as an "upper layer protocol" riding on top of FC as the transmission protocol. In other words, FC handles the physical connectivity, encoding, and transmission of data, such as SCSI commands, from one endpoint to another. FC is actually capable of transmitting other kinds of data, but we are focused on SCSI for block storage here. An FC network has the inherent features of providing lossless delivery of raw block data, as well as "in order" delivery of packets, both of which improve reliability and efficiency.

Fibre Channel does require specially designed hardware. Servers need to have a "Host Bus Adapter" port or card (HBA), and the storage devices also need to have an FC interface, which is more often referred to as a front-end port in this case. An FC network is usually referred to as a SAN (Storage Area Network).  There are a number of different topologies that an FC SAN can be configured to use.

A typical scenario consists of at least two dedicated FC Switches with multiple physical ports, with each switch representing a physically separate "Fabric", and each server or storage device having at least one connection to each Fabric. This provides the added features of redundancy and the potential for improved performance. If a switch, HBA, or fiber optic cable fails, the connection to the storage is not lost. In certain cases, the multiple connections can be used to increase bandwidth and throughput. Fibre Channel is a very feature-rich environment, and has a variety of mechanisms to control things like which initiators can "see" which target devices, for example.

It is also worth noting here that the development of FC SANs made it feasible for manufacturers to design large devices with many hard drives, generally referred to as "storage arrays". These typically use a data protection method such as RAID (Redundant Array of Inexpensive Disks), which can combine groups of physical hard drives into arrays, with either multiple copies of data or a calculated parity that enables reconstruction of data. If a hard drive fails (which is usually inevitable with spinning disks), it can simply be removed and replaced, and the data is rebuilt on the drive. The disk space is divided up between servers on the SAN as desired, in units called Logical Unit Numbers (LUNs). Thus the space can be used much more efficiently and dynamically.

Currently, FC speeds up to 128 Gbps (128GFC) are available. However, the most common implementation in use today is 32GFC, usually with a mix of lower-speed devices on the SAN.

Key disadvantages of an FC SAN. The overall result of using an FC SAN is a system that is very reliable, very fault tolerant, and very expensive. Server HBA cards are costly, FC Switches and Storage Arrays are quite expensive, and even the fiber optic cables connecting all this together are not cheap.

What is the iSCSI Protocol?

The Internet Small Computer Systems Interface (iSCSI) protocol was designed to utilize existing TCP/IP network infrastructure to transport block storage data. An iSCSI implementation is referred to as a SAN as with FC, but in this case it is specifically an iSCSI SAN. The capabilities and limitations of this technology are different from those of an FC SAN in several ways.

The TCP/IP Protocol is a combination of protocol "layers" such as the Transport Control Protocol (TCP) and the Internet Protocol (IP). These standards are approved and maintained by the Internet Engineering Task Force (IETF). This is basically the building block that most of the Internet as we know it is built on. The TCP/IP Addresses that are used to direct packets of data from one point on the Internet to another are an aspect that most of us are familiar with.

The iSCSI Protocol, put simply, is SCSI commands encapsulated into TCP/IP packets which are then sent over whatever physical network is available. iSCSI was ratified as a standard by the IETF in 2004. The first obvious advantage to iSCSI is that most servers are already connected to an Ethernet network and have at least one IP address. Many modern storage arrays have iSCSI capability, so an iSCSI SAN can be built right on an existing network. The second is that there is no need to purchase expensive FC hardware and run a second set of fiber optic cables.

Another advantage to this method is that there is no need to learn Fibre Channel skills; a company that has an Ethernet network is likely to already have personnel with networking knowledge, and iSCSI is relatively easy to set up.

Key disadvantages of iSCSI versus Fibre Channel. There are, however, some distinct disadvantages to using iSCSI compared to FC. First, TCP/IP has an inherent processing overhead, which can decrease performance and bandwidth. Ethernet does not guarantee "in order" frame delivery, which causes TCP to detect when packets are lost or dropped and retransmit them. Another disadvantage is redundancy: FC SANs are typically built with multiple HBAs on the servers, dual isolated Fabrics, and multiple target ports on the storage arrays.

What is Fibre Channel over Ethernet (FCoE)?

Ethernet networks have been in existence in some form since the mid-1970s, invented at the Xerox Palo Alto Research Center (PARC), and only for internal use. However, Xerox permitted the specification to be made public, and it was approved as a standard in 1983 by the Institute of Electrical and Electronics Engineers (IEEE) 802.3 Committee. Since then, Ethernet has become ubiquitous in offices, data centers, and even most homes. Speeds have increased from 3 Mbps to a whopping 100 Gbps and higher.

FCoE is a somewhat newer standard that was part of the International Committee for Information Technology Standards (INCITS) T11 FC-BB-5 standard, published in 2009. Basically it allows the encapsulation of native Fibre Channel packets into Ethernet frames. FCoE does require some enhancements to the Ethernet network, but most if not all modern Data Center Ethernet switches support this. For servers, FCoE can run on a "standard" Ethernet interface, but performance is much better when using a Converged Network Adapter (CNA). Since FCoE is a layer running directly above the Ethernet layer and does not involve TCP/IP, it is not routable over non-contiguous networks.

Key advantages of FCoE over Fiber Channel. The advantages of FCoE over pure FC are primarily in reduced cost, no need to buy dedicated FC hardware, and not as many advanced skills needed to install and manage it.  The disadvantages are that with FCoE, storage traffic is being sent over the same network as other traffic, and contention is possible, and that only some storage arrays support native FCoE.

Key advantages of FCoE over iSCSI.The advantages of FCoE over iSCSI are mainly performance and reliability, however with a very fast Ethernet network in both cases, the difference may only show up in very data-intensive situations.  The disadvantage (depending on one's point of view) are that FCoE still requires some FC skills, and some enhanced hardware.

Fibre Channel vs FCoE vs iSCSI: Which is Best, When, and Why?

So how can we decide which is best?  First of all, in cases where a datacenter already has an FC SAN installed, unless it is aging and slow, it is difficult to justify switching from that to an Ethernet or TCP/IP based solution. This is compounded by the presence of storage equipment that does not support a protocol other than FC.

Looking ahead, however, eventually most of the components of an FC SAN will approach the end of its useful life and need to be replaced. In many cases, a fast, modern Ethernet network is already in place, and servers will already have Ethernet interfaces, possibly even already equipped with CNAs.

So the choice to start migrating toward FCoE or iSCSI may be the right one. If the purchase of storage equipment capable of supporting those protocols or "hybrid" switches that can bridge between FC storage and Ethernet networks is a possibility, this vision could become a reality.The use of iSCSI for low-cost test lab environments is a good use case and quite popular.

There is a concept called the "Iron Triangle" which is often applied to computing equipment, as well as other areas. The statement goes:  "Fast, good, or cheap – pick any two".  In the case of storage equipment and protocols, it might make more sense to replace the word "good" with "reliable", but in general this idea applies to this discussion. What are your priorities?  Can you afford "Fast and Reliable" or do you need to find a compromise? They are all viable solutions – it's your choice.