iSCSI Review

Introduction
iSCSI
iSCSI under a microscope
Others (iFCP, FCIP)
Conclusion

Introduction

With the release of the Fibre Channel and SAN based on it the storage world staked on a network access to storage devices. Almost everyone announced that the future belonged to the storage area networks. For several years the FC interface was the only standard for such networks but today many realize that it's not so. The SAN based on the FC has some disadvantages, which are the price and difficulties of access to remote devices. At present there are some initiatives which are being standardized; they are meant to solve or diminish the problems. The most interesting of them is iSCSI.

The word iSCSI can often be seen in newspapers and ads of leading storage device makers. However, different sources have very different views, and some consider the iSCSI an indisputable leader for data storage systems in the near future, others have already given it up for lost yet before it was born.

Sun in opposition to IP Storage

Sun got into opposition to IP Storage. Sun Microsystems won't produce storage area devices with an IP access. Mark Canepa, Vice-President of Sun, who is in charge of production of all storage area systems said that IP Storage was just a dream, reports Byte and Switch.

"Canepa said TCP/IP is impractical for use in storage area networks... because the processing involved reading and forwarding packets appropriately introduces lag time or latency. Canepa asserted that if storage over IP does have a future, it will be a long time coming three to five years or maybe never. Storage cannot run on top of a general-purpose stack designed for everything; storage traffic has specific needs. The dream of slapping together a TCP/IP network because everyone knows how easy it is to do it, won't work. The technology hurdles are greater than people realize. Sun is betting that Fibre Channel will be the main driver for SAN communication for some time to come." So far nobody among storage system makers took a so definite stand against IP Storage. Competitors of Sun - Hewlett-Packard and IBM more or less support these technologies.

HP promises iSCSI support

"The final version of the new technology must appear in the first quarter of 2002, HP will begin shipping a variety of storage products centered around the emerging iSCSI (Internet Small Computer System Interface) standard for linking data storage systems over Internet Protocol (IP), according to Mark Thompson, worldwide marketing manager for HP's networked storage division."

HP admits that users of Fibre Channel systems feel quite comfortable and have a stronger bent for the modernized technology of FCIP than for iSCSI. At the same time HP believes that the experience of working with solutions based on the IP protocol, especially with the Ethernet, will make iSCSI solutions more attractive for many customers.

IBM releases an iSCSI based product

The IBM TotalStorage™ IP Storage 200i provides storage that is directly attachable to an Ethernet LAN. This high-performance storage product implements newly emerging industry standards for the transport of SCSI storage protocols over IP, commonly referred to as iSCSI.

www.storage.ibm.com/snetwork/iSCSI/index.html

Well, let's make go into the details ourselves and establish our own pros and cons.

iSCSI

iSCSI (Internet Small Computer System Interface) is a TCP/IP-based protocol for establishing and managing connections between IP-based storage devices, hosts and clients.

SNIA - IP Storage Forum: http://www.snia.org/

iSCSI describes:

Transport protocol for SCSI which operates on top of TCP
New mechanism for encapsulating SCSI commands on an IP network
Protocol for a new generation of data storage systems that natively use TCP/IP

But it's known that rules of packet delivery differ for IP and SCSI. In IP packets are delivered without a strict order, it is also in charge of data recovery, which takes more resources. At the same time, in SCSI, as a channel interface, all packets must be delivered one after another without delay, and breach of the order may result in data losses. In spite of the fact that this problem, according to some experts, brings ambiguity into practical use of the iSCSI technology, today there are devices that prove its viability. The engineers developing the iSCSI managed to solve this problem to some degree. The specification of the iSCSI requires a longer packet's head. The head includes additional information which speeds up packet assembling by a great margin.

According to a senior system engineer of a Utah's university, the only obstacle for popularization of the Ethernet as a base technology for establishing storage area networks is a relatively great latency (close to 75 microseconds) because of the peculiarities of the TCP/²Ð stack. It can be a crucial problem in High-End systems in case of simultaneous access to thousands of files.

Experts working on iSCSI address the problem of latency with a careful attention. And although there are a lot of means developed to reduce influence of parameters which cause delays in processing of IP packets, the iSCSI technology is positioned for middle-level systems.

iSCSI develops quite rapidly. The need in a new standard was so strong that during 14 months after the proposal on the iSCSI by IETF in February 2000 we got a lot of devices demonstrating capabilities of their interaction. The Draft 0 on iSCSI published in July 2000 initiated realization of the technology. In January 2001 IP Storage Forum was created within the SNIA (Storage Networking Industry Association) which had already 50 members in half a year; and a product released in April 2000 soon won the Enterprise Networking Product Prize.

So, what is so attractive in the iSCSI for majors of the IT industry who do not even consider contradictions of this standard.

Here are the most important applications and functions which can be realized with data storage systems used:

Within the frames of those which can effectively be realized using modern methods:

· Consolidation of data storage systems · Data backup · Server clusterization · Replication · Recovery in emergency conditions Here are new capabilities which can effectively be realized with the IP Storage:

· SAN geographic distribution · QoS · Safety In addition, new storage area systems with the iSCSI being native for them provide more advantages:

· A single technology for connection of storage systems, servers and clients within LAN, WAN, SAN · Great experience of industry in Ethernet and SCSI technologies · Possibility of substantial geographic remoteness of storage systems · Possibility to use management means for TCP/IP networks To transfer data to storage devices with the iSCSI interface it's possible to use not only data carriers, communicators and routers of existent LAN/WAN but also usual network cards on the client's side. But it is followed by considerable expenses of processor power on the client's side which uses such card. According to the developers, the software iSCSI realization can reach data rates of Gigabit Ethernet at a significant, about 100%, CPU load. That is why it is recommended using special network cards which support mechanisms of CPU unload before TCP stack processing. At present (June 2002), such cards are produced by Intel.

The Intel PRO/1000T IP Storage Adapter (http://www.intel.ru/ru/network/connectivity/products/iscsi/index.htm) is offered at 700USD. It contains a powerful Xscale processor, 32M memory and transfers calculations related with iSCSI and TCP/IP and calculations of checksums of TCP, IP frames to the integrated processor. According to the company it can be as efficient as 500Mbit/s at 3-5% CPU load of a host system.

iSCSI under a microscope

Fig. 1. IP network with iSCSI devices used

Here, each server, workstation and storage device support the Ethernet interface and a stack of the iSCSI protocol. IP routers and Ethernet switches are used for network connections.

The SAN makes possible to use the SCSI protocol in network infrastructures, thus, providing high-speed data transfer at the block level between multiple elements of data storage networks.

The Internet Small Computer System Interface also provides a block data access, but over TCP/IP networks.

An architecture of a pure SCSI is based on the client/server model. A client, for example, server or workstation, initiates requests for data reading or recording from a target - server, for example, a data storage system. Commands which are sent by the client and processed by the server are put into the Command Descriptor Block (CDB). The server executes a command which completion is indicated by a special signal alert. Encapsulation and reliable delivery of CDB transactions between initiators and targets through the TCP/IP network is the main function of the iSCSI, which is due to be implemented in the medium untypical of SCSI, potentially unreliable medium of IP networks.

Below is a model of the iSCSI protocol levels which allows us to get an idea of an encapsulation order of SCSI commands for their delivery through a physical carrier.

Fig. 2. Model of lower levels of the iSCSI protocol

The iSCSI protocol controls data block transfer and confirms that I/O operations are truly completed. In its turn, it is provided via one or several TCP connections.

The iSCSI has four components:

iSCSI Address and Naming Conventions.
iSCSI Session Management.
iSCSI Error Handling.
iSCSI Security.

Address and Naming Conventions

As the iSCSI devices are participants of an IP network they have individual Network Entities. Such Network Entity can have one or several iSCSI nodes.

Fig. 3. Model of Network Entities

An iSCSI node is an identifier of SCSI devices (in a network entity) available through the network. Each iSCSI node has a unique iSCSI name (up to 255 bytes) which is formed according to the rules adopted for Internet nodes. For example, fqn.com.ustar.storage.itdepartment.161. Such name has an easy-to-perceive form and can be processed by the Domain Name System (DNS). An iSCSI name provides a correct identification of an iSCSI device irrespective of its physical location. At the same time in course of handling data transfer between devices it's more convenient to use a combination of an IP address and a TCP port which are provided by a Network Portal. The iSCSI protocol together with iSCSI names provides a support for aliases which are reflected in the administration systems for better identification and management by system administrators.

Session Management

The iSCSI session consists of a Login Phase and a Full Feature Phase which is completed with a special command.

The Login Phase of the iSCSI is identical to the Fibre Channel Port Login process (PLOGI). It is used to adjust various parameters between two network entities and confirm an access right of an initiator. If the iSCSI Login Phase is completed successfully the target confirms the login for the initiator; otherwise, the login is not confirmed and a TCP connection breaks.

As soon as the login is confirmed the iSCSI session turns to the FULL Feature Phase. If more than one TCP connection was established the iSCSI requires that each command/response pair goes through one TCP connection. Thus, each separate read or write command will be carried out without a necessity to trace each request for passing different flows. However, different transactions can be delivered through different TCP connections within one session.

Fig. 4. iSCSI Write example

At the end of a transaction the initiator sends/receives last data and the target sends a response which confirms that data are transferred successfully.

The iSCSI logout command is used to complete a session - it delivers information on reasons of its completion. It can also send information on what connection should be interrupted in case of a connection error, in order to close troublesome TCP connections.

Error Handling

Because of a high probability of errors in data delivery in some IP networks, especially WAN, where the iSCSI can work, the protocol provides a great deal of measures for handling errors.

So that error handling and recovery can work correctly both the initiator and the target must be able to buffer commands before they are confirmed. Each terminal must have a possibility to recover selectively a lost or damaged PDU within a transaction for recovery of data transfer.

Here is the hierarchy of the error handling and recovery after failures in the iSCSI:

The lowest level - identification of an error and data recovery on the SCSI task level, for example, repeated transfer of a lost or damaged PDU.

Next level - a TCP connection which transfers a SCSI task can have errors. In this case there is an attempt to recover the connection.

At last, the iSCSI session can be damaged. Termination and recovery of a session are usually not required if recovery is implemented correctly on other levels, but the opposite can happen. Such situation requires that all TCP connections be closed, all tasks, underfulfilled SCSI commands be completed, and the session be restarted via the repeated login.

Security

As the iSCSI can be used in networks where data can be accessed illegally, the specification allows fpr different security methods. Such encoding means as IPSec which use lower levels do not require additional matching because they are transparent for higher levels, and for the iSCSI as well. Various solutions can be used for authentication, for example, Kerberos or Private Keys Exchange, an iSNS server can be used as a repository of keys.

Others (iFCP, FCIP)

The IP Storage (IPS) work group was created within the frames of developing network storage technologies in the Internet Engineering Task Force (IETF); it has the following directions:

iSCSI (Internet Small Computer Systems Interface)
FCIP (Fibre Channel over TCP/IP)
iFCP (Internet Fibre Channel Protocol)
iSNS (Internet Storage Name Service)

In January 2001 IP Storage Forum was established within SNIA (Storage Networking Industry Association). Today the Forum includes three subgroups: FCIP, iFCP, iSCSI, each representing a protocol which is under the IETF protection.

FCIP - a tunnel protocol based on the TCP/IP which is designed for connection of geographically far FC SANs without affecting FC and IP protocols.

iFCP - TCP/IP based protocol for connection of FC data storage systems using the IP infrastructure together or instead of FC switching and routing elements.

iSCSI - described above...

For better understanding of positioning of these three protocols there is a diagram of networks based on them.

Fig. 5. Diagram of IP Storage networks

Fibre Channel over IP

The most revolutionary protocol among these three is Fibre Channel over IP. It doesn't bring in any changes into the SAN structure and organization of storage area systems. The main idea of this protocol is to make functional integration of geographically remote storage networks.

Here is the stack of the FCIP protocol:

Fig. 6. Lower levels of the FCIP protocol

FCIP helps to effectively solve a problem of geographical distribution, and integration of SANs on large distances. This protocol is entirely transparent for existent FC SANs and involves usage of infrastructure of modern MAN/WAN networks. So, if you want to merge geographically remote FC SANs with new functionality enabled you will have to get just one FCIP gateway and connection to MAN/WAN networks. A geographically distributed SAN based on the FCIP is taken by SAN devices as a usual FC network, and it is seen as a usual IP traffic for a MAN/WAN network it is connected to.

FCIP IETF IPS Working Group Draft Standard specifies:

rules of encapsulation of FC frames for delivery through TCP/IP;
rules of using encapsulation for creation of a virtual connection between FC devices and elements of an FC network;
TCP/IP environment for support of creation of a virtual connection and support of FC traffic tunneling through an IP network including safety, integrity of data and a data rate issue.

Here are some applied problems which can be successfully solved using the FCIP protocol: remote backup, data recovery and a shared data access. With high-speed MAN/WAN communications one can also use synchronous data doubling and a shared distributed access to data storage systems.

iFCP

Internet Fibre Channel Protocol is a protocol which provides FC traffic delivery over the TCP/IP transport between iFCP gateways. In this protocol an FC transport level is replaced with a transport of the IP network, the traffic between FC devices is routed and switched by the means of TCP/IP. The iFCP protocol allows connecting current FC data storage systems to an IP network with a support of network services which are necessary for these devices.

Here is how an iFCP protocol stack looks like:

Fig. 7. Lower levels of the iFCP protocol

According to the specification iFCP:

overlays FC frames for their delivery to a predetermined TCP connection;
FC services of message delivery and routing are overlapped in the iFCP gateway device; therefore, network structures and components of the FC do not mix in one FC SAN but are managed by the TCP/IP means;
dynamically creates IP tunnels for FC frames

An important feature of the iFCP is that this protocol provides an FC device-to-device connection via an IP network which is a more flexible scheme in comparison to the SAN-to-SAN. For example, if the iFCP has a TCP connection between pairs of N_Ports of two FC devices such connection can have its own QoS level which will be different from a QoS level of another pair of FC devices.

Conclusion

I'm quite sure that in the near future the Fibre Channel won't disappear and the FC SAN market will be further developing. At the same time the IP Storage protocols will make possible to use effectively storage area networks in those applications for which the FC can't provide an effective realization. With the FCIP and iFCP protocols data storage networks will be geographically distributed. And the iSCSI will make possible to use advantages of the SAN in the spheres which are still not or ineffectively realized within popular technologies.

P.S.

The rapid development of data storage networks is what the conception of the World Wide Storage Area Network based on. WWSAN provides for an infrastructure which will support a high-speed access and storage of data distributed all over the world. The conception is very close to the WWW but is based on different services. One of examples is servicing a manager who travels around the world with presentations. WWWSAN provides for transparent transfer of "mobile" data according to how their owner travels all around the world. Therefore, wherever such manager can be, he will always have a high-speed access to the data he needs, and an operation with them won't require a complicated ineffective synchronization via the WWW.

The conception of building the World Wide Storage Area Network excellently fits in the development of modern IP Storage technologies.

Terms and abbreviations:

SAN - Storage Area Network
CDB - command descriptor block.
PDU - Protocol Data Unit.
QoS - Quality of Service (usually describes a network through latency and band of a signal).
SNIA - Storage Networking Industry Association.
DNS - Domain Name Server.
PLOGI - Fibre Channel Port Login.
iSCSI - Internet Small Computer Systems Interface
FCIP - Fibre Channel over TCP/IP
iFCP - Internet Fibre Channel Protocol
iSNS - Internet Storage Name Service
WWSAN - World Wide Storage Area Network

Literature:

SAN after Fibre Channel, - Lucas Mearian
IP Storage Tutorial, - SNIA, http://www.snia.org/
iSCSI Technical White Paper, - SNIA, http://www.snia.org/
Internet Fibre Channel Protocol (iFCP) - A Technical Overview, - SNIA, http://www.snia.org/
Storage Forum, - HP EMEA, June 12-13, 2002.

Vladamir Savyak (svv@ustar.kiev.ua)

Write a comment below. No registration needed!