RDMA is an important technology in the data center as it allows servers to directly access each other’s memory without involving either server’s operating system.  As a result, it becomes possible to achieve high throughput with low-latency in massively parallel server clusters.  To facilitate clustering over Ethernet, the Internet Wide Area RDMA (iWARP) protocol was created to update the RDMA Consortium’s RDMA over TCP standard.  iWARP is typically implemented in hardware in the RDMA network interface card (NIC) and offloads TCP processing to increase efficiency.  iWARP, however, sacrifices performance because it is built upon TCP.  In existing deployments, efficiency is less than native InfiniBand links and so iWARP has had limited market penetration.

With PFC-enabled DCB Ethernet networks, the TCP layer is no longer needed for managing flows and securing packet delivery.  A direct RDMA mapping over the Ethernet MAC layer frame structure is defined in the protocol RDMA over CEE (RoCEE) to leverage DCB enhancements and reduce transport overhead.   The ETS protocol defined by DCB specifically addresses RDMA applications by supporting strict priority class 15 with absolute bandwidth availability in the converged network environment to provide low latency performance.  By being able to provide the throughput, low latency, and losslessness required to carry HPC applications, DCB enables the transport of HPC traffic over a converged network with better resource utilization, vastly improved performance compared to running iWARP over conventional Ethernet, more flexible resource allocation, a “greener” footprint,  and increased scalability.

By making Ethernet lossless, DCB improves the efficiency and reliability required for a converged network.  However, it is DCB’s versatility which makes it so well-suited for the data center.  By bringing together the LAN, SAN, and HPC networks into a single network topology, DCB will enable network administrators to consolidate network resources, personnel, and management tools for significant OPEX and CAPEX savings.

QCN specifies how to support congestion management of long-lived data flows within network domains of limited bandwidth-delay product, especially those supporting higher layer protocols that are particularly sensitive to packet loss or latency.  QCN enables bridging mechanisms that signal increasing congestion to end stations capable of limiting their transmission rate or pausing to avoid frame loss.  The protocol also extends the ability to bridges to use priority remapping to automatically “defend” a congestion notification domain against sources that are not aware of or support congestion notification.  The result is that long-lived data flows have a significantly reduced chance of frame loss compared to networks without some mechanism for congestion notification.

There is some debate as to the need for QCN given the capabilities of ETS and PFC to make Enhanced Ethernet completely lossless.  Link-by-link guaranteed bandwidth is provided by ETS and, with PFC, the lossless nature of Fibre Channel can be achieved over Ethernet using DCB.  The fact that native Fibre Channel has no equivalent of QCN suggests that QCN may not be necessary for FCoE.  From a deployment perspective, QCN requires that every device in the data path supports QCN.  Since QCN is implemented in hardware, not firmware, network administrators must purchase new equipment to implement it.

The delay of implementing QCN in FCoE devices is mainly because Fibre Channel SANs tend to be very flat with only 2 to 3 hops.  In contrast, Ethernet networks tend to have a hierarchical structure, and 5 or more hops is not uncommon.  This extra complexity can aggravate congestion issues and is the type of problem QCN is designed to address.  ETS and PFC do very well across a few hops but, for a reliable end-to-end FCoE link, QCN may be needed to manage congestion where the network is oversubscribed.

Is QCN essential for FCoE deployments?  This may be the wrong question.  Converged networks will bring together a wider variety of traffic types and applications than just those required to support FCoE, and QCN, as part of the Enhanced Ethernet enabled by DCB, plays an important role in improving overall network performance and reliability.  For example, while PFC and ETS can manage the congestion that can arise from oversubscription, the manner in which they do so can aggravate that congestion across every link along a data path.  QCN, in contrast, pushes congestion out to the edge of the network and slows the rate at which incoming packets are introduced until the congestion is relieved.  With this approach, a point of congestion is prevented from spreading to other links.

Another important role played by QCN is to support multiple disparate SAN that are aggregated over the same converged network.  While multiple virtual SANs can be mapped to a single PFC class (and thus treated as a single priority flow that requires no QCN), disparate SANs need to be configured into their own separate parallel PFC priority.  QCN is then required to manage congestions events between the various PFC classes.  In fact, QCN is needed to manage congestion between any disparate lossless traffic classes.  This is especially true given that as T11-BB-5 moved to T11-BB-6 almost a year ago, part of their focus was to expand the current scope of FCoE into long reach applications.  As FCoE technology moves deeper into core Ethernet networking, we should see more QCN implementation for FCoE applications as well.

In addition to FCoE, DCB enhances convergence Ethernet technology in a way that attracts other applications onto the unified network.  For example, iSCSI over DCB and RDMA over CEE are two emerging technologies that run over the DCB network.  I will talk more about RDMA over CEE in a later blog.  As for iSCSI over DCB, QCN plays a key role in maintaining the overall reliability of long distance communications.  The testing of QCN interoperability at the DCB Plugfest shows that manufacturers are working hard to bring the maturity of QCN up to the same level as PFC and ETS.

[1] http://www.communities.hp.com/online/blogs/eyeonblades/archive/2010/05/21/this-is-not-the-10gb-network-you-re-looking-for.aspx

In addition to all of the major SAN players you’d expect to see at the FCoE plugfest, there were many new players who attended with an interest in capitalizing upon the opportunities FCoE promises.  FCoE is proving itself as a reliable and cost-effective strategy for carrying storage traffic over Ethernet, and interest of investment in FCoE is high because FCoE opens avenues into new markets for many companies.  By implementing DCB and FCoE in their product lines, many traditionally Ethernet-based companies hope to have access into the SAN market that never existed before.

One of the purposes of these plugfests is to enable new players to catch up to the rest of the industry in terms of interoperability, and we are seeing that this is the case with both DCB and FCoE.  These new players bring new innovations, increase market choice and diversity, and will help to drive down equipment cost through market competition.  The wide interest and participation in these plugfests proves the viability of DCB and the converged applications it enables like FCoE, iSCSI over DCB, and low overhead High Performance Computing (HPC) using the recently announced RDMA over Converged Ethernet (RoCE) standard.  (For more information about RoCE, please refer to the recent press release, http://www.infinibandta.org/content/pages.php?pg=press_room_item&rec_id=663).

There can be no question that there is already tremendous market momentum for DCB and FCoE as FCoE continues to experience a wider adoption and acceptance among manufacturers.  Such broad industry support stands as a solid testament to the maturity of these technologies and the seriousness with which vendors are standing behind them. The efforts of equipment vendors to drive these technologies forward will also help accelerate adoption within the traditionally conservative data center market.

These plugfests also give data center managers confidence that DCB and FCoE are near-term technologies.  While plugfests are focused on proving out a technology rather than testing productized equipment, the success experienced at these plugfests is a strong indicator that compliant and deployable equipment will soon be available.  Currently, equipment deployment is still at an early stage with most network operators not turning on the FCoE capabilities of new 10GbE equipment.  However, given the fast pace of technology development and standardization, it won’t be a surprise to see faster adoption in data centers.

While FCoE is the primary application leveraging the advantages offered by DCB, DCB is a widely applicable technology and its adoption is not totally dependent upon FCoE deployment alone.  Converged Ethernet based on DCB technology also represents an opportunity for many manufacturers to enter markets they have traditionally avoided because it eliminates many barriers to entry.  The speed with which FCoE is maturing as a technology combined with the race between manufacturers to gain early market share will likely lead to a wide variety of equipment being available for initial deployment and thus accelerated adoption.

To avoid any possible confusion, it’s worth clarifying the difference between DCB and FCoE and why there were separate plugfests.  DCB is an infrastructure technology comprised of four primary protocols – Data Center Bridging eXchange (DCBX), Enhanced Transmission Selection (ETS), Priority Flow Control (PFC), and Quantized Congestion Notification (QCN).  Testing at the DCB Plugfest focused on the unified network and the ability to run various applications such as FCoE, iSCSI over DCB, and RDMA over DCB.  For its part, the FCoE plugfest focused on verifying FCoE as alternative storage network technology and its prerequisite Ethernet enhancements such as PFC and DCBX that are included in the DCB specification.  Testing at the FCoE plugfest probed deeply into the protocol and performance was compared against native network traffic.  For example, testing included verifying the login process with FIP, maintaining virtual links, and measuring read/write/mixed I/O performance for real storage applications.

Through both plugfests, the overall mechanisms as well as deep implementation issues of DCB were thoroughly tested.  Equipment from different manufacturers was connected to verify interoperability and compliance to the standards.  JDSU has long been a primary supporter of DCB and FCoE and stood at the center of bringing the interoperability process together, both by participating in the test definition process as well as by supplying the monitoring and analysis tools required to verify performance.  JDSU provided participates with its industry-leading Xgig platform, comprised of Analyzer, Jammer, Load Tester, and Expert analysis functions.  With these tools, not only could manufacturers verify that devices were performing as expected by manipulating traffic and capturing results for comprehensive analysis, they could immediately identify and troubleshoot the source of any interoperability issues that arose.

Both plugfests were highly successful.  Much progress has been made, including the following achievements:

Exciting interoperability and performance results: Participants were able to simultaneously demonstrate interoperability of their products while creating a lossless Ethernet fabric.  These results were impressive, given the number of vendors present and the fact that many of them were participating for the first time.

First time testing of Quality Congestion Notification (QCN): The plugfests were also the first time the industry has come together for interoperability testing of QCN.  QCN is an important element of DCB and Enhanced Ethernet.  I’ll discuss its role in a later blog.

Converged Network Testing: The purpose of DCB is to enable the converged Ethernet network.  Such a network has to run many different applications while respecting and maintaining the different traffic properties these applications require.  At previous plugfests, manufacturers had been able to verify both FCoE and iSCSI capabilities.  This time, participants also showed how DCB provides a robust and efficient foundation for high performance computing (HPC) running RDMA and iWARP.  I’ll discuss RDMA in a later blog as well.

DCB is well on track to serving as the underlying technology that consolidates the data center through a single, converged network.  The encouraging results of these plugfests, as well as the increasing support of industry leaders, continues to demonstrate the improving maturity of DCB-enabled Ethernet networks.

Next time I’ll discuss the impact of DCB and these plugfests on the industry as a whole.

iSCSI was originally designed to provide an inexpensive way to transport storage data over even unreliable Ethernet links.  Introduced more than a half decade ago, iSCSI has done well in the Small/Medium business segment.  However, iSCSI could not penetrate the data center because of its relatively poor performance and reliability compared to Fibre Channel, despite its appealing low cost.

There are several factors that have prevented iSCSI from extending its reach into Fibre Channel’s domain:

Higher Latency: By the nature of Ethernet, errors result in retransmission of packets.  As a consequence, packet latency can vary widely.

Lower Throughput: Retransmission of packets also erodes overall bandwidth capacity, resulting in lower packet throughput compared to Fibre Channel.

Shared Bandwidth: iSCSI storage shares the link with other types of traffic, further impacting latency and throughput.

Non-deterministic: Retransmission of packets, the fact that packets to the same destination can take different paths, and unpredictable bandwidth requirements from other applications increase system jitter.  As a result, transport of storage traffic is not deterministic.

Many of the components of DCB directly address iSCSI’s limiting factors and boost performance.  By creating a “lossless” network using DCB technologies, the overhead of transmitting iSCSI substantially decreases while at the same time increasing efficiency by eliminating bandwidth wasted on retransmissions.  In addition, prioritizing traffic flows with PFC (Priority Flow Control) and creating dedicated bandwidth allocations with Enhanced Transmission Selection (ETS) allows better control of shared bandwidth even when other applications are running.  Because TCP guarantees packet delivery from an upper layer, iSCSI technology does not have stringent requirements on the quality of the physical link and has a more relaxed BER, giving it the flexibility to support any level of storage applications.  Finally, the argument for iSCSI and 10GBASE-T in the data center is compelling from a cost perspective.

The important question is whether iSCSI has what it takes, even when enhanced by DCB, to provide the performance and reliability required in the data center.  Certainly, iSCSI latency and throughput can be improved substantially, but will these numbers be enough to compete with Fibre Channel in the data center and SAN?

There is also the belief throughout the network industry to consider that “All links eventually become Ethernet.”  This belief makes it an attractive proposition to move directly to Ethernet.  Given Ethernet’s increasing reach into the heart of the storage network, it is not unrealistic to suppose that the SAN will someday be entirely Ethernet based.  What’s up for debate is how close that someday actually is.

In any case, the lossless nature of DCB and enhanced Ethernet brings benefits to all Ethernet applications, not just FCoE in the data center or other converged network technologies.  Conventional Ethernet as a whole will benefit from the innovations being developed for converged networks.  From this perspective, DCB is not just an enabling technology for FCoE but a powerful foundation for a more reliable and higher performance network altogether.

This month, there will be two plugfests to help bring the completion of this standard closer: the DCB Plugfest hosted by the Ethernet Alliance and the FCoE Plugfest hosted by the FCIA.  The two plugfests are a week apart – 5/17-21 for the DCB Plugfest and 5/24-28 for the FCoE Plugfest – and this is intentional.  FCoE is actually a combination of several technologies but can be thought of in terms of two primary components:

1. enhancements to Ethernet through DCB (Data Center Bridging) that allow Ethernet to carry Fibre Channel traffic by creating a lossless channel that meets the minimum specifications for Fibre Channel reliability and
2. bridging between Ethernet to the SAN/Fibre Channel.  At the two plugfests, manufacturers will be able to verify the performance and interoperability of FCoE from both sides of the channel.

Plugfests are critical to the successful implementation and adoption of FCoE.  The primary purpose of these plugfests is to ensure the interoperability of new FCoE equipment between different vendors.  These plugfests are also especially important for new players to the FCoE market as it allows them to ‘catch up’, so to speak, in terms of interoperating with all the other vendors.

However, plugfests are important beyond just demonstrating interoperability.  Specifically, they provide a visible milestone for the FCoE industry, giving positive proof of the progress that is being made to carry this standard from an abstract paper specification to being implemented in concrete products.  Together, they show the solidarity of the main players and their willingness to cooperate and, as a result, increase overall confidence in the ability of FCoE to serve the industry as promised.

The plugfests also enable manufacturers to see the potential and future of FCoE more clearly.  As each stage of the technology is proven to be real, the applications and possibilities it enables become more real as well.  One can already see the growing acceptance of and excitement for FCoE throughout the industry.

JDSU is taking a leading role in these plugfests and the general overall adoption of FCoE by designing the test plans for both plugfests.  Based on its extensive experience with Ethernet and Fibre Channel, the JDSU plugfest team has put together a comprehensive approach for not only verifying interoperability between FCoE products but also for helping manufacturers identify any problem issues so that they can comply faster.

JDSU’s approach to FCoE has been to recognize that both sides of the channel are based on different standards and so require different test metrics and troubleshooting techniques.  Because the Xgig Analyzer monitors and analyzes both Ethernet and Fibre Channel traffic on the same platform, this enables developers to correlate data as it crosses the Ethernet and Fibre Channel domains and allows end-to-end analysis and interoperability testing.

The DCB and FCoE Plugfests promise to be exciting as they bring us another step closer to the future of the SAN.  I’ll share more details of the plugfests in coming blogs.

DCB Plugfest: http://ethernetalliance.org/events/interoperability_test_events/data_center_bridging_plugfest

FCoE Plugfest: http://www.fibrechannel.org/component/content/article/159

Cost: 10GBASE-T clearly has the cost advantage since it is the incumbent interconnect technology in today’s data centers.  Between SFP+ copper and SFP+ optical, SFP+ copper has a lower cost.  However, as I stated in my last blog, it is not enough to consider cost alone.

Performance: SFP+ optical is the clear winner for performance in terms of distance.  10GBASE-T also provides sufficient distance for most rack applications.  SFP+ passive copper cable, however, is turning out to fall short of expectations.  Although DAC was supposed to reach 20 m reliably when it was introduced over two years ago, many companies are discovering that they are only able to reliably reach 7 m with DAC.  This is unfortunate given that it is generally accepted that the minimal distance for rack level interconnect is 10 m.  The recent introduction of active copper (optical active cable) has helped overcome DAC’s distance limitations by extending the reach of DAC to at least 20 m. However, the cost and power consumption of active DAC is higher than passive DAC (see below table).

Power: 10GBASE-T, as it is implemented today, is simply not practical for high-density switch environments.  Typical power consumption is 8 W, with some implementations running as low as 4 W per port.  Power efficiency, however, is improving.  Certainly over time power efficiency will increase, especially given the overall drive to reduce power consumption across all network and communication applications.  While 10GBASE-T may not be ready from a power perspective for FCoE-based data centers today, at some point in the future it may be.

Power Consumption Comparison

This table compares the average power consumption of various interconnects. The discrepancy between 10GBASE-T and other interfaces is quite dramatic.

Reliability: Reliability is the cornerstone of Fibre Channel SANs, and it plays a significantly different role in FCoE networks than it does over traditional Ethernet.  Data always eventually arrives, whether over FCoE or Ethernet.  However, with Fibre Channel and FCoE the physical link is assumed to reliable enough to be considered “lossless”; i.e. there is no need for the protocol layer to concern itself with whether data arrived or not.  In contrast, TCP compensates for the assumed lossy nature of the physical link by retransmitting data at the protocol layers.

Over their respective distances, SFP+ copper and optical links are highly reliable with a bit error rate (BER) equivalent to that of Fibre Channel links. 10GBASE-T offers sufficient reliability when used with TCP but the noise level on the wire is high enough to limit reliability to a BER of 1×10^-12.  This is a full 3 orders of magnitude lower than the minimum BER required for FC of 1×10^-15.  Give the relatively lossy nature of 10GBASE-T, FCoE may be more sensitive to 10GBASE-T compared to TCP, making it difficult to maintain sufficient reliability for compatibility with Fibre Channel.

From a cost perspective, 10GBASE-T would be the ideal interconnect to use given its existing presence in the SAN, if only it offered sufficient reliability and more efficient power consumption.  Likewise, SFP+ DAC would be appropriate to use in more applications if it had sufficient reach.  Although more expensive, SFP+ optical links offer the best reliability and performance for FCoE-based data centers while still providing a cost-effective alternative to Fibre Channel.

Again, the performance, efficiency, and reliability of both DAC and 10GBASE-T will improve over time.  We anxiously expect to see 10GBASE-T-based FCoE products available on the market in the near future.

The point of contention is that there are a tremendous number of interconnects within a data center.  A key strategy in keeping price down for FCoE, then, is to use the lowest cost connector and cable that can still get the job done.  Since even a small savings per connector/cable becomes significant when multiplied over so many interconnects, the resulting reduction in overall cost to migrate a SAN to FCoE can be substantial.

Currently, the SFP+ connector is the primary interconnect for FCoE links and supports both optical and copper interfaces.  The optical interface offers the best signal quality and can reach as far as 300 m with multimode type fibre.  Alternatively, the copper SFP+ cable is expected to reach up to 20 m.  Ideally, copper interconnects are preferred because of their lower cost compared to optical interconnects.  These SFP+ copper cables are referred to as “direct attach cable” or DAC for short.  They are also known as “twinax”.  Due to signal integrity issues, passive copper twinax is reported to reach up to 7 m distance only.  Upgrading to active twinax cable improves the reach to 20 m.

In addition, given the dramatic cost advantage and potential reuse of some existing network infrastructure, there are those in the industry who are proposing the use of 10GBASE-T with a standard RJ-45 connector for FCoE links.   The idea of pushing 10GBASE-T for use in the FCoE network infrastructure is driven by the competing iSCSI storage technology. Recently, some industry leaders have announced new 10GBASE-T-based products to support data center applications and, in particular, for 10GbE iSCSI storage applications. With the assistance of enhanced Ethernet, iSCSI has improved the overhead efficiency and becomes a potential candidate for critical SAN applications. Leveraging the low cost as well as long reach of 10GBASE-T connections, iSCSI products potentially gain some positive momentum when competing with FCoE.  I personally think this is the main reason that these FCoE vendors are seriously considering adopting 10GBASE-T technology into their products.

Certainly using 10GBASE-T sounds like a good idea.  However, cost is only one parameter of the FCoE value proposition and, for networks handling mission-critical data, it is not necessarily the most important.  Concerns have been raised about the less reliable data integrity performance of the 10GBASE-T interface compared to twinax cable.  This is important because FCoE technology adopts a similar data processing mechanism and maintains the same level of data integrity, performance, and latency as native Fibre Channel. For FCoE to succeed as a disruptive technology, it must also provide the performance and reliability of Fibre Channel while at the same time reducing system cost.  Cost is certainly an important factor in the successful adoption of FCoE, but such success cannot come if performance and reliability suffer materially as a consequence.

There are other factors beyond equipment cost to account for as well, such as power efficiency.  Consider the following quote from Michael Vizard of Ziff-Davis’ Enterprise Technology Group: “Google engineers have already warned their bosses that the cost of the electricity needed to run the company’s servers will soon be a lot greater than the actual purchase price of the servers.”[1] Since even a small savings per connector/cable becomes significant when multiplied over so many connectors, the resulting power savings between different interconnect technologies can be substantial as well.  As a result, one of the biggest challenges of deploying 10GBASE-T technology will be its still too-high power consumption.

Each of these factors – cost, performance, reliability, and power – need to be balanced when determining which interconnect to use within an FCoE-based data center.  Next time I’ll look at each of these in depth to evaluate the viability of the use of SFP+ copper and 10GBASE-T links.

[1] http://serverdesignsummit.com/English/Conference/Conference_Info.html

Interoperability is essential to successfully deploying FCoE technology in the field and building a robust tools and equipment ecosystem.  In order to verify interoperability, engineers need comprehensive testing tools that enable them to observe equipment behavior under real-world operating conditions.  Since the Data Center Bridging (DCB) and FCoE specifications are still evolving, these tools must be specifically designed to progress with the standards in order to enable early developers to conduct functional testing at each of the various protocol draft stages.  Developers also need to be able to verify performance at all protocols layers – including Priority Flow Control (PFC) response time, a key factor in enabling lossless Ethernet and FCoE – as well as ensure that the latency of the FCoE network is in alignment with native Fibre Channel requirements.

See our media alert on the JDSU news site for more details about our presence at these events!