A FCoE Community: Welcome to our community for developers, engineers and resellers interested in FCoE technology innovation.

In a previous blog,  I discussed how enhanced Ethernet potentially improves the performance of iSCSI.  Combined with 10GBASE-T interconnect technology, iSCSI running over Ethernet enhanced by DCB (Data Center Bridging) is being seen by some as a cost-effective and viable alternative to FCoE.

iSCSI was originally designed to provide an inexpensive way to transport storage data over even unreliable Ethernet links.  Introduced more than a half decade ago, iSCSI has done well in the Small/Medium business segment.  However, iSCSI could not penetrate the data center because of its relatively poor performance and reliability compared to Fibre Channel, despite its appealing low cost.

There are several factors that have prevented iSCSI from extending its reach into Fibre Channel’s domain:

Higher Latency: By the nature of Ethernet, errors result in retransmission of packets.  As a consequence, packet latency can vary widely.

Lower Throughput: Retransmission of packets also erodes overall bandwidth capacity, resulting in lower packet throughput compared to Fibre Channel.

Shared Bandwidth: iSCSI storage shares the link with other types of traffic, further impacting latency and throughput.

Non-deterministic: Retransmission of packets, the fact that packets to the same destination can take different paths, and unpredictable bandwidth requirements from other applications increase system jitter.  As a result, transport of storage traffic is not deterministic.

Many of the components of DCB directly address iSCSI’s limiting factors and boost performance.  By creating a “lossless” network using DCB technologies, the overhead of transmitting iSCSI substantially decreases while at the same time increasing efficiency by eliminating bandwidth wasted on retransmissions.  In addition, prioritizing traffic flows with PFC (Priority Flow Control) and creating dedicated bandwidth allocations with Enhanced Transmission Selection (ETS) allows better control of shared bandwidth even when other applications are running.  Because TCP guarantees packet delivery from an upper layer, iSCSI technology does not have stringent requirements on the quality of the physical link and has a more relaxed BER, giving it the flexibility to support any level of storage applications.  Finally, the argument for iSCSI and 10GBASE-T in the data center is compelling from a cost perspective.

The important question is whether iSCSI has what it takes, even when enhanced by DCB, to provide the performance and reliability required in the data center.  Certainly, iSCSI latency and throughput can be improved substantially, but will these numbers be enough to compete with Fibre Channel in the data center and SAN?

There is also the belief throughout the network industry to consider that “All links eventually become Ethernet.”  This belief makes it an attractive proposition to move directly to Ethernet.  Given Ethernet’s increasing reach into the heart of the storage network, it is not unrealistic to suppose that the SAN will someday be entirely Ethernet based.  What’s up for debate is how close that someday actually is.

In any case, the lossless nature of DCB and enhanced Ethernet brings benefits to all Ethernet applications, not just FCoE in the data center or other converged network technologies.  Conventional Ethernet as a whole will benefit from the innovations being developed for converged networks.  From this perspective, DCB is not just an enabling technology for FCoE but a powerful foundation for a more reliable and higher performance network altogether.

It has been three years since FCoE was first introduced in the T11 standard.  And over these years, there has been growing interest in the Ethernet industry to adopt this new technology as the standard nears final ratification.

This month, there will be two plugfests to help bring the completion of this standard closer: the DCB Plugfest hosted by the Ethernet Alliance and the FCoE Plugfest hosted by the FCIA.  The two plugfests are a week apart – 5/17-21 for the DCB Plugfest and 5/24-28 for the FCoE Plugfest – and this is intentional.  FCoE is actually a combination of several technologies but can be thought of in terms of two primary components:

1. enhancements to Ethernet through DCB (Data Center Bridging) that allow Ethernet to carry Fibre Channel traffic by creating a lossless channel that meets the minimum specifications for Fibre Channel reliability and
2. bridging between Ethernet to the SAN/Fibre Channel.  At the two plugfests, manufacturers will be able to verify the performance and interoperability of FCoE from both sides of the channel.

Plugfests are critical to the successful implementation and adoption of FCoE.  The primary purpose of these plugfests is to ensure the interoperability of new FCoE equipment between different vendors.  These plugfests are also especially important for new players to the FCoE market as it allows them to ‘catch up’, so to speak, in terms of interoperating with all the other vendors.

However, plugfests are important beyond just demonstrating interoperability.  Specifically, they provide a visible milestone for the FCoE industry, giving positive proof of the progress that is being made to carry this standard from an abstract paper specification to being implemented in concrete products.  Together, they show the solidarity of the main players and their willingness to cooperate and, as a result, increase overall confidence in the ability of FCoE to serve the industry as promised.

The plugfests also enable manufacturers to see the potential and future of FCoE more clearly.  As each stage of the technology is proven to be real, the applications and possibilities it enables become more real as well.  One can already see the growing acceptance of and excitement for FCoE throughout the industry.

JDSU is taking a leading role in these plugfests and the general overall adoption of FCoE by designing the test plans for both plugfests.  Based on its extensive experience with Ethernet and Fibre Channel, the JDSU plugfest team has put together a comprehensive approach for not only verifying interoperability between FCoE products but also for helping manufacturers identify any problem issues so that they can comply faster.

JDSU’s approach to FCoE has been to recognize that both sides of the channel are based on different standards and so require different test metrics and troubleshooting techniques.  Because the Xgig Analyzer monitors and analyzes both Ethernet and Fibre Channel traffic on the same platform, this enables developers to correlate data as it crosses the Ethernet and Fibre Channel domains and allows end-to-end analysis and interoperability testing.

The DCB and FCoE Plugfests promise to be exciting as they bring us another step closer to the future of the SAN.  I’ll share more details of the plugfests in coming blogs.

DCB Plugfest: http://ethernetalliance.org/events/interoperability_test_events/data_center_bridging_plugfest

FCoE Plugfest: http://www.fibrechannel.org/component/content/article/159

Last time I wrote about the drive for a copper-based interconnect in FCoE-based data centers and the key factors impacting the decision process for whether to use SFP+ copper or 10GBASE-T links in lieu of SFP+ optical links.  I’ll address each of these factors – cost, performance, power, and reliability – in detail:

Cost: 10GBASE-T clearly has the cost advantage since it is the incumbent interconnect technology in today’s data centers.  Between SFP+ copper and SFP+ optical, SFP+ copper has a lower cost.  However, as I stated in my last blog, it is not enough to consider cost alone.

Performance: SFP+ optical is the clear winner for performance in terms of distance.  10GBASE-T also provides sufficient distance for most rack applications.  SFP+ passive copper cable, however, is turning out to fall short of expectations.  Although DAC was supposed to reach 20 m reliably when it was introduced over two years ago, many companies are discovering that they are only able to reliably reach 7 m with DAC.  This is unfortunate given that it is generally accepted that the minimal distance for rack level interconnect is 10 m.  The recent introduction of active copper (optical active cable) has helped overcome DAC’s distance limitations by extending the reach of DAC to at least 20 m. However, the cost and power consumption of active DAC is higher than passive DAC (see below table).

Power: 10GBASE-T, as it is implemented today, is simply not practical for high-density switch environments.  Typical power consumption is 8 W, with some implementations running as low as 4 W per port.  Power efficiency, however, is improving.  Certainly over time power efficiency will increase, especially given the overall drive to reduce power consumption across all network and communication applications.  While 10GBASE-T may not be ready from a power perspective for FCoE-based data centers today, at some point in the future it may be.

Power Consumption Comparison

This table compares the average power consumption of various interconnects. The discrepancy between 10GBASE-T and other interfaces is quite dramatic.

Reliability: Reliability is the cornerstone of Fibre Channel SANs, and it plays a significantly different role in FCoE networks than it does over traditional Ethernet.  Data always eventually arrives, whether over FCoE or Ethernet.  However, with Fibre Channel and FCoE the physical link is assumed to reliable enough to be considered “lossless”; i.e. there is no need for the protocol layer to concern itself with whether data arrived or not.  In contrast, TCP compensates for the assumed lossy nature of the physical link by retransmitting data at the protocol layers.

Over their respective distances, SFP+ copper and optical links are highly reliable with a bit error rate (BER) equivalent to that of Fibre Channel links. 10GBASE-T offers sufficient reliability when used with TCP but the noise level on the wire is high enough to limit reliability to a BER of 1×10^-12.  This is a full 3 orders of magnitude lower than the minimum BER required for FC of 1×10^-15.  Give the relatively lossy nature of 10GBASE-T, FCoE may be more sensitive to 10GBASE-T compared to TCP, making it difficult to maintain sufficient reliability for compatibility with Fibre Channel.

From a cost perspective, 10GBASE-T would be the ideal interconnect to use given its existing presence in the SAN, if only it offered sufficient reliability and more efficient power consumption.  Likewise, SFP+ DAC would be appropriate to use in more applications if it had sufficient reach.  Although more expensive, SFP+ optical links offer the best reliability and performance for FCoE-based data centers while still providing a cost-effective alternative to Fibre Channel.

Again, the performance, efficiency, and reliability of both DAC and 10GBASE-T will improve over time.  We anxiously expect to see 10GBASE-T-based FCoE products available on the market in the near future.

Part of the market appeal of FCoE is the promise of bridging the SAN and LAN with the reliability of Fibre Channel and the cost economies of Ethernet.  From a protocol development and interoperability perspective, manufacturers of FCoE equipment are on track to deliver fully capable and compliant devices for use in existing storage area networks.  There are still questions, however, as to which cable interconnect technology is most appropriate for use in FCoE-based data centers.

The point of contention is that there are a tremendous number of interconnects within a data center.  A key strategy in keeping price down for FCoE, then, is to use the lowest cost connector and cable that can still get the job done.  Since even a small savings per connector/cable becomes significant when multiplied over so many interconnects, the resulting reduction in overall cost to migrate a SAN to FCoE can be substantial.

Currently, the SFP+ connector is the primary interconnect for FCoE links and supports both optical and copper interfaces.  The optical interface offers the best signal quality and can reach as far as 300 m with multimode type fibre.  Alternatively, the copper SFP+ cable is expected to reach up to 20 m.  Ideally, copper interconnects are preferred because of their lower cost compared to optical interconnects.  These SFP+ copper cables are referred to as “direct attach cable” or DAC for short.  They are also known as “twinax”.  Due to signal integrity issues, passive copper twinax is reported to reach up to 7 m distance only.  Upgrading to active twinax cable improves the reach to 20 m.

In addition, given the dramatic cost advantage and potential reuse of some existing network infrastructure, there are those in the industry who are proposing the use of 10GBASE-T with a standard RJ-45 connector for FCoE links.   The idea of pushing 10GBASE-T for use in the FCoE network infrastructure is driven by the competing iSCSI storage technology. Recently, some industry leaders have announced new 10GBASE-T-based products to support data center applications and, in particular, for 10GbE iSCSI storage applications. With the assistance of enhanced Ethernet, iSCSI has improved the overhead efficiency and becomes a potential candidate for critical SAN applications. Leveraging the low cost as well as long reach of 10GBASE-T connections, iSCSI products potentially gain some positive momentum when competing with FCoE.  I personally think this is the main reason that these FCoE vendors are seriously considering adopting 10GBASE-T technology into their products.

Certainly using 10GBASE-T sounds like a good idea.  However, cost is only one parameter of the FCoE value proposition and, for networks handling mission-critical data, it is not necessarily the most important.  Concerns have been raised about the less reliable data integrity performance of the 10GBASE-T interface compared to twinax cable.  This is important because FCoE technology adopts a similar data processing mechanism and maintains the same level of data integrity, performance, and latency as native Fibre Channel. For FCoE to succeed as a disruptive technology, it must also provide the performance and reliability of Fibre Channel while at the same time reducing system cost.  Cost is certainly an important factor in the successful adoption of FCoE, but such success cannot come if performance and reliability suffer materially as a consequence.

There are other factors beyond equipment cost to account for as well, such as power efficiency.  Consider the following quote from Michael Vizard of Ziff-Davis’ Enterprise Technology Group: “Google engineers have already warned their bosses that the cost of the electricity needed to run the company’s servers will soon be a lot greater than the actual purchase price of the servers.”[1] Since even a small savings per connector/cable becomes significant when multiplied over so many connectors, the resulting power savings between different interconnect technologies can be substantial as well.  As a result, one of the biggest challenges of deploying 10GBASE-T technology will be its still too-high power consumption.

Each of these factors – cost, performance, reliability, and power – need to be balanced when determining which interconnect to use within an FCoE-based data center.  Next time I’ll look at each of these in depth to evaluate the viability of the use of SFP+ copper and 10GBASE-T links.

[1] http://serverdesignsummit.com/English/Conference/Conference_Info.html

There’s a new webinar available on demand at Lightwave! I’ve worked with Stephen Hardy, through even an earthquake (around minute 7) to talk about how Fibre Channel over Ethernet (FCoE) technology maps the impeccable advantages of Fibre Channel on performance, security, and effectiveness in terms of storage transportation onto Ethernet network. As such, FCoE requires also elevated enhancements (Data Center Bridging (DCB) protocols) on the conventional Ethernet to be more competent for critical data exchanges. The presentation will discuss from the testing perspectives, the challenges brought by this new technology, in particular, how the testing methodologies differ from LAN based Ethernet testing to SAN focused testing in converged Ethernet environments . The presentation will also present the the proof-of-concept test setup to demonstrate FCoE and DCB technologies and compare the performance differences with discrete networks.

Recently JDSU served as the technical lead for the SC09 demonstration sponsored by the Ethernet Alliance in November and was responsible for both designing the network architecture and managing the variety of test metrics used to verify compliance and interoperability.  JDSU also served as the sole testing tool provider at the third FCoE plugfest held November 16-20 by the Fibre Channel Industry Association (FCIA).

Interoperability is essential to successfully deploying FCoE technology in the field and building a robust tools and equipment ecosystem.  In order to verify interoperability, engineers need comprehensive testing tools that enable them to observe equipment behavior under real-world operating conditions.  Since the Data Center Bridging (DCB) and FCoE specifications are still evolving, these tools must be specifically designed to progress with the standards in order to enable early developers to conduct functional testing at each of the various protocol draft stages.  Developers also need to be able to verify performance at all protocols layers – including Priority Flow Control (PFC) response time, a key factor in enabling lossless Ethernet and FCoE – as well as ensure that the latency of the FCoE network is in alignment with native Fibre Channel requirements.

See our media alert on the JDSU news site for more details about our presence at these events!

JDSU today announced that it has completed its acquisition of the Network Tools business of Finisar Corporation.   The acquisition immediately establishes JDSU as the world’s leading provider of storage area network (SAN) protocol test tools, software and services.

“This acquisition provides JDSU with unique capabilities to address the SAN test market’s best growth opportunities, such as Fibre Channel over Ethernet,” said Dave Holly, president of JDSU’s Communications Test and Measurement business segment.  “We welcome the Network Tools team to JDSU and look forward to combining our respective strengths to deliver test innovation for the benefit of our customers.”

JDSU expects the business to contribute revenue in the mid-to-high single-digit millions of dollars and neutral operating income in the first quarter of fiscal year 2010.

To see the full press release, go to the JDSU site! And if you’re looking for our new home, go to jdsu.com/snt!

This past month, the FC-BB-5 working group of the T11 Technical Committee completed its work on the FCoE standard and approved the final standard. This vote stems back to November when a letter ballot for approval resulted in more than two-thirds votes in favor of the standard. Four members, however, had cast opposing votes. The committee continued work on the standard, addressing comments from both dissenting and affirmative ballots. As a result, during its plenary meeting on June 4, the T11 unanimously approved to forward the standard to INCITS for further processing and public review with no dissenting votes.

The goal of FCoE is to enable LAN and SAN data convergence in data centers and simplify the network by bringing multiple technologies under the umbrella of a unified interface. Instead of separating over their differences or overriding the concerns of a minority of members, the drafting committee has worked towards a resolution that addresses the issues of all members concerned. The solidarity of the standard’s approval is a welcome portent of the success of FCoE.

After completing their work on converging LAN and SAN has been done, T11 is now focusing on IPC traffic for cluster computing. During the week of T11 June meeting, a new protocol proposal, RDMA over FCoE, was presented for discussion. The goal of this new protocol is to maintain high data transmission performance by taking advantage of low overhead structure of Fibre Channel. In the meantime, IO convergence is to be achieved by FCoE protocol. If this is successful, we may finally see the end of InfiniBand.

Certainly there is much work left to be done to finalize FCoE as a robust standard capable of enabling true network convergence. However, with this announcement, another milestone has been passed on the road to FCoE, and we are another step closer to achieving true network convergence in data centers.

The final of the four testing challenges developers of FCoE network equipment face when verifying equipment behavior, performance, and responsiveness is being able to not only test both the LAN and SAN components of the network simultaneously but to also combine test results and expertise onto a single, converged platform.

Currently, the LAN and SAN are separate networks which have their own tools and different testing focuses. For example, LAN testing focuses on the switch and network only. SAN testing, in contrast, requires network-to-end-device verification as all hosts, fabrics, and targets contribute to the overall performance of the network. Given that the LAN fabric is based on best effort delivery while the SAN has complete traffic control end-to-end governed by Fibre Channel’s protocol for zero frame loss, LAN QoS testing does not apply to storage networks. Rather, SAN testing measures the performance of Fibre Channel links through mechanisms such as buffer to buffer credit management from the host through the fabric to the storage. In addition, as SAN testing must verify the delivery of every single frame, it requires a wholly different dimension of performance and latency measurements compared to LAN testing.

Convergence of the LAN and SAN networks, however, needs to extend beyond simply carrying traffic out to all aspects of network management and testing. As the LAN and SAN become a single logical network under FCoE, analysis and monitoring tools need to accommodate this convergence as well, enabling developers to utilize a single test and development platform.

A common misconception is that as the network has converged on Enhanced Ethernet, converged testing should follow Ethernet’s traditional QoS-based structure. However, since all the new, enhanced features are designed to enable storage data to be converged with LAN data with performance comparable to a separate SAN, testing must actually focus on the SAN’s more stringent requirements. Thus, when designing a test setup for a converged Enhanced Ethernet network, developers must step away from QoS-based testing, which is a measure of lossy network performance, in order to focus on performance testing of a lossless network with zero tolerance for packet drop.

The challenge of testing Enhanced Ethernet is how to combine the expertise required for testing lossy traffic with that required for testing lossless traffic. As the testing focus of these two different types of traffic are difficult to reconcile, the analysis instrumentation itself must be integrated as a converged platform. In this way, it becomes possible to test the various convergence properties such as the priority groups and traffic classes as well as the influences these have on the converged network as a whole.