A FCoE Community: Welcome to our community for developers, engineers and resellers interested in FCoE technology innovation.

It has been three years since FCoE was first introduced in the T11 standard.  And over these years, there has been growing interest in the Ethernet industry to adopt this new technology as the standard nears final ratification.

This month, there will be two plugfests to help bring the completion of this standard closer: the DCB Plugfest hosted by the Ethernet Alliance and the FCoE Plugfest hosted by the FCIA.  The two plugfests are a week apart – 5/17-21 for the DCB Plugfest and 5/24-28 for the FCoE Plugfest – and this is intentional.  FCoE is actually a combination of several technologies but can be thought of in terms of two primary components:

1. enhancements to Ethernet through DCB (Data Center Bridging) that allow Ethernet to carry Fibre Channel traffic by creating a lossless channel that meets the minimum specifications for Fibre Channel reliability and
2. bridging between Ethernet to the SAN/Fibre Channel.  At the two plugfests, manufacturers will be able to verify the performance and interoperability of FCoE from both sides of the channel.

Plugfests are critical to the successful implementation and adoption of FCoE.  The primary purpose of these plugfests is to ensure the interoperability of new FCoE equipment between different vendors.  These plugfests are also especially important for new players to the FCoE market as it allows them to ‘catch up’, so to speak, in terms of interoperating with all the other vendors.

However, plugfests are important beyond just demonstrating interoperability.  Specifically, they provide a visible milestone for the FCoE industry, giving positive proof of the progress that is being made to carry this standard from an abstract paper specification to being implemented in concrete products.  Together, they show the solidarity of the main players and their willingness to cooperate and, as a result, increase overall confidence in the ability of FCoE to serve the industry as promised.

The plugfests also enable manufacturers to see the potential and future of FCoE more clearly.  As each stage of the technology is proven to be real, the applications and possibilities it enables become more real as well.  One can already see the growing acceptance of and excitement for FCoE throughout the industry.

JDSU is taking a leading role in these plugfests and the general overall adoption of FCoE by designing the test plans for both plugfests.  Based on its extensive experience with Ethernet and Fibre Channel, the JDSU plugfest team has put together a comprehensive approach for not only verifying interoperability between FCoE products but also for helping manufacturers identify any problem issues so that they can comply faster.

JDSU’s approach to FCoE has been to recognize that both sides of the channel are based on different standards and so require different test metrics and troubleshooting techniques.  Because the Xgig Analyzer monitors and analyzes both Ethernet and Fibre Channel traffic on the same platform, this enables developers to correlate data as it crosses the Ethernet and Fibre Channel domains and allows end-to-end analysis and interoperability testing.

The DCB and FCoE Plugfests promise to be exciting as they bring us another step closer to the future of the SAN.  I’ll share more details of the plugfests in coming blogs.

DCB Plugfest: http://ethernetalliance.org/events/interoperability_test_events/data_center_bridging_plugfest

FCoE Plugfest: http://www.fibrechannel.org/component/content/article/159

Last time I wrote about the drive for a copper-based interconnect in FCoE-based data centers and the key factors impacting the decision process for whether to use SFP+ copper or 10GBASE-T links in lieu of SFP+ optical links.  I’ll address each of these factors – cost, performance, power, and reliability – in detail:

Cost: 10GBASE-T clearly has the cost advantage since it is the incumbent interconnect technology in today’s data centers.  Between SFP+ copper and SFP+ optical, SFP+ copper has a lower cost.  However, as I stated in my last blog, it is not enough to consider cost alone.

Performance: SFP+ optical is the clear winner for performance in terms of distance.  10GBASE-T also provides sufficient distance for most rack applications.  SFP+ passive copper cable, however, is turning out to fall short of expectations.  Although DAC was supposed to reach 20 m reliably when it was introduced over two years ago, many companies are discovering that they are only able to reliably reach 7 m with DAC.  This is unfortunate given that it is generally accepted that the minimal distance for rack level interconnect is 10 m.  The recent introduction of active copper (optical active cable) has helped overcome DAC’s distance limitations by extending the reach of DAC to at least 20 m. However, the cost and power consumption of active DAC is higher than passive DAC (see below table).

Power: 10GBASE-T, as it is implemented today, is simply not practical for high-density switch environments.  Typical power consumption is 8 W, with some implementations running as low as 4 W per port.  Power efficiency, however, is improving.  Certainly over time power efficiency will increase, especially given the overall drive to reduce power consumption across all network and communication applications.  While 10GBASE-T may not be ready from a power perspective for FCoE-based data centers today, at some point in the future it may be.

Power Consumption Comparison

This table compares the average power consumption of various interconnects. The discrepancy between 10GBASE-T and other interfaces is quite dramatic.

Reliability: Reliability is the cornerstone of Fibre Channel SANs, and it plays a significantly different role in FCoE networks than it does over traditional Ethernet.  Data always eventually arrives, whether over FCoE or Ethernet.  However, with Fibre Channel and FCoE the physical link is assumed to reliable enough to be considered “lossless”; i.e. there is no need for the protocol layer to concern itself with whether data arrived or not.  In contrast, TCP compensates for the assumed lossy nature of the physical link by retransmitting data at the protocol layers.

Over their respective distances, SFP+ copper and optical links are highly reliable with a bit error rate (BER) equivalent to that of Fibre Channel links. 10GBASE-T offers sufficient reliability when used with TCP but the noise level on the wire is high enough to limit reliability to a BER of 1×10^-12.  This is a full 3 orders of magnitude lower than the minimum BER required for FC of 1×10^-15.  Give the relatively lossy nature of 10GBASE-T, FCoE may be more sensitive to 10GBASE-T compared to TCP, making it difficult to maintain sufficient reliability for compatibility with Fibre Channel.

From a cost perspective, 10GBASE-T would be the ideal interconnect to use given its existing presence in the SAN, if only it offered sufficient reliability and more efficient power consumption.  Likewise, SFP+ DAC would be appropriate to use in more applications if it had sufficient reach.  Although more expensive, SFP+ optical links offer the best reliability and performance for FCoE-based data centers while still providing a cost-effective alternative to Fibre Channel.

Again, the performance, efficiency, and reliability of both DAC and 10GBASE-T will improve over time.  We anxiously expect to see 10GBASE-T-based FCoE products available on the market in the near future.

Part of the market appeal of FCoE is the promise of bridging the SAN and LAN with the reliability of Fibre Channel and the cost economies of Ethernet.  From a protocol development and interoperability perspective, manufacturers of FCoE equipment are on track to deliver fully capable and compliant devices for use in existing storage area networks.  There are still questions, however, as to which cable interconnect technology is most appropriate for use in FCoE-based data centers.

The point of contention is that there are a tremendous number of interconnects within a data center.  A key strategy in keeping price down for FCoE, then, is to use the lowest cost connector and cable that can still get the job done.  Since even a small savings per connector/cable becomes significant when multiplied over so many interconnects, the resulting reduction in overall cost to migrate a SAN to FCoE can be substantial.

Currently, the SFP+ connector is the primary interconnect for FCoE links and supports both optical and copper interfaces.  The optical interface offers the best signal quality and can reach as far as 300 m with multimode type fibre.  Alternatively, the copper SFP+ cable is expected to reach up to 20 m.  Ideally, copper interconnects are preferred because of their lower cost compared to optical interconnects.  These SFP+ copper cables are referred to as “direct attach cable” or DAC for short.  They are also known as “twinax”.  Due to signal integrity issues, passive copper twinax is reported to reach up to 7 m distance only.  Upgrading to active twinax cable improves the reach to 20 m.

In addition, given the dramatic cost advantage and potential reuse of some existing network infrastructure, there are those in the industry who are proposing the use of 10GBASE-T with a standard RJ-45 connector for FCoE links.   The idea of pushing 10GBASE-T for use in the FCoE network infrastructure is driven by the competing iSCSI storage technology. Recently, some industry leaders have announced new 10GBASE-T-based products to support data center applications and, in particular, for 10GbE iSCSI storage applications. With the assistance of enhanced Ethernet, iSCSI has improved the overhead efficiency and becomes a potential candidate for critical SAN applications. Leveraging the low cost as well as long reach of 10GBASE-T connections, iSCSI products potentially gain some positive momentum when competing with FCoE.  I personally think this is the main reason that these FCoE vendors are seriously considering adopting 10GBASE-T technology into their products.

Certainly using 10GBASE-T sounds like a good idea.  However, cost is only one parameter of the FCoE value proposition and, for networks handling mission-critical data, it is not necessarily the most important.  Concerns have been raised about the less reliable data integrity performance of the 10GBASE-T interface compared to twinax cable.  This is important because FCoE technology adopts a similar data processing mechanism and maintains the same level of data integrity, performance, and latency as native Fibre Channel. For FCoE to succeed as a disruptive technology, it must also provide the performance and reliability of Fibre Channel while at the same time reducing system cost.  Cost is certainly an important factor in the successful adoption of FCoE, but such success cannot come if performance and reliability suffer materially as a consequence.

There are other factors beyond equipment cost to account for as well, such as power efficiency.  Consider the following quote from Michael Vizard of Ziff-Davis’ Enterprise Technology Group: “Google engineers have already warned their bosses that the cost of the electricity needed to run the company’s servers will soon be a lot greater than the actual purchase price of the servers.”[1] Since even a small savings per connector/cable becomes significant when multiplied over so many connectors, the resulting power savings between different interconnect technologies can be substantial as well.  As a result, one of the biggest challenges of deploying 10GBASE-T technology will be its still too-high power consumption.

Each of these factors – cost, performance, reliability, and power – need to be balanced when determining which interconnect to use within an FCoE-based data center.  Next time I’ll look at each of these in depth to evaluate the viability of the use of SFP+ copper and 10GBASE-T links.

[1] http://serverdesignsummit.com/English/Conference/Conference_Info.html