Mark Gritter (markgritter) wrote,
Mark Gritter

Geekout time: InfiniBand vs. Ethernet

I'm somewhat surprised to find Andy Bechtolsheim coming down on the side of Infiniband. Andy made his 2nd fortune by betting on Gigabit Ethernet when the hot new thing was ATM.

The core technical issue here seems to be that, after spending lots of money buying lots of SAN equipment (mostly FibreChannel), data center owners and operators have decided that it's stupid and expensive having one physical-layer network technology for storage and another for WAN/LAN traffic. I agree. So, the question is--- what will replace it?

I would probably come down on the side of Ethernet, more Ethernet, and faster Ethernet. 10Gb prices are coming down and it's a well-understood protocol. But, one difficulty is that Ethernet doesn't provide a congestion notification mechanism. Of course, modern Ethernet is switched, not CSMA/CD, but you still get packet drop when you shove packets into the switch faster than it can shove them back out again. So Ethernet is lossy, and this makes SANs unhappy. FibreChannel manages packet buffers so that connections are not lossy.

(I don't entirely get this part of the story... it is not at all clear to me that layer 2 is the right place to provide a "lossless" service. The end-to-end principle still applies. But it may be that storage-type applications benefit immensely from the additional L2 services. On the other hand, it's not clear to me how much credit to give FC for being "better" and how much for simply getting above 1Gb faster than Ethernet.)

So, one branch is to "fix" Ethernet. Define an encapsulation for FibreChannel and provide the additional services necessary to make Ethernet perform like FibreChannel. Then you can start rolling out the same 10Gb NICs to handle both storage and inter-server (or WAN) traffic. This is FibreChannel over Ethernet or FCoE. (Sun's a part of the FCoE effort, too.)

The other branch is to look for a new protocol. Sun will be releasing a lot of products supporting InfiniBand, which has so far seen use mainly in high-performance computing interconnects. InfiniBand has congestion-management features already, so it can carry storage traffic (maybe even encapsulated FibreChannel?) as well as IP interconnection traffic.

Why is Andy on the "new protocol" side this time?

Bechtolsheim noted that his former employer, Cisco Systems, is making a strong push for unified networking on Ethernet. However the unfinished state of the IEEE standards and the relative slow progress on standards for technologies such as Ethernet will be an impediment, he said.

"Ethernet is inherently a multi-vendor market and there are political and social problems with it," Bechtolsheim said. "People want to prevent anyone from getting a competitive edge, so they drag out the standards processes until everyone can catch up," he said.

"Cisco has a history of launching pre-standard products, but when you are connecting to the storage area network and the local area network that will be more challenging. The issue is performance and compatibility," he added.

While I agree with his arguments, I think there is a real possibility that in the medium-term the Ethernet-based approach is going to win. There is too much mindshare invested in FibreChannel and Ethernet to make the jump to a different technology. I think it will probably come down to time-to-market execution; if the Ethernet standard is ready "soon" then I think operators will wait for it.
Tags: geek, networking, technology
  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.