Full-stack Philosophies

James Morle's Blog

RSS Feed

Sane SAN 2010: Fibre Channel – Ready, Aim, Fire

Posted on 10:45 am September 30, 2010 by James Morle

In my last blog entry I alluded to perhaps not being all that happy about Fibre Channel. Well, it's true. I have been having a love/hate relationship with Fibre Channel for the last ten years or so, and we have now decided to get a divorce. I just can't stand it any more!

I first fell in love with Fibre Channel in the late 90s: How could I resist the prospect of leaving behind multi-initiator SCSI with all it's deep, deep electrical issues? Fibre Channel let me hook up multiple hosts to lots of drives, via a switch, and it let me dynamically attach and detach devices from multiple clustered nodes without reboots. Or so I thought. The reality of Fibre Channel is that it was indeed a revelation in its day, but some of that promise never really materialised until recently. And now it's too late.

I have a number of problems with Fibre Channel as it stands today, and I'm not even going to mention the fact that it is falling behind in bandwidth. Whoops, I just did - try to pretend you didn't just read that. The real problems are:

  1. It is complex
  2. It is expensive
  3. It is unreliable
  4. It is slow

Complexity

Complexity. Complexity, complexity, complexity. I hate complexity. Complexity is the IT equivalent of communist bureaucracy - it isn't remotely interesting, it wastes colossal amounts of time, and it ultimately causes the system to go down. Don't confuse complexity with challenge - Challenge is having to solve new and interesting problems, Complexity is having to fix the same old problems time and time again and having to do it standing on one leg. So why do I think Fibre Channel is complex? For these reasons:

  1. The stack
  2. YANT

The Stack

If you have ever tried to manage the dependencies associated with maintaining a fully supported Fibre Channel infrastructure then you can probably already feel a knot in your stomach. For everyone else, let me explain.

Every component in a Fibre Channel stack needs to be certified to work with the other components. Operating System version, multipath I/O (mpio) drivers, HBA device drivers, HBA firmware, switch type, switch firmware and storage array firmware. So what happens when you want to, for example, upgrade your MPIO drivers? It is pretty standard for the following process to occur:

  • I want to upgrade to MPIO v2
  • MPIO v2 requires array firmware v42
  • Array firmware v42 requires HBA device driver v3.45
  • HBA device driver v3.45 requires the next release of the Operating System
  • The next release of the Operating System is not yet supported by the Array firmware
  • etc, etc

I think you get the point. But also remember that this wonderful array is shared across 38 different systems, all with different operating systems and HBAs, so the above process has to be followed for every single one, once you have a target release of array firmware that might work across all the platforms. If you are really really lucky, you might get a combination within those factorial possibilities that is actually certified by the array vendor.

Complex enough? Now add YANT...

YANT

Yet Another Networking Technology. I'm all in favour of having different types of networking technology, but not when the advantage is minuscule. All that training, proprietary hardware, cost, and so on: To justify that, the advantage had better be substantial. But it isn't. Compare Fibre Channel to 10Gbps Ethernet, which is a universal networking standard, and it just doesn't justify its own existence. To be fair to Fibre Channel, it was the original version of what we are now calling Converged Networking - it has always supported TCP/IP and SCSI protocols, and used to be way faster than Ethernet, but it just never got the traction it needed in that space.

Expensive

It's tough to argue against this one, Fibre Channel is expensive. 10Gbps Ethernet is also expensive, but the prices will be driven down by volume and ubiquity. In addition, Ethernet switches and so forth can be shared (if you must, that is: I'm still a fan of dedicated storage networks for reasons of reliability), whereas Fibre Channel must be dedicated. Infiniband is expensive too, and will probably stay that way, but it is providing a much higher performance solution than Fibre Channel.

Unreliable

What? Unreliable?

Yes, it's true. It's not an inherent problem with the technology itself; Fibre Channel is actually incredibly robust and I can't fault that fact. However, the promise of real-life reliability is shattered by:

  • Large Fabrics
  • Complexity

What is the point of large fabrics? I can see the point of wanting to stretch I/O capability over a wide area, such as remote replication and so forth, but that does not imply that the whole storage universe of the enterprise should be constructed as a giant fabric, does it? Networks should be composed of relatively small, interconnected,  failure domains, so that traffic can flow, but the impact of a failure is limited in scope. Building a large fabric is going against that, and I've lost count of the number of catastrophic failures I've seen as a result of building The Dream Fabric.

Complexity; we're back there again. Reliability is inversely proportional to complexity: High complexity = Low reliability, and vice versa. This is particularly true while we still entrust humans to administer these networks.

Slow

This is the final nail in the coffin. Times have changed, and Fibre Channel has no space in the new world. The way I see it, there are now just two preferred ways to attach storage to a server:

  • Ethernet-based NFS for general use
  • Infiniband-based for very low latency, high bandwidth use

The former approach is a 'high enough' performance solution for most current requirements, with ease of use and well understood protocols and technology. I'm not saying it's quicker than Fibre Channel (though it certainly can be), just that it is fast enough for most things and is easy to put together and manage. The latter method, Infiniband (or similar), is a step up on both Ethernet and Fibre Channel in both higher bandwidth and lower latency, especially when used with RDMA. Infiniband has been a technology searching for a commercial purpose for some time now, and I believe that time has now come, via the route of semiconductor-based storage devices. Consider the following numbers:

  • Fibre Channel Latency: 10-20us (est)
  • Infiniband/RDMA Latency: 1us (est)

Now let's see how these latencies compare to the those of a physical disk read,  and a read from a DRAM-based storage device:

  • Disk Read: 8,000 us (ie 8ms)
  • DRAM-based Storage read: 15us (source: TMS Ramsan 440 specification)
  • Ratio of FC latency to Disk Latency: 1:800 (1.25%)
  • Ratio of FC latency to DRAM Latency: 1:1.5  (80%)
  • Ratio of IB latency to Disk Latency: 1:8000 (0.125%)
  • Ratio of IB latency to DRAM latency: 1:15 (6.67%)

When comparing to disk reads, the Fibre Channel latency does not add much to the total I/O time. However, when accessing DRAM-based storage, it becomes a hugely dominant factor in the I/O time, whereas Infiniband is still single-digit percentage points. This is why I suggest that Fibre Channel has no role in the forthcoming high-performance storage systems. Fibre Channel is neither simple enough for simple systems, nor fast enough for high-performance systems.


8 comments on “Sane SAN 2010: Fibre Channel – Ready, Aim, Fire

  1. Hi James - I really enjoyed your blog entry. The point about latency across Fibre Channel vs Infiniband for SSD is particularly interesting.
    Do you think iSCSI still has a place at the commodity end? At the RAC & HA SIG I've seen very little take up of iSCSI. The ratio of responses to the storage question we ask is usually around 4:1 Fibre Channel to NFS and we're just beginning to see the odd delegate using Infinband (in the Exadata database machine).

    • HI David,

      Many thanks, glad you enjoyed it.
      I don't think iSCSI will stick around, but that's just an opinion. I don't think it matured sufficiently before FCoE came around, and I'm not even sure about FCoE! I've got another blog post in the wings that elaborates my opinion a little further, too.
      Cheers

      James

  2. Hi James
    First off I really enjoyed reading this article and I could certainly relate to many of the points you've raised. Ironically in hindsight I would say though that maybe a lot of these points are no longer valid now that there are platforms available to actually see the under utilisation and lower physical layer issues of the FC stack. I'd even be suggest that maybe FC is not at all expensive but greatly over-provisioned and poorly managed. Also it could be argued that maybe FC is more than adequate to meet the speeds of Enterprise mission critical applications - it's the imbalance of througput which is the problem?
    James I would love to have your insight and for you to share your experiences with myself and others on an upcoming webinar on the 30th March entitled, "FC SAN: Are You Making the Most Of It?"

    I hope you can make it!

    Here's the registration link:
    http://info.virtualinstruments.com/webinar-fc-san-myths.html

    • Hi Archie,

      I guess that you are coming at it from a different angle - where lots of underutilised systems are connected via FC to the storage. My viewpoint is from the selfish position of making large-scale database servers perform properly, and so it's a slightly different spin with lots of overlap.
      I still think that FC is dead, though, sorry :( . Even in your situation of increasing utilisation and so forth, I believe that 10Gbps Ethernet is a better fit. I also don't think that making better use of bandwidth availability has anything to do with lowering the latency, which was one of the main points of my blog entry :)
      Cheers

      James

  3. James- I'm pretty deep in storage, and I have to say, I disagree pretty thoroughly with your analysis.

    10 gig ethernet is expensive because the bits required to make it work reasonably are expensive. Because ethernet is not an arbitrated protocol (the clients are "dumb" and send whatever they want, whenever they want), ethernet switches have to do a lot of extra work to actually approach the theoretical throughput. The switch hardware required to get ethernet to scale throughput up actually increases exponentially with the link speed.

    8 gigabit FC has only been out for a bit more than 3 years, and yet we already see 8 gigabit FC switches which are less expensive than 4 gigabit FC switches were 3 years ago. 16 gigabit FC is out, and FC can continue to scale, while ethernet is largely stagnant because the per-port cost is not coming down.

    In terms of complexity, FC is different than ethernet, but not really all that more complex. Particularly if you add configuring iSCSI on top of ethernet, the complexity is largely the same.

    At a protocol layer, FC is actually *simpler* when wrapping SCSI, and in my opinion, complexity at the protocol layer (and additional "wrapping" protocols) is more deadly to functionality than complexity at the management layer.

    90% of the complexity I see in FC is designing systems without "single point of failure"- and correctly handling multipathing.

    • Hi Ross,

      Thanks for reading my blog, and thanks for your comments. I'm glad that somebody from a storage vendor has made a stand, though you clearly have a FC bias given that your company only makes FC products :)
      I think that you might have missed the main point of the article, though. My summary line is "Fibre Channel is neither simple enough for simple systems, nor fast enough for high-performance systems."
      I'm saying that at the low end it is too complex and Ethernet (specifically with NFS, not iSCSI) is a better fit. And at the high end it's not fast enough for emerging storage devices. Like all networking technologies (including Ethernet), it's prone to over complexity at design stage too.
      Cheers

      James

  4. Pingback: Quote of the day » Уста Явор Тревненски

  5. Pingback: Quote of the day » Master Yavor from Tryavna

Leave a Reply