Showing results for 
Search instead for 
Did you mean: 
Create Post

Paving the Road to Your Storage System

Level 9

A storage system on its own is not useful. Sure, it can store data, but how are you going to put any data on it? Or read back the data that you just stored? You need to connect clients to your storage system. For this post, let’s assume that we are using block protocols like iSCSI or traditional block storage systems. This article also applies to file protocols (like NFS and SMB) and to some extent even to hyper-converged infrastructure, but we will get back to that later.

Direct attaching clients to the storage system is an option. There is no contention between clients on the ports, and it is cheap. In fact, I still see direct attached solutions in cases where low cost wins over client scalability. However, direct attaching your clients to a storage system does not really scale well in number of clients. Front-end ports on a storage array are expensive and limited.

Add some network

Therefore, we add some sort of network. For block protocols, that is a SAN. The two most common used protocols are the FC protocol (FCP) and iSCSI. Both protocols use SCSI commands, but the network equipment is vastly different: FC switches vs. Ethernet switches. Both have their advantages and disadvantages, and IT professionals will usually have a strong preference for either of the two.

Once you have settled for a protocol, the switch line speed is usually the first thing that comes up. FC commonly uses 16Gbit and 32Gbit switches that have been entering the market lately. Ethernet, however, is making bigger jumps, with 10Gbit being standard within a rack or wiring closet and 25/40/100Gbit commonly used for uplinks to the data center cores.

The current higher speeds of Ethernet networks are often one of the arguments why “Ethernet is winning over FC.” 100Gbit Ethernet has already been on the market for quite some time, and the next obvious iteration of FC is “only” going to achieve 64Gbit.


Once you start attaching more clients to a storage system than it has storage ports, you start oversubscribing. 100 servers attached to 10 storage ports means you have on average 10 servers on each storage port. Even worse, if those servers are hypervisors running 30 virtual machines each, you will now have 300 VMs competing for resources on a single port.

Even the most basic switch will have some sort of bandwidth/port monitoring functionality. If it does not have a management GUI that can show you graphs, third-party software can pull that data out of the switch using SNMP. As long as traffic in/out does not exceed 70% you should be OK, right?

The challenge is that this is not the whole truth. Other, more obscure limitations might ruin your day. For example, you might be sending a lot of very small I/O to a storage port. Storage vendors often brag with 4KB I/O performance specs. 25,000 4KB IOps only accounts for roughly 100MB/s or 800Mbit (excluding overhead). So, while your SAN port shows a meager 50% utilization, your storage port or HBA could still be overloaded.

It becomes more complex once you start connecting SAN switches and distributing clients and storage systems across this network of switches. It is hard to keep track of how much storage and client ports traverse the ISLs (Inter Switch Links). In this case, it is a smart move to keep your SAN topology simple and to be careful with oversubscription ratios. Do the oversubscription math, and look beyond the standard bandwidth graphs. Check error counters, and in an FC SAN that has long distance links, check whether the Buffer-to-Buffer credits deplete on a port.

Ethernet instead of FC

The same principles apply to Ethernet. One argument why a company chooses an Ethernet-based SAN is because it already has LAN switches in place. In these cases, be extra vigilant. I am not opposed to sharing a switch chassis between SAN and normal client traffic. However, ports, ISLs, and switch modules/ASICS are prime contention points. You do not want your SAN performance to drop because a backup, restore, or large data transfer starts between two servers, and both types of traffic start fighting for the available bandwidth.

Identically, hyper converged infrastructure solutions like VxRail and other VMware VSAN place high demands on the Ethernet uplinks. Ideally, you would want to ensure that VMware VSAN uses dedicated, high-speed uplinks.

Which camp are you in? FC or Ethernet, or neither? And how do you ensure that the SAN doesn’t become a bottleneck? Comment below!

Level 13

Good Article


Nice write up

We use Ethernet on the front end, FC over the back end.  Servers have FC links separate from their Ethernet links, meaning we don't worry about backups causing application slowness or failures due to simple data throughput on servers' Ethernet ports.

We investigated FCOE; it wasn't ready for prime time when we looked into it eight years ago.  Technologies have further silo'd Fiber Channel as vendors fail or consolidate.  Where are the old Brocade and MacData or equivalent solutions today?  Operating under different brand names--or out of business, or running on VERY different GUI's (but still doing the same thing).  Where are the old tape-drive robots we used to use?  Dying or gone, replaced with resilient/redundant SSD (or other) technologies.

Change is the only constant, and everything old is new again.  How many times have we seen the old mainframe/dumb terminal come and go?  At least three times here.  Maybe more where you work?  It might be called Citrix or VDI, or might have been Solaris or something else.  Just as those old IBM terminals ran on green screens with Thin-Net or Thick-Net or Token-Ring, so too do we see changes coming to the network--just like always.

Level 14

Currently have Ethernet from client to server.  Servers are blades, run VMWare and connect to SANs via FC.  SANs have internal FC.  Moving to HCI on VxRail so should be interesting.  SAN monitoring software alerts on bottlenecks as does VMWare.  We are only moving because of the age of the current hardware and the ever increasing requirements of storage (this gives us a chance to add SSD as well as spinning disks to improve performance).  We also have seasonal variations on what we need so will be able to spin up servers to cover extra demand and add extra CPU and memory without having to add extra physical servers.  All this without going mad on overprovisioning. 

Level 20

Ethernet all the way now that iSCSI and NAS connectivity is so fast... we use multiple etherchannel interfaces to connect to storage most of the time.  Often it's 4 10G interfaces.

We use FC to our high-end SAN's. However, we got bit by the "tiering" feature a few years ago. Some of our mission critical systems got demoted to lower-level disks because they weren't as "busy" as some of the lower priority servers. It took us a while to figure out what was going on. We thought we were having hardware problems.

Level 9

I fully agree that we're on a pendulum, seeing old technologies emerge as something new again! Technology improvements in LAN speeds and HCI as an architecture is currently pushing us towards Ethernet. I wonder if/when FC will be revived though.. there's still plenty of developments there and we still sell a lot of SAN switches for existing and new installs. So it's definitely not dead (yet)...

Level 9

Does that mean you're (actively) phasing out old FC arrays and switches over time, to go 100% Ethernet? Or would you accept FC if someone comes in with a cheap FC solution?

Level 9

Yeah that is a pain in the behind in tiering systems! Depending on the system you might be able to restrict it to the upper two tiers of storage (e.g. SSD and FC). Some won't give you that option though, and you risk it falling from SSD straight down to NL-SAS when the monthly peak is over...

About the Author
Based out of the south of the Netherlands, working with everything from datacenters to operating systems. I specialize in storage, back-up & server virtualization systems and will blog about experiences in the field and industry developments. Certified EMCIEe, EMCTAe and EMCCAe on a number of Dell EMC products and constantly trying to find the time to diversify that list. When not tinkering with IT you can find me on a snowboard, motorbike or practicing a variety of sports.