In Part One we ventured into the world of building a SAN storage for our production or post production workflows. In this part we’ll complete the listing of devices required.
Of course, the first requirement of a SAN is that you have multiple workstations, each like a young chick wanting a bite of the food that mother bird (SAN) is dangling before it. A workstation can be:
- An NLE (Adobe Premiere Pro, Avid Media Composer, FCP-X, etc.)
- A VFX system (Maya, Nuke, etc.)
- A full finishing system (Autodesk Smoke, Adobe After Effects, etc.)
- A color grading system (DaVinci Resolve, Speedgrade, etc.)
- A testing or viewing system
- A render machine or farm
- A sound editing or mixing system
And many more. I’m not going to include the costs of the workstation in our analysis. Some like Mac Pros, others like off-the-shelf products, and some just like to build their own. The workstation will need to connect to your network, but we’ll get into that in the next section.
As I mentioned in Part One a SAN is not just the storage array but also the network. It needs its own network, and that’s how it is designed to work. The first thing we have to figure out is how fast the network should be.
How to estimate network bandwidth requirement
Continuing with our test case, you’ll need a network capable of sustaining a rate of 600 MB/s (roughly 5 Gbps). There are five ways to go here:
- 1 GbE Ethernet
- 1 GbE multiplied (E.g. 4 x 1 GbE = 4 Gbps)
- Fiber Channel (From 2 Gbps to 16 Gbps)
- Fiber Channel over Ethernet (FCoE)
- iSCSI over 10 GbE Ethernet, or other forms of 10 GbE
1 GbE is surprisingly usable in small post houses. Walter Biscardi of Biscardi Creative Media has written about his experiences with a 1 GbE network here. Also read his follow-up article on the limits of 1 GbE. 1 Gbps means about 125 MB/s. In real-world use, you can get up to 100 MB/s on a well-designed system. That’s about 5 streams of Prores HQ at 220 Mbps. This is possibly the cheapest system to set up, but also the most restrictive. The moment you want to edit something like Red RAW or Prores 444 or uncompressed HD, etc., your system will come to a stand-still.
In the second case, what you’re doing is trying to combine four 1GbE Ethernet to get 4 Gbps. This process is called link aggregation or bonding. Bob Zelin has written about it here. In our example case, this will not work. In fact, I don’t recommend this architecture because it’s ‘temporary’ and high-maintenance, having the same limitations as 1 GbE. You’ll need to add more cables and switches as you scale.
Fiber Channel (FC) is the current king of the SAN world. It is also expensive, which is one reason why SANs are expensive. For what it’s worth, FC works, because it has been around long enough and most vendors and integrators know how to make it work. Don’t discount it from your considerations. FCoE, the fourth option, is not in widespread use.
iSCSI is newer technology, and a competitor to FC. It can be used with regular Ethernet technology. However, not everyone agrees that iSCSI is better, and FC still maintains top billing at least for now. Nobody really knows which way technology will swing. If you’re currently looking to develop your own SAN, you need to live on the bleeding edge of technology – you need to pick a side that might be champions today but out of the league a few years from now, or the other way around. Today, that war is between FC vs 10 GbE Ethernet. For further research, you can start with the following:
- FC 8 Gbps vs 10 GbE Ethernet (PDF)
- Comparing Performance Between iSCSI, FCoE, and FC
- FC and Ethernet at the same time
In our test case, we’ll need an 8 GFC FC (8 Gbps, 800 MB/s throughput, 1.6 GB/s in full duplex) network. If we’re going with 10 GbE, then that’s what we have. In either case, we’re aiming for a network with a bandwidth potential of 1 GB/s. I believe that any SAN you build today must be ready for at least this much bandwidth.
1 GB/s is 37 streams of Prores HQ or about 7 streams of 5K 3:1 Red Epic/2.5K uncompressed CinemaDNG footage. If you have compositing stations that demand more than this (editors rarely do), you could use DAS solutions for better speed. It does not make sense to spend money on the whole SAN just to feed one compositor. If you’re a VFX facility, then it’s another matter.
The network architecture
You have invited your guests (workstations and storage array) for dinner, now it’s time to place them on the table. Your choice of network is dependent on the underlying architecture (FC, iSCSI, 10 GbE Ethernet, etc.). I won’t go into detail, so we’ll only consider one use-case.
Under Fiber Channel, you’ll be arranging everything in what is called a Switched Fabric (FC-SW) architecture. Instead of stringing them along in one chain (if one fails the entire thing goes down), you connect each computer (workstation, node) to a Switch. Hence, ‘switched’ fabric. This is what it looks like:
This means you need a switch.
The switch is the mediator. Every node goes through the switch. All devices can talk to each other (if you allow it). The big four brands that make switches are:
- Cisco Systems
A completely redundant SAN will deploy two switches, so that if one fails the other continues the job as if nothing happened. One example of an 8-port 8 Gbps FC switch that isn’t very expensive is the QLogic 3810 FC switch.
Important: Each node that connects to a switch gets the full bandwidth. E.g., an 8 Gbps switch will potentially give 8 Gbps to each workstation, though it is limited by the speed of the storage array (600 MB/s or 5 Gbps in our example). You can have a faster array (more disks) and then use more than one connector (if your array throughput is greater than 8 Gbps), i.e., two or more per switch.
The switch has multiple ports to connect cables to. So, we need cables.
Connecting the storage to our network
Do you know the problem with USB 2.0 or HDMI connectors, where there is more than one kind of connector for the same thing? Fiber Channel is similar, but far worse. There are many kinds of connectors to choose from, of which two are common for SANs:
- SC Connector
- LC Connector
The LC connector is the one that you would probably be using. Many manufacturers prefer you use their entire SANs rather than just bits and pieces of them. This way, they can add some ‘secret sauce’ for which they charge more. It works, though. The QLogic switch I’ve referred to above is supposed to work with third-party SAN systems (They call it transparent routing).
In addition to this you also have the choice of wiring with copper or optical cable. Copper is used for short distances, whereas fiber optic cables are preferred for larger facilities where the server room might be in the basement and a workstation might on the third floor or whatever.
Then there’s the choice of Singlemode fiber vs Multimode fiber. You’ll be using the latter most likely because they are larger in diameter (50/125 and 62.5/125 in microns), faster and are only good for short distances. They are usually orange in color:
The choice of connector and cable type is entirely up to you, and the parts of your system. Even here, you have multiple choices that might make or break your system. You need to find a switch that you can control as a DIY system.
If you have two switches, your network will look like this:
Ah, but we’re not done yet. We left out two things in the Storage and Workstation parts. Now it’s time to add them.
The Host Bus Adapter or Network Interface Card
None of the fiber optic connectors are available on workstations. Where do you connect that cable then? You’ll need to create a new port, which means you’ll need a PCIe (or similar) card that plugs into the motherboard. This is called the Host Bus Adapter (HBA).
One HBA that supports full duplex 8 Gbps is the ATTO Celerity FC-82EN:
You’ll need one for each workstation or storage array. The costs can add up quickly!
If you’re on 10 GbE instead of FC, you don’t need an HBA. But you need a Network Interface Card, or NIC. These tend to be cheaper than HBAs, sometimes by half or more. All the manufacturers that make switches also make HBAs. If you’re using iSCSI, then the NIC won’t be so simple. It will need a special hardware component called an ‘initiator’ that separates it (read: more expensive) from cheap NICs. There’s no free lunch.
As you have seen, each and every hardware choice you make must be part of an overall plan. It is critical that you have the entire thing mapped out before you even begin to buy anything.
So far, we’ve looked at parts that can be bought online and assembled together. That was the easy part. The hard part is finding the right software and using it to take advantage of the network as you’ve designed it.
This means, you shouldn’t put together a SAN until you know which software to run and how to run it. So, what else is new?
Don’t you wish these things were plug-and-play? They’re not. When you install an HBA, you’ll also need to install its software. Once each HBA is installed, you’ll need to check if the firmware is up to date (just like you’d do on a GPU). Here’s an article on how to set up HBAs.
What about the storage? Your storage array might be one large logical drive, or might be partitioned. In either case, it needs the LUN treatment. The logical segments (LUNs) need to be configured and zoned so that a central server can control who can read and write to what. You don’t want somebody writing over your source footage!
Anyway, when you buy a storage array, it might be limited by the number of LUNs you can create. This in turn affects the number of workstations that can connect to it.
The software that manages your storage array is critical. You may have heard of FreeNAS, which is a very popular open source solution for DIY NAS builds. Similarly, you need to find the SAN equivalent. The ideal software should work with HFS+ (Mac) and NTFS (Windows) file systems at the same time. One such operating system is Openfiler, which is both a NAS and SAN.
If you want to run your own operating systems (which is what you’ll want, really), then you need a ‘layer’ that runs over the OS that makes each node see all files as one. Such a software is called a shared storage file system or SAN file system.
They run over the operating system and create a file-system that can be read by everybody. What this means, though, is that this software must be installed on every workstation in the network. These are usually licensed per node – the more the computers, the more the licensing. It adds up!
Some names are:
- Red Hat GFS
- VMWare VMFS
- QFS, from Sun
What you really need is a file system that has block-level access. Otherwise your whole SAN is a waste of time. XSan, which is compatible with StorNext, is free with OS X (you still have to pay for OS X, though!).
A metadata controller is software that takes care of file locking among other things. E.g., if more than one person wants to read the same file at the same time, that’s cool. But what if they want to write to the same file at the same time? That’s not cool. Software like XSan and metaSAN includes a metadata controller.
It is not uncommon for metadata (this is not video metadata, mind you) to be transferred by a different network, or the Ethernet network that connects all the workstations at the facility. This keeps it ‘safe’.
These are just the basic things. Configuring each software, making sure all the devices in the network can see and communicate with each other, and then ensuring the total bandwidth is utilized in the most efficient manner – these tasks are not easy, even for experienced pros.
For anyone trying to build a SAN, configuring software is the most nerve-racking and frustrating part. Be prepared.
In Part Three we’ll look at the costs of putting together a DIY SAN, and then compare some of the readily available solutions in the market. That should give you enough information to get started.