In the last two chapters we’ve looked at the options for recording data to internal and external media.
So, what do you do with all this data on a set?
Many things! You could:
- Make backups.
- Transcode to different versions for dailies, post production, broadcast, streaming, etc.
- Grade it and view on an external display.
- Study it with scopes or other devices.
- Delete it. Just kidding!
No I’m not. You can delete it, and if you work long enough in this industry, you will delete it. It’s never fun.
Memory cards are not tapes. They have a far higher failure rate, and have many compatibility problems, depending on the file system used, formatting applied, physical structure of the media, and a million other things we don’t really want to know. Even with all this unreliability, many ‘professionals’ shoot on memory cards without copying the files to a more reliable storage solution. I have seen projects where the shooter shoots for weeks without backing up. I find this kind of workflow irresponsible and unprofessional. But then again, it’s their life and career.
Let’s look at our options one by one:
One of the first things you should do as soon as you have recorded media is make backups of it. There is no excuse for not doing so.
You’ll need to figure out early how many backups you’ll need. Some thoughts:
- Are you using a bank guarantee or insurance?
- Will your footage be archived for many years?
- Will you need to read this data years from now?
If you’ve answered yes to any of these questions, you’ll appreciate a tape backup solution, the most common being the LTO (Linear Tape-Open) standard.
LTO tapes are not meant for regular reading and use, so this backup will go directly to storage unless there’s a catastrophe with your other backups.
LTO tapes are marked from 1 to 6, and the most widespread as of 2012 is LTO4 and LTO5. LTO6 has just been ratified according to Wikipedia, and you’ll find tapes hitting the market soon, if not already. The difference isn’t too great to get excited about. These tapes are meant to last for about 15 to 30 years.
What? Only 30 years? Want more? Then archive to film. Even though there are many claims of media storage products being able to survive for centuries, film is the only solution that has actually demonstrated its ability to last for at least a century under proper care. Even writing a 2K file over 16mm is possible, if one wants to save money on film.
Personally, I find LTO tapes a waste of time and money. My old cassette tapes still play great, twenty years after I bought them. I bet a properly cared for DVD will last at least 20 years, if not more. A well-maintained hard drive solution will offer greater reliability. It’s just like the natural world – Nothing lasts forever, so make copies to keep it going.
But what if you don’t need LTO tapes and long term archival solutions? Most productions have a shelf life of only a few months, let alone years.
Your post production facility might be anywhere from a million-dollar-an-hour boutique facility to a laptop in a corner of your apartment. Either way, it needs your footage in ‘one chunk’ to get started. This means, you need a storage solution that has all your footage in one physical location. For simplicity’s sake, I’ll call this Source Footage Storage (SFS).
The SFS could be:
- One drive
- Many drives in JBOD
- Many drives in basic RAID
- Many drives in a nested RAID
- A NAS or SAN cluster, usually in RAID
- The cloud, etc
If some of this doesn’t make sense, read AFRAID, a RAID Primer.
Considerations for an SFS:
- Good read speed
- Acceptable Redundancy
- Good drive space efficiency
Rule of thumb: On average, if your total footage is X (e.g. 300 GB), your SFS will need to be a size of at least 2X (600 GB). This means you should have at least one backup of your source footage, either as two drives with the same information or as a mirror (RAID 1) or as parity (RAID 5/6).
Do you need RAID? Yes, if you can afford to set it up. The point of RAID is that you can keep working even if a drive (or more, depending on your RAID level) fails. If that isn’t a concern, then just copy (duplicate) your source footage manually to another drive system – and wait for the day when it will become a concern.
Good, so you know you need at least two copies of your source footage for post production. Is that enough? No!
I recommend at least one more backup stored in a different physical location. Your SFS might be destroyed in a calamity, or stolen, or might decide to die, etc. Having a cloud backup won’t hurt either, if you have the connectivity. Remember, upload speeds are typically much lower than download speeds, so it might take ages for your footage to be ‘beamed up’.
To recap, here’s the rule of thumb: Always have at least 3 copies of your source footage – two in RAID for uninterrupted working, and the third in a different location as safety.
To give you an example, let’s say you have 10 hours of 50 Mbps source footage. That’s roughly 220 GB of data. Going by my rule of thumb, your SFS will need to be at least 440 GB (Actually 30% to 50% more for overhead and drive efficiency). And you’ll need a third drive which can hold 220 GB of data in a different physical location.
220 GB in the cloud? I don’t know. At a nominal speed of 2 Mbps, it will take about 11 days (24/7) just to upload all this data. I find it more sane (not to mention cheaper and faster) just to buy another reusable hard disk and store it in a third physical location.
Are we done? Maybe. Or maybe not.
Your Producer might need a Producer’s copy. Your VFX Supervisor might need a VFX copy. Your spouse might demand a copy to see what you’ve been up to all day. Whatever it is, multiply, and arrive at a fixed figure.
Knowing how many backups you need is only the first step. The second is to build a system that will take care of all these backups in the chaotic environment of a production set, reliably. Traditionally, this job description falls under the name: Data Wrangling.
The data wrangler’s office is a computer and storage bay that I’m going to call a Data Station.