What is Data Wrangling?
Data wrangling is the process of copying, backing up, and organizing all the files created on a film shoot.
These files usually come from cameras, audio recorders, and sometimes other sources like drones or phones. On larger productions, a Digital Imaging Technician (DIT) manages this task. A DIT has training, special hardware, and dedicated software.
On smaller shoots, the budget may not allow to hire for this role. Yet, your footage and audio must be safe and well organized.
This guide explains how to handle data wrangling with a small, inexperienced team, where the responsibility falls on an assistant director or cameraman – or the director or cinematographer does it himself or herself.
I’ll try to recommend free software tools and affordable drives that work on both Mac and Windows computers. The goal is to keep the workflow as idiot-proof as possible. By the end of the shoot, the crew should have reliable backups and clear folders that make post production as smooth as it can be.
Let’s get started.
Drives and Formats
Every shoot needs external hard drives where the footage and audio will be copied.
How many drives do you need?
At least two drives are required. One serves as the main copy, and the other is the backup. A third drive is helpful if the production wants a handoff copy to take away at the end of each day.
To answer the question definitively, I recommend at least three drives, where the third drive is taken to another location for extra safety – daily or weekly.
What kind of drives do you need?
Portable spinning hard drives are cheap and hold large amounts of data. Solid-state drives (SSDs) are faster, more durable, and smaller. Either option works for short films and low-budget features.
Here’s how I would pick the right drive:
- Primary Drive: SSD over USB-C or Thunderbolt (the fastest you can afford between drive and Laptop interface).
- Secondary Drive:
- SSD if data rate is high, when you’re shooting RAW or Prores 4444 or above (or similar).
- Spinning drives if the data rate is low, and someone is available to man the transfer.
- SSD if the data rate is low, but there’s no one free to man the transfer, so it’s better to get it done quickly.
- Third Drive: The same as the Secondary Drive.
SSDs are best, though you can save some money if your secondary and third drives are just spinning drives. You can store a lot more data on these drives.
If you need a shortcut to find out the drive size and transfer speeds you need, check out the 15-minute rule below.
Tip: Buy a label sticker set and abel the drives clearly. A simple naming convention like “PRIME_A” and “BACKUP_A” prevents confusion. The letter changes as you have more than one drive. The third could just be called “OFFSITE_A” and so on.
What file format is best?
Drives should be formatted in exFAT. This is a file system that both Mac and Windows can read and write.
Format the drives before the shoot begins. Remove any bloatware or files on the drives and wipe them clean. Always confirm the computer recognizes the drives after formatting.
Try transferring some large files and check the actual transfer speeds. Cheaper drives get slower as they heat up, or as the cache fills up. Don’t go by the maximum speeds listed.
If you’re looking for the best SSDs, check out this guide I wrote:
How do You Organize Your Drives?
A folder structure is the set of rules for how files are stored. It just means your video and audio files go into specific folders so you can find them easily later.
A consistent structure prevents chaos. When someone who hasn’t been on the shoot looks at a drive, they should immediately know where to find camera files, sound files, and notes.
A simplest structure looks like this:
- PROJECT_NAME/
- DAY_01/
- CAMERA_A/
- CAMERA_B/
- SOUND/
- STILLS/
- DAY_02/
- CAMERA_A/
- CAMERA_B/
- SOUND/
- STILLS/
- DOCUMENTS/
- TRANSFER_LOGS/
- REPORTS/
- MISCELLANEOUS_FILES/
- DAY_01/
Each shooting day gets its own folder. Inside that folder, files are separated by source. Do not rename or reorganize the folders that come directly from the camera card or audio recorder. Copy them into the right place exactly as they appear on the card.
The “DOCUMENTS” folder holds everything that is not media. Logs, reports, or any notes from the set should be placed here. Keeping these items in a single location avoids scattering them across the drive.
The “STILLS” folder contains any photographs. “MISCELLANEOUS_FILES” is for anything that isn’t anticipated but you need to save it anyway.
A slightly better structure is if you can also have one more layer to separate files into scenes:
- PROJECT_NAME/
- DAY_01/
- SCENE #1
- CAMERA_A/
- CAMERA_B/
- SOUND/
- STILLS/
- SCENE #2
- CAMERA_A/
- CAMERA_B/
- SOUND/
- STILLS/
- SCENE #1
- DAY_02/
- SCENE #3
- CAMERA_A/
- CAMERA_B/
- SOUND/
- STILLS/
- SCENE #4
- CAMERA_A/
- CAMERA_B/
- SOUND/
- STILLS/
- SCENE #3
- DOCUMENTS/
- TRANSFER_LOGS/
- REPORTS/
- MISCELLANEOUS_FILES/
- DAY_01/
The typical low budget film goes straight to the edit. If an editor or editor’s assistant opens the drives, it’s much easier for them to find scenes and organize everything in the editing program. To them, “Days” don’t make any sense. Scenes are more important.
So why do I recommend “Days’ at all? Because it’s idiot-proof. Even if someone screws up due to pressure or negligence, the day’s files will at least be under that folder and it will make things a bit easier.
By the way, don’t make the camera reset the file names ones the project starts, so you don’t have duplicate file names and overwrite something by mistake.
Why Verification Matters
Copying files is not enough. Computers sometimes make errors while copying, especially with large video files. If a copy is incomplete or corrupted, the file may not play back later.
To prevent this, use checksum verification.
When software copies a file with checksum verification, it compares the original fingerprint with the copy’s fingerprint. If they match, the copy is good. If they do not match, something went wrong. This is the safest way to transfer footage.
Free Software to use to Copy and Ensure Checksum Verification
You’re in luck! Several free tools support checksum verification. They make copying and verifying simple, even for beginners.
DaVinci Resolve, the free version, has a feature called the Clone Tool. It copies from a source drive or card to one or more destination drives. It also verifies each copy with checksums. This feature works on both Mac and Windows.
So, rather than copy using Finder or Windows Explorer, you start Resolve and use that. Once you get the process it’s simple, and doesn’t take any additional time. As a bonus, you also get to see your footage on the laptop.
There are other tools, but don’t bother with them. The simplest choice is to use DaVinci Resolve on both Mac or Windows. Since the software is free, every crew member can install it. That way the process looks the same no matter which computer is used.
Don’t overthink it.
Step-by-Step Backup Process
The backup process should be the same every day. Repetition reduces mistakes. Here is a clear set of steps:
- Insert the camera card into the reader, typically connected to the laptop via USB-C.
- Open DaVinci Resolve. Go to the Media Management section and choose the Clone Tool.
- Select the card as the source.
- Select at least two drives as destinations, such as PRIME_A and BACKUP_A or whatever.
- Start the copy. Wait until the verification is complete. The faster the drives the lesser the wait.
- Compare the folder sizes of the source card and the destination copies as a quick extra check.
- Once verified, eject the card and return it to the camera crew.
- Copy to the third drive from PRIME_A, and so on. This can be a bit slower. You can set it to copy and walk away while it’s done.
- Fill out the transfer log. Record the card ID, file size, destinations, and your initials. This step is the least likely to be performed! But it’s just a way to ensure you have copied the drives and have not dreamt it!
- Place the completed log in the DOCUMENTS/TRANSFER_LOGS folder on both drives. See below about transfer logs.
- Do the same for photographs and audio files. If sound files need to be divided into different characters, or whatever, let the Production Sound Mixer do it at his or her end. All you do is copy the folders as is.
This process ensures every card is cloned to two or three drives with checksum verification. The log confirms that nothing was skipped.
Peace of mind.
What is the use of Transfer Logs?
A transfer log is a record of each card that was copied. It is both a checklist and a reference. Logs can be kept on paper or digitally in a spreadsheet. A digital log is easier to share.
Here is a simple log template:
- Date
- Card ID (such as A001 or B003)
- File size (in gigabytes)
- Copied to (list the drive names)
- Verified (yes or no)
- Operator name
Logs keep the process transparent. At the end of the shoot, the logs prove that each card was backed up to multiple drives and checked.
It’s a good habit to keep, but I’ve found in practice it’s one step too many for most assistants or main crew in all that pressure and lack of sleep.
Daily Workflow Example
Imagine a two-camera shoot with one sound recorder. On the first day, Camera A fills card A001, Camera B fills card B001, and Sound fills card S001.
It’s not necessary that all cards be handed over to the “designated data wrangler” at the same time.
The most practical thing to do is to divide your days into two sessions: Lunch and Wrap.
At lunch time and after the wrap, the data wrangler inserts A001 into the computer. Resolve clones it to PRIME_A and BACKUP_A, and hopefully OFFSITE_A. The log is updated.
The same process repeats for B001 and S001. By the end of the day, all three cards are verified and logged on two or three drives.
The cards are cleared only after two verified copies exist.
On day two, the same process repeats, but the files go into the DAY_02 folder. By the end of the shoot, the drives contain clearly organized folders for each day and each source.
The 15-minute Rule
This is for low budget filmmakers who can’t afford a DIT or Data Wrangler. The rule is:
I can’t be bothered to wait for more than 15 minutes to make a transfer!
Imagine you only have a 30-40 minute lunch break. Are you really going to spend all that time waiting for a drive to copy? Similarly, post wrap, how long do you want to hang around the set waiting for a copy? And it might not just be you, it might also be the owner of the cards and laptop, etc.
That’s why I’ve learned I want to ensure my work as a wrangler is restricted to 15 minutes or less. This puts a cap on the data and transfer speeds as well, and lets you reverse engineer what kinds of drives you’ll need.
E.g., let’s say you’re filming with a RAW codec with a data rate of 500 MB/s. All morning, you shot about 20 minutes of footage. The total data is about 585 GB.
Let’s say you’re using the Samsung T7 Shield 2TB SSD (Amazon, B&H) as your PRIME_A. It is advertised with a maximum sequential write speed of up to 1,000 MB/s. In real-world and benchmark tests, its performance is usually consistent. Still, let’s just say you’re going to get 850 MB/s.
If that’s the case, it will take you about 12 minutes to copy the footage. Add a few minutes to insert, eject and make a few mouse clicks, and you’re done within 15-minutes! Now you can go for lunch or head home by the time the crew has done loading everything. Hope you get the idea.
End-of-Shoot Checks
When production wraps, take time to confirm that everything is correct.
Once you’re back home there’s nothing much you can do if something had gone wrong anyway, so relax and mentally prepare yourself to spend a few hours. Check that each day folder exists on both drives. Compare the logs to the folders.
Make sure no card is missing. Look for duplicate or misplaced files.
If possible, keep the OFFSITE drives in a separate location. Having copies in more than one place protects against theft, fire, or drive failure.
Rules to Remember
Keep these rules in mind on every shoot:
- Never erase a card until two verified copies exist.
- Double check the above rule!
- Always back up to at least two drives.
- Always use checksum verification.
- Never rename or modify camera card folders.
- Always use transfer logs.
These rules are as simple as can be. You’re going to be copying your data anyway, just be consistent about it.
The goal is to create a safe and repeatable workflow. For a small crew with little data wrangling experience, clarity matters more than speed.
This way, even without a DIT or dedicated data wrangler, small productions can manage data safely. The tools are free. The drives are affordable. The steps are straightforward.
All you have to do is ensure you, or someone else, sticks to a clear routine.
How hard can that be? I guess you’ll find out!
