Data Management & Procedure Best Practices

Data Management

In principle, data management sounds simple: you download the camera media, put it on a hard drive and back it up. In reality, it is a process fraught with danger and rife with chances to mess up badly. Handling the recorded media is an enormous responsibility; a single click can erase an entire day’s shooting!

The film industry has spent decades developing methodical and careful processes regarding the handling of exposed film stock; loaders (the member of the camera crew who loads the film mags with raw stock and then downloads the exposed footage) are trained in standard methods and procedures; they keep extensive standardized paperwork and prepare the film in a prescribed way for delivery to the film lab, which in turn keeps a thorough paper trail of every roll of film. With digital video, in some ways the job is made more difficult by the fact that we are dealing entirely with computer files. If the DIT is not handling it the crew member who does this is titled Loader (the official term) but sometimes called the media manager, data wrangler or any one of a number of similar terms, sometimes depending on what area of the US or what country you are in.

Basic Principles

Certain core principles apply:
• Cover your rear.
• Have a standard procedure and be methodical.
• Maintain all logs.

Let’s talk about these in detail.

Cover your Rear

When everything is going OK, people rarely even notice what the Loader is doing; it seems routine and automatic. When something does go wrong, the entire production can turn into a blame machine. You don’t want to be the person who ends up bearing responsibility for a major disaster. The most important protection against this is, of course, to not screw up, but it is also important to be able to demonstrate that it wasn’t you. Film camera assistants have a long standing tradition of immediately and unreservedly owning up to mistakes they make: DPs, ADs and directors respect them for this. However, if it really wasn’t you that messed up, you need to have the procedures and paperwork to show it.

Some people interpret this rule as just being “avoid blame.” That’s not the point at all. The real issue is make sure nothing goes wrong so that there is no blame to go around. Protect the production from mistakes and you’ll be hired again and recommended to others. If you do screw up but immediately own up to it, the DP and production will still know that you are someone who can be trusted as part of the production process: mistakes happen — the point is to fix them and prevent them from happening again.

Standard Procedures

As camera assistants have learned over the decades, the number one way to ensure against mistakes is to have a methodical and organized way of doing things and then do it the same way every time. Observe any good crew of camera assistants working: there is almost a ritualistic aura to the way they work. They are also very focused and attentive to every detail at all times. Their repetitive actions are practiced and reliable. Their procedures are standardized industry-wide.

Maintain Your Logs

Film camera assistants have quite a few forms that need to be filled out and kept up to date: camera reports, film stock inventories, camera logs and so on. In the digital world, we are lucky that a great deal of this work is done by the camera and the various software applications we use immediately after the camera. Most downloading apps (such as Shot Put Pro, SilverStack, Double Data and others) also create logs that can track the files. Many Loaders also maintain separate logs either handwritten or more commonly as spreadsheets. An example of this type of log prepared by Jillian Arnold is shown below.

Procedure — Best Practices

By far the biggest danger is accidently erasing data which has no backup — this fear hangs over any operation where media is handled. Practices vary between different data managers and may be adapted or changed for various productions if the producer or insurance company requires certain procedures but they all have one basic goal: ensuring the safety of recorded data by clearly marking what media is empty and what media has recorded data on it.

One fundamental principle is that there should be one and one only person on the set who is allowed to format the media. This addresses the most basic of all dangers in digital workflow — being absolutely and unfailingly sure that the data has been downloaded and backed up and so it is safe to format. You certainly don’t want to have conversations on set like “Is this ready to format? I thought Danny already did that?” There is no room for uncertainty. Designating one person to be responsible for formatting helps keep this under control. It is not the entire solution of course; a rigid adherence to a standardized procedure is necessary also, just as camera assistants have always done.

The “business” section of Von Thomas’ DIT cart includes a large RAID for data ingest and management. He also sometimes networks his Mac Book Pro laptop as well for it’s Thunderbolt capability.

Locked and Loaded

One typical method is for the second AC to remove the media (whether SSD or card) from the camera and immediately engaging the record lock tab (if there is one). It is delivered to the DIT or Loader locked. After downloading, it is returned to the AC with the record lock still engaged. This way only the AC (the designated formatter, in this case) is authorized to disengage the lock, put the media back in the camera and format the card. This is one method only and different DITs and camera crews will have their own way of doing this.

Naturally there are variations on this procedure, such as when the camera crew doesn’t want to take the time to format media. This varies by the type of camera you are using. For example, it is very quick and simple to format a drive with the Alexa; on the other hand, formatting media for the Phantom takes quite a bit of time. What is important about this process is not so much who does it as it is that it be an established procedure understood by everyone on the crew and that it be religiously observed at all times: any deviation from procedure is always flirting with danger. Remember the basic religion of being a camera assistant: establish procedures and do it the same way every time — be methodical!

A Sony F3 with a Ki Pro mounted between the body and the battery. (Courtesy of Aja).

Signals —Disaster Prevention

Without a doubt, the greatest fear is that someone might erase/ format a card or hard drive that has footage on it that has not been stored elsewhere. There is no DIT, loader or camera assistant on earth that has not had this nightmare. The protection, as always, is to develop procedures, make sure the whole crew knows what they are and then stick to them.

Is this card ready for formatting? There are many systems but perhaps the most reliable is to use tape. Typically, green means “It’s OK to format.” Say something like “This drive is ready for formatting.” Red tape means “Not Ready for format.” Keep the red tape on it until you are absolutely sure it is finished and you have two tested backups. Use paper tape for marking cards and SSDs, not camera tape or gaffer tape. Camera tape can leave a sticky gum residue on the media and who wants to stick that into a camera? There are no hard and fast rules; it is whatever the crew agrees on. The important thing is consistency and communication. Many people make it a practice to not only mark the cards but to add a verbal signal as well, such as “these cards are ready for formatting.” Always putting them in a consistent location is important too. This might mean a small box on the DIT cart or something similar.

Always Scrub

Make it a habit to always scrub through (preview) the footage, even if only at high speed. A visual check is the only way to be certain the footage is good. You can also be watching for other problems — if you catch something no one else noticed, be sure to let them know. It is always best for a production to know about problems right away, when a reshoot is not a huge problem, as it will become once they have wrapped that location or set.

Do not scrub through the original media. This has two potential problems: one, having done that, you may think that you have downloaded the footage and two, it is what really gets downloaded to the hard drives that matters. It is usually best to use the camera companies software to preview/scrub the footage. This is going to have the least potential for problems. If you use some other software to preview and something doesn’t look right, then you can’t be sure if it is a problem with the footage or if there is just a playback issue and the material is fine.

Download>Scrub to check>Mark as ready for format.

Double Data takes a different approach to data ingest. It is designed to manage your data all the way through the process.

Three Drives

Most DITs consider three copies of the footage to be a minium. Hard drives die; files get corrupted. Backups are your only protection. Hard drives that are used to transfer the footage to post, archives or the production company are called shuttle drives. As an example, the hard drives might be:

• One for the editor.
• One backup for the client/producer.
• One backup for you (so you can be the hero when something bad happens).

An alternate process might be:

• All files on the DIT RAID drives.
• A shuttle drive of all files delivered to the producer.
• A shuttle drive of all files delivered to the post house.

The VFX people may also need files delivered. Obviously, the DIT can erase all drives on the DIT cart once the shuttle drives have been delivered to the producer/post house and checked, but many DITs prefer to keep the files live on their hard drives as long as possible (meaning until the next job) just as an emergency backup.

To be prudent, production and the post house should also make backups of all files as soon as they are received; it’s just common sense. Archiving to LTO tapes as soon as possible is a sensible method of protecting against unforeseen disasters. We’ll talk about LTO tapes in a moment.

Some productions do not allow the media to be erased until it has been confirmed by the post house; often this is a requirement of the insurance company. Some insurance companies will not give clearance to format the media until it has been transferred to LTO tape and stored in a vault. Obviously this requirement has a big effect on the amount of media (SSD drives, Phantom mags, compact flash, SxS cards, and so on) that need to be ordered and also has meaning for the planning of the DIT/post workflow. This again points to the importance of having a pre-production planning meeting which gets all of the parties together to work out the details of how it’s going to be done. Since downloading drives takes time, some DITs request that they get the media before it is filled up completely. Some crews make it a practice to only fill up drives half way for this reason. Figure 10.7. Double Data takes a different approach to data ingest. It is designed to manage your data all the way through the process.

Pomfort’s Silverstack offers a wide variety of features in addition to ingest: a clip library, transcoding, native playback, metadata search, user metadata and even clipping warnings and focus assist for help when viewing on small screens. It also generates reports as XML, HTML or PDF as well as Final Cut Pro XML files.

Do Not Drag and Drop

One principle that is universal no matter what procedures are in play: never “drag and drop.” Anyone who uses computers is familiar with the idea of grabbing a folder or file with the mouse and dragging it to a new location. It’s simple enough but for a couple of reasons, it is to be avoided with video files. Some cameras produce video files that are far more complex than just “, file.two” etc. There are often associated files in addition to the video clips.


What is missing when you merely drag-and-drop files is a confirmed check that the copied files exactly match the originals. One method of doing this is CRC — Cyclic Redundancy Checking. It sounds complicated, but it’s actually simple in principle. Copied files are divided into predetermined lengths that are divided by a fixed number, which results in a remainder, which is appended onto the copied file. When the check is performed, the computer recalculates the remainder and compares it to the transmitted remainder. If the numbers do not match, an error is detected and a warning displayed.

For example, Double Data remarks that a prominent feature of their data management software is that a big green check mark appears on screen when the files have been transferred and thoroughly double-checked for accuracy. There is a great value to this. Remember that working on the set is pressured, hectic, chaotic and often involves insanely long hours. There is a great value in having your software give you a clear and understandable signal that all is OK. All professional ingest software includes CRC.

Other File Copy Checks

A far simpler method is file size comparison, which is self -explanatory — the process merely compares the size of the original file and the copied file — it may be OK for a quick and dirty look but not nearly accurate enough for real verification. Two other widely used methods are MD5 Checksum or Message Digest 5; it’s similar to CRC but a bit more complex. It is an algorithm which creates an alphanumeric string associated with each file that is as unique as a fingerprint. It is commonly used when downloading files from the internet; by comparing the MD5 checksum you calculate against the published checksum of the original file, you can have some assurance that the file that arrives on your hard drive has not been tampered with in some way. This also applies to copying video/audio files. If the two MD5 checksums match up, there is little chance that fatal errors have crept in during the copy process.

Another method is byte verification, which takes file size comparison to the next level — it compares every byte of data to make sure the copy exactly matches the original.

Excerpt from The Filmmaker’s Guide to Digital Imaging for Cinematographers, Digital Imaging Technicians, and Camera Assistants by Blain Brown © 2015 Taylor and Francis Group. All Rights Reserved.

Related posts:

Tell us what you think!

Latest Tweets

Stay Informed

Click here to register with Focal Press to receive updates.

about MasteringFilm

MasteringFilm, powered by bestselling Routledge authors and industry experts, features tips, advice, articles, video tutorials, interviews, and other resources for aspiring and current filmmakers. No matter what your filmmaking interest is, including directing, screenwriting, postproduction, cinematography, producing, or the film business, MasteringFilm has you covered. You’ll learn from professionals at the forefront of filmmaking, allowing you to take your skills to the next level.