Data Handling & Checksums

Transferring recorded material from the original media should never be done using a simple file copy. There are different software and hardware solutions available that will copy a file to one or several destinations and perform a checksum calculation, comparing the material on the original media to the copies, to provide a minimum effort safety check of data integrity. We strongly recommend using copy software and tools with checksum calculation and verification. Checksum files should be calculated and generated as sidecard files at the first copy process.

There are many different types of checksums, which are all meant for the same:
Safe and fast copy at the same time.

For example, there’s a checksum file called ‘MHL (Media Hash List)’. The structure of the file is fixed. It’s an XML based file which always contains the following information:

• creator info
  • Name
  • Username
  • Hostname
  • Tool
  • start date
  • finishing date
• hash
  • Path to file
  • filesize in bytes
  • last modification of the file
  • xxhash (calculated)
  • hashdate

The ‘creator info’ is placed at the beginning of every ‘MHL’ checksum file. The rest of the ‘MHL’ file contains every single file incl. the xxhash value, which was calculated at the copy & checksum generation process.

Changing a file or some values inside the ‘MHL’ checksum file results in an error during the checksum verify process. Checksums should always be generated during the offload of the original camera negative and should remain next to the files until the final backup process. We strongly recommend a copy and checksum workflow from beginning (e.g. On-Set) to end (e.g. final LTO backup).

The price for copy tools ranges from free to several thousand dollars, depending on the convenience and ease of use of the solution:

• Rsync is a free terminal application. It requires a computer expert to run a verified data transfer and even then, is rather cumbersome to use. If it is not already installed on your computer, you can find it at http://rsync.samba.org. On OSX and Linux systems its installed per default and you can run it from terminal. Please keep in mind, that “Rsync” cannot generate a so-called checksum file, it only compares files sizes.
• Imagine Products ShotPut Pro6 (from $49) http://www.imagineproducts.com
• Pomfort Silverstack +XT (from $59) or Silverstack LAB (from $199) is a professional software for DITs with all necessary functionalities regarding verified data copy and checksum generation. The program keeps all material in a database, can edit and export the metadata, can also create clip reports as PDF and provides a simple interface for quality control. There are different versions available. Silverstack is available at http://pomfort.com.
• Codex Digital introduced a device named Vault-XL, which follows their concept of the digital lab on the set, but with support for SXR/XR capture drives, SxS cards as well as CFast 2.0 cards. The fully featured version is a powerful standalone unit that handles a secure data transfer, can be used to directly playback or export dailies, provides a data management and metadata server. It also supports backup on parallel two or parallel four LTO-7 tapes. There's also a software only version available called Codex Production Suite.
• The Sync Factory B.V.  Hedge for Mac (from $49) https://hedge.video/