EN
GUI of Pmfort Silverstack in dark mode listing clips of a porject

File Formats & Data Handling

Essential knowledge

  • Copy & Paste?

    Copying Data using MHL

    Transferring recorded material from the original media should never be done using a simple file copy. There are different software and hardware solutions available that will copy a file to one or several destinations and perform a checksum calculation, comparing the material on the original media to the copies, to provide a minimum-effort safety check of data integrity. We strongly recommend using copy software and tools with checksum calculation and verification. Checksum files should be calculated and generated as sidecar files at the first copy process.

    There are many different types of checksums, all for the same purpose: safe and fast copying. For example, there’s a checksum file called MHL (Media Hash List). The structure of the file is fixed. It’s an XML-based file which always contains the following information:

    • creator info
      • name
      • user name
      • host name
      • tool
      • start date
      • finishing date
    • hash
      • path to file
      • file size in bytes
      • last modification of the file
      • xxhash (calculated)
      • hash date
    Snippet of media hash list (white text on black background)

    The ‘creator info’ is placed at the beginning of every MHL checksum file. The rest of the MHL file contains every single file, including the xxhash value, which was calculated at the copy and checksum generation process.

    Changing a file or some values inside the MHL checksum file results in an error during the checksum verification process. Checksums should always be generated during the offload of the original camera negative and should remain next to the files until the final backup process. We strongly recommend a copy and checksum workflow from beginning (e.g. on set) to end (e.g. final LTO backup).


    The ASC MHL

    The ASC (American Society of Cinematographers) is working on a new MHL version (Version 2) that is called ASC MHL. All currently available documentations and specifications can be found on their website: https://theasc.com/asc/asc-media-hash-list

    Logo graphic of the ASC Media Hash List

    The ASC MHL is created for each copy that happens during production or post. This results in a chain of custody and damaged, renamed files and directories can be detected easily. ASC MHL is still in an early stage of introduction, but it looks very promising. 


    Tools for verified file transfers

    The price for copy tools ranges from free to several thousand dollars, depending on the convenience and ease of use of the solution:

    • Rsync is a free terminal application. It requires a computer expert to run a verified data transfer and even then, is rather cumbersome to use. If it is not already installed on your computer, you can find it at http://rsync.samba.org. On OSX and Linux systems it is installed per default and you can run it from terminal. Please keep in mind that “Rsync” cannot generate a so-called checksum file, it only compares file sizes.
    • Imagine Products ShotPut Pro http://www.imagineproducts.com
    • Pomfort Silverstack or Silverstack XT (from $59) is professional software for DITs, with all necessary functionalities regarding verified data copy and checksum generation. The program keeps all material in a database, can edit and export the metadata, can also create clip reports as PDF files, and provides a simple interface for quality control. There are different versions of Silverstack available at http://pomfort.com.
    • The Sync Factory B.V.  Hedge (macOS only) https://hedge.video
    • OWC Software Copy That (Beta) https://software.owcdigital.com/copy-that
    • Codex Digital introduced a device named Vault-XL, which follows their concept of the digital lab on set, but with support for SXR/XR capture drives, SxS cards, and CFast 2.0 cards. The fully featured version is a powerful standalone unit that handles a secure data transfer, can be used to directly playback or export dailies, and provides a data management and metadata server. It also supports backup on parallel two or parallel four LTO-7 tapes. There is a software only version available called Codex Production Suite.

  • ARRIRAW File Format

    ARRIRAW is ARRI’s format for uncompressed, unencrypted, and uncompromised sensor data. It can be considered a digital version of the camera negative. ARRIRAW is the only format that fully retains the camera's natural color response and great exposure latitude as unprocessed sensor data.

    Like film negative, ARRIRAW data has to be developed – or rather processed – in order to convert the single channel image, which represents the raw Bayer pattern sensor readout, into a color image suitable for normal viewing. The originally recorded raw data remains pristine, always providing the flexibility to go back and refine the results. This makes ARRIRAW the perfect format for digital cinematography and high-quality visual effects production.

    Our camera line-up supports in-camera recording of ARRIRAW or MXF/ARRIRAW data. MXF/ARRIRAW is ARRIRAW within a container format (*.mxf), allowing each clip to be contained in just one file, as opposed to several thousand files, because with the regular ARRIRAW format (*.ari) every single frame is a separate file.

    Older ALEXA cameras (ALEXA Classic, ALEXA Plus/Plus 4:3, ALEXA M, and ALEXA Studio) require an ARRI-certified external recorder which understands the T-Link signal.


    ARRIRAW Processing

    ARRI has built long-term relationships with postproduction equipment manufacturers through the ARRI Partner Program. These relationships have facilitated all of the leading compositing and color correction tools being able to process ARRIRAW files out of the box.

    In addition, ARRI offers a software development kit (SDK) for ARRIRAW processing, containing documentation of the ARRIRAW processing pipeline and a library – executable code – that the vendor can incorporate into their application. ARRI also supports vendors who wish to implement the ARRIRAW processing procedure on their own, through comprehensive documentation of a three-phase color processing pipeline and continuous direct support.

    Infographic explaining the certification process for 3rd party tool processing ARRIRAW


    Debayering
    The first phase of ARRIRAW processing is the most compute intensive. ARRIRAW images (like all camera raw images) have only one ‘color’ channel (in fact, it’s a color-coded luminance channel). A color reconstruction algorithm calculates the missing components for each pixel based on the type and position of colored filters on the camera sensor. ARRI cameras use the Bayer pattern color filter array. The term 'color reconstruction' therefore is also known as ‘debayering’. The Bayer pattern filters the light hitting the sensor so that 50% of the sensor’s photosites are used to represent green, 25% of the photosites represent red, and the remaining 25% represent blue. Therefore, the debayer algorithm needs to reconstruct 75% of red, 50% of green, and 75% or blue color information:

    Infographic showing the de-bayering process from luminance value-image to color array to rgb image

    The image above shows a single-channel capture from the sensor on the left, and on the right the reconstructed image facilitating the color filter array. Half of the reconstructed image's green values are interpolated from the surrounding photosites rather than captured, as are three-quarters of the red and three-quarters of the blue.

    The output quality of the image depends on the debayering algorithm. Generally speaking, a simpler algorithm will process faster, but will also involve a higher probability of color errors.

    For VFX, however, the images are often processed using the native sensor pixel count and then downscaled to e.g. 4K or 2K at a later stage. Using this approach takes advantage of the luminance resolution, which correlates to the sensor pixel count.

    ARRIRAW SDK and Third-Party Implementations
    In many cases, ARRI's SDK is fast and adaptable enough to satisfy the processing needs of an application. But in some cases, especially when the product uses custom hardware (or standard hardware in a non-standard way), Partner Program members may want to implement the ARRIRAW processing pipeline themselves. In this scenario, the SDK serves as a reference against which the member's developers can test their results before submitting evaluation imagery to ARRI's Workflow Group. In a few applications, the vendor's product offers both processing solutions, giving the user the option of maximum throughput with a very good match to the SDK, or bit-for-bit matching of other products that use the reference SDK implementation.


    ARRI Reference Tool (ART) replaces ARRIRAW Converter (ARC)

    The ARRI Reference Tool is a software application that provides a graphical user interface for ARRI’s reference SDK. ART is a combined tool for viewing, rendering, metadata, and look files. It combines and therefore replaces the ARRIRAW Converter, ARRI Color Tool, and ARRI Meta Extract.

    You can download your copy of the ARRI Reference Tool here.

    The legacy version of ARRIRAW Converter can be found here.

    wkflw_ARRIReferenceTool_screenshot_look_diver

    Testing Your Workflow
    ARRI recommends running tests to define:

    • How the intended look can be achieved using the chosen toolset(s).
    • The different tools that need to be set up to meet the production's expectations of consistent results.
    • The settings required for a chosen tool to most closely match the SDK reference results, if no other reference set of results is available.

    You can use an ARRIRAW frame grab as test material. ARRI provides sample footage shot with our cameras on FTP server and Webgate.io.



  • HDE Encoding

    HDE or CODEX High Density Encoding is a lossless, variable bitrate encoding scheme for ARRIAW data. It can reduce the footprint of ARRIRAW data by up to 40 – 50%, but deliver a bit-exact match to the original files when it's decoded. So, HDE does not compromise on quality.

    Supported by leading content providers and postproduction applications, HDE is widely used on ARRIRAW data from any ARRI camera.

    A graph showing the datarates of various recording formats of ALEXA Mini LF.


    HDE, like ARRIRAW, comes in two file types: as an MXF-wrapped clip for ALEXA 35 or as a single-frame *.arx file sequences for all previous cameras.
    The move to the MXF container significantly reduces redundancy from metadata that will be static throughout a clip. ALEXA 35 processing is fully aligned towards metadata transport in an MXF container. Single frame ARRIRAW and HDE therefore is only available for earlier camera models.


    HDE via CODEX Device Manager

    There are two options to convert uncompressed ARRIRAW data into HDE data:

    • CODEX Device Manager is a system extension for macOS which can automatically present a (virtual) HDE drive when a drive with ARRIRAW data is mounted. The virtual drive can be accessed with established data management tools which creates HDE files on the fly, as they are written to the copy target.
    • The ARRIRAW HDE Transcoder is a standalone app for macOS, Windows and CentOs, which needs to be pointed at a source drive or folder containing the ARRIRAW data and at a destination folder where it will create the HDE files.

    ALEXA 35 customers can use either option to convert MXF/ARRIRAW files to MXF/HDE files: CODEX Device Manager 7.0.4 and later or ARRIRAW HDE Transcoder.

    Customers with an earlier camera can only use CODEX Device Manager 7.0.4 and later to convert to *.arx or MXF/HDE files. Customers with an ALEXA Mini or AMIRA in addidtion need a CODEX CFast HDE License ("HDE for any CFast 2.0 reader") unless a CODEX CFast 2.0 reader is used to offload the camera data.

  • ProRes File Format

    Recording clips in Apple ProRes 4444 XQ preserves the full sensor quantization in logarithmic encoding, with the same range of colors available in ARRIRAW. Images recorded in a 4:4:4 codec are almost indistinguishable from uncompressed HD or UHD material. This makes internal recording attractive to feature film productions for the big screen, too. Recording in any of the high-end 4:2:2 codecs provides perfect source material for web or TV applications.

    In 2010, ALEXA was the first digital motion picture camera on the market offering in-camera recording of Apple QuickTime/ProRes files onto SxS PRO cards, providing the full range of codecs from Apple ProRes 422 Proxy to Apple ProRes 4444 and later 4444 XQ.

    In 2019, ALEXA Mini LF introduced a new method for Apple ProRes recording: sound, images, and metadata are no longer wrapped inside a QuickTime *.mov container but use the common MXF (Material eXchange Format) container.

    MXF is an open standard that is widely supported and allows for more flexible access to metadata. Using the same container also allows us to record ARRIRAW and Apple ProRes to the same drive without reformatting. Apple is supporting MXF/Apple ProRes. They have published the "SMPTE RDD 44:MXF – Mapping and Application of Apple ProRes" (on IEEE.org).

    ARRI cameras that currently support QuickTime/Apple ProRes will continue to use QuickTime.


    camera_faq_mxf

    Please Note:
    Apple ProRes is a variable bitrate codec, therefore a longer recording time than initially indicated on the camera may be possible.
    Apple ProRes White Paper (2022)

    wkflw_ProRes_compression_size