GUI of Pmfort Silverstack in dark mode listing clips of a porject

File Formats & Data Handling

Essential knowledge

  • Copy & Paste?

    Original camera data never should be transferred using a simple file copy (e.g. with macOS Finder, Windows Explorer or a simple command line copy). To protect yourself from a really bad experience – like having lost a day's worth of footage because missing or corrupt files weren't noticed before the original recording media had been erased – your minimum standard should be checksum-verified data backups.

    Checksums

    A checksum is a small block of data that represents a file for the purpose of detecting errors that may have been introduced when the file was transmitted. There are different types of checksums, which are calculated with different algorithms, but all for that same purpose.
    A data management tool which offers a "checksum verify" will calculate and compare the checksum of a file versus the checksum of its copy. The result of the verification (verified or failed), along some additional information then can be stored in checksum directory listing called media hash list (MHL).

    MHL

    A Media Hash List is an XML file which contains:

    • the creator info (name, user name, host name, tool, checksum calculation start date and finishing date) followed by
    • the hash for each file that was processed (path to file, size in bytes, last modification date, hash value, hash creation date).

    The next time the files will be copied, the hash values from the MHL file will be used as the reference for the verification. If any file was modified in any way (i.e. corrupted or edited), the checksums will no longer match and the verification will fail. Checksums therefore should be created as early as possible, i.e. when the original camera data is copied from the recording media and should be kept along with the original files all the way into archival.

    Though most agree that checksums are vital, different tools, companies or individuals may utilize any of the verification methods/checksum types. This makes it difficult to track where a corruption or unintended modification occurred within the production’s lifespan.

    To overcome this issue, a sub-committee of the American Society of Cinematographers, together with members from studios, software companies, DITs, and post facilities, developed an extension to the original MHL concept, the ASC MHL (version 2).
    In a nutshell, the ASC MHL refines the XML structure of the MHL file and adds a so-called "chain file", which serves as an MHL history. The chain file keeps track of the path and file hash of each ASC MHL files so their integrity, in turn, can be verified. This enables productions to exactly determine, where in the chain the problem occurred.

    More information on the ASC MHL can be found on the ASC website.

    Data Management Software

    The following is a list of data management tools offering the recommended level of security though verification. Note that this list may not be exhaustive!

    • Pomfort offers a range of products for media asset management and digital imaging. Its entry-level software for data management is Offload Manager, while at the top end its feature-packed Silverstack Lab tool offers clip management, QC, and dailies. Pomfort software is macOS only.
    • Imagine Products focuses on data backup to disk and LTO. Its data management software is called ShotPut Pro and is available for macOS and Windows.
    • Hedge offers tools for data management, reporting, transcoding, review, and archiving. Its copy tool is called OffShoot (formerly Hedge). Hedge software is available for macOS, iPadOS, and Windows. 
    • YoYotta software facilitates full offload, transcode, conform, archive, and restore functions.
    • OWC, manufacturer of professional storage hardware solutions, also offers software for backup, disk management, and archival. Its data management software is called Copy That (macOS only).
    • Quine offers the free Quine Copy software, providing basic copy and media management features, as well as transcoding options.
    • OTTOMATIC o/PARASHOOT is not a copy tool but ensures a secure offload and erasing process on set by checking if files on a card were already copied. It also offers a reversible process for erasing the camera media, so long as the drive was not erased in-camera.
  • ARRIRAW File Format

    ARRIRAW is ARRI’s format for uncompressed, unencrypted, and uncompromised sensor data. It can be considered a digital version of the camera negative. ARRIRAW is the only format that fully retains the camera's natural color response and great exposure latitude as unprocessed sensor data.

    Like film negative, ARRIRAW data has to be developed – or rather processed – in order to convert the single channel image, which represents the raw Bayer pattern sensor readout, into a color image suitable for normal viewing. The originally recorded raw data remains pristine, always providing the flexibility to go back and refine the results. This makes ARRIRAW the perfect format for digital cinematography and high-quality visual effects production.

    Our camera line-up supports in-camera recording of ARRIRAW or MXF/ARRIRAW data. MXF/ARRIRAW is ARRIRAW within a container format (*.mxf), allowing each clip to be contained in just one file, as opposed to several thousand files, because with the regular ARRIRAW format (*.ari) every single frame is a separate file.

    Older ALEXA cameras (ALEXA Classic, ALEXA Plus/Plus 4:3, ALEXA M, and ALEXA Studio) require an ARRI-certified external recorder which understands the T-Link signal.


    ARRIRAW Processing

    ARRI has built long-term relationships with postproduction equipment manufacturers through the ARRI Partner Program. These relationships have facilitated all of the leading compositing and color correction tools being able to process ARRIRAW files out of the box.

    In addition, ARRI offers a software development kit (SDK) for ARRIRAW processing, containing documentation of the ARRIRAW processing pipeline and a library – executable code – that the vendor can incorporate into their application. ARRI also supports vendors who wish to implement the ARRIRAW processing procedure on their own, through comprehensive documentation of a three-phase color processing pipeline and continuous direct support.

    Debayering
    The first phase of ARRIRAW processing is the most compute intensive. ARRIRAW images (like all camera raw images) have only one ‘color’ channel (in fact, it’s a color-coded luminance channel). A color reconstruction algorithm calculates the missing components for each pixel based on the type and position of colored filters on the camera sensor. ARRI cameras use the Bayer pattern color filter array. The term 'color reconstruction' therefore is also known as ‘debayering’. The Bayer pattern filters the light hitting the sensor so that 50% of the sensor’s photosites are used to represent green, 25% of the photosites represent red, and the remaining 25% represent blue. Therefore, the debayer algorithm needs to reconstruct 75% of red, 50% of green, and 75% or blue color information:

    The image above shows a single-channel capture from the sensor on the left, and on the right the reconstructed image facilitating the color filter array. Half of the reconstructed image's green values are interpolated from the surrounding photosites rather than captured, as are three-quarters of the red and three-quarters of the blue.

    The output quality of the image depends on the debayering algorithm. Generally speaking, a simpler algorithm will process faster, but will also involve a higher probability of color errors.

    For VFX, however, the images are often processed using the native sensor pixel count and then downscaled to e.g. 4K or 2K at a later stage. Using this approach takes advantage of the luminance resolution, which correlates to the sensor pixel count.

    ARRIRAW SDK and Third-Party Implementations
    In many cases, ARRI's SDK is fast and adaptable enough to satisfy the processing needs of an application. But in some cases, especially when the product uses custom hardware (or standard hardware in a non-standard way), Partner Program members may want to implement the ARRIRAW processing pipeline themselves. In this scenario, the SDK serves as a reference against which the member's developers can test their results before submitting evaluation imagery to ARRI's Workflow Group. In a few applications, the vendor's product offers both processing solutions, giving the user the option of maximum throughput with a very good match to the SDK, or bit-for-bit matching of other products that use the reference SDK implementation.


    ARRI Reference Tool (ART) replaces ARRIRAW Converter (ARC)

    The ARRI Reference Tool is a software application that provides a graphical user interface for ARRI’s reference SDK. ART is a combined tool for viewing, rendering, metadata, and look files. It combines and therefore replaces the ARRIRAW Converter, ARRI Color Tool, and ARRI Meta Extract.

    You can download your copy of the ARRI Reference Tool here.

    The legacy version of ARRIRAW Converter can be found here.

    Testing Your Workflow
    ARRI recommends running tests to define:

    • How the intended look can be achieved using the chosen toolset(s).
    • The different tools that need to be set up to meet the production's expectations of consistent results.
    • The settings required for a chosen tool to most closely match the SDK reference results, if no other reference set of results is available.

    You can use an ARRIRAW frame grab as test material. ARRI provides sample footage shot with our cameras on FTP server and Webgate.io.



  • High Density Encoding (HDE)

    HDE or CODEX High Density Encoding is a lossless, variable bitrate encoding scheme for ARRIAW data. It can reduce the footprint of ARRIRAW data by up to 40 – 50%, but deliver a bit-exact match to the original files when it's decoded. So, HDE does not compromise on quality.

    Supported by leading content providers and postproduction applications, HDE is widely used on ARRIRAW data from any ARRI camera.


    HDE, like ARRIRAW, comes in two file types: as an MXF-wrapped clip for ALEXA 35 or as a single-frame *.arx file sequences for all previous cameras.
    The move to the MXF container significantly reduces redundancy from metadata that will be static throughout a clip. ALEXA 35 processing is fully aligned towards metadata transport in an MXF container. Single frame ARRIRAW and HDE therefore is only available for earlier camera models.


    HDE for ALEXA 35

    You have two options to turn ALEXA 35 MXF/ARRIRAW data into MXF/HDE data:

    • The CODEX Device Manager is a system extension for macOS that can be controlled from the menu bar. The software can automatically present a virtual HDE volume on the desktop, next to the actual CODEX drive containing the original ARRIRAW data. The data on that virtual drive can then be copied using industry standard media asset management tools (copy tools). The HDE files are created on-the-fly at the time they are transferred.
    • The ARRIRAW HDE Transcoder is a standalone app for macOS, Windows and CentOs. To create HDE files, you need to point the software at a source (drive or folder) containing the ARRIRAW data and set a destination where it will create the HDE files. No third party tool is needed to transfer the data.

    HDE for other ARRI cameras

    For drives containing ARRIRAW data from any earlier ARRI camera, the only option to apply HDE is CODEX Device Manager. Customers with an ALEXA Mini or AMIRA in addidtion need a CODEX CFast HDE License ("HDE for any CFast 2.0 reader") unless you still have a CODEX USB CFast 2.0 card reader to access the cards.

  • ProRes File Format

    Recording clips in Apple ProRes 4444 XQ preserves the full sensor quantization in logarithmic encoding, with the same range of colors available in ARRIRAW. Images recorded in a 4:4:4 codec are almost indistinguishable from uncompressed HD or UHD material. This makes internal recording attractive to feature film productions for the big screen, too. Recording in any of the high-end 4:2:2 codecs provides perfect source material for web or TV applications.

    In 2010, ALEXA was the first digital motion picture camera on the market offering in-camera recording of Apple QuickTime/ProRes files onto SxS PRO cards, providing the full range of codecs from Apple ProRes 422 Proxy to Apple ProRes 4444 and later 4444 XQ.

    In 2019, ALEXA Mini LF introduced a new method for Apple ProRes recording: sound, images, and metadata are no longer wrapped inside a QuickTime *.mov container but use the common MXF (Material eXchange Format) container.

    MXF is an open standard that is widely supported and allows for more flexible access to metadata. Using the same container also allows us to record ARRIRAW and Apple ProRes to the same drive without reformatting. Apple is supporting MXF/Apple ProRes. They have published the "SMPTE RDD 44:MXF – Mapping and Application of Apple ProRes" (on IEEE.org).

    ARRI cameras that currently support QuickTime/Apple ProRes will continue to use QuickTime.


    Please Note:
    Apple ProRes is a variable bitrate codec, therefore a longer recording time than initially indicated on the camera may be possible.
    Apple ProRes White Paper (2022)