Which is the Best Archive File Format for Video?

By Sareesh Sudhakaran

Disclosure: Links in this post may be to our affiliates; sales through affiliate links may benefit this site. Please help support wolfcrow and buy from Amazon. It won’t cost you anything extra.

In the early stages of the computer race nobody could have predicted its explosive growth. The speed in which information (good or bad, useful or useless) is generated, shared and transmitted is unprecedented. Even with today’s knowledge, saddled with trillions of streams of data, the future looks a whole sight scarier.

Everything is going digital; into computer files that will be stored somewhere.

This article will try to compare different file formats, in the hope of discovering the best archive file format for video. Before you start thinking of archival, you need to know the answer to two critical questions:

  • What is valuable to you?
  • How long do you want it to last?
  •  

    What is valuable to you?

    Only you can decide what is valuable. You could be a filmmaker, a government or corporate employee, a scientist, or just a person. For some people, even the most treasured works of humanity are low priority when compared to their personal ‘stuff’.

    I take it you know how to prioritize what is valuable to you, and why it is so. Be careful when you draw up this list, because there is a definite cost associated with each item. The more there are, the more expensive it gets. Think of Einstein’s space-time. If an object takes more space or time, it needs more resources to continue its existence, and this is true even of digital files. Bigger files need more space and resources. Storing files for longer needs more resources.

    The definition of archival

    Depending on whom you ask, you will get many definitions of archival. According to Wikipedia, the definition of archival by archivists is in the order or hundreds of years; while the definition of archival by computer engineers might just be in the order of a few tens of years.

    E.g., LTO backup systems are designed to last for about 30 years. For some, this is a sufficient archival period.

    But not for me. If 30 years is archival, then what do you call preserving work for hundreds of years? Archival. When I use the term ‘archival’ it implies a storage system that lasts for as long as current technology allows.

    An archival solution is one which attempts to store information for the longest period. The best archival solution must store this information longer than any other solution, without corrupting or changing it.

    The best archive file format for video must preserve the video in the best possible quality for the longest duration, while ensuring maximum compatibility. This last part is tough to guess, due to the unpredictability of computer technology. Just remember this: Human history has shown that anything of value can only be preserved with great sacrifices. Nature isn’t that kind to things that want to stay beyond their use-by date.

    This is where digital technology has an upper hand. Unlike analog signals, digital has the potential to stay uncorrupted – as long as it is copied to a newer file format.

    The contenders

    I’m not going to create a comprehensive list, but I’ll try to add as many formats as I can remember. If a particular file format isn’t on the list, please feel free to do your own research.

    The following chart lists many file formats for images and video, and a few properties that will help us compare them (click to enlarge):

    Archive File Format Comparison

    Important: The information in this chart isn’t meant to be accurate. Things are slightly more complicated than I make them out to be. For exact and correct values, please contact those responsible.

    First of all, why make such a chart? It is bring into focus how difficult it is to pin down the right file format. If we wanted to preserve a piece of paper, we could look at the appropriate technology and hit the ground running. Where file formats are concerned, the answer isn’t that cut-and-dried.

    What can we learn from this simple comparison? Here are some pointers:

  • A file format with a free license (zero royalty and unlimited rights) is preferable.
  • A file format that offers the maximum flexibility in file size, resolution, color and bit depth is preferable.
  • A file format that supports the most color spaces is preferable.
  • A file format that supports all kinds of compression, and can also accept fully uncompressed sources, is preferable.
  • A file format for video must also support audio, in all its forms.
  • A file format must support flexible metadata.
  • A true archival file format must give the greatest odds of compatibility in the future. We can only guesstimate this from current trends.
  • The file format must be compatible with all popular operating systems.
  •  

    The best archive file format for video?

    Here’s my opinion: Based on the chart prepared, I feel TIFF is the ideal format for archival. Here are my reasons:

  • TIFF is almost royalty-free
  • It can take 32-bit float images, which is as good as it gets
  • It can accept all kinds of compression
  • It can act like a wrapper, and is the basis for metadata design
  • It supports high resolutions and most color spaces
  • It is ubiquitous – there’s hardly an application, software, program or library that can’t deal with TIFF
  •  
    When you look at the others, they ‘fail’ on certain counts. You might not think of them as failures, and we won’t know for sure until we’re both dead.

    As an aside, here are some reasons why I rejected the others:

  • RAW always needs debayering, and cannot be the ‘best interpretation’ of your work. It is prone to ‘interpretations’.
  • All file formats that cannot support truly uncompressed video is lossy by nature, and future manipulations will be on a lower quality ‘master’.
  • Proprietary file formats aren’t guaranteed to last, nor are they ubiquitous.
  • Transport streams and multiplexed video formats are at the borderline between acceptable and unacceptable quality.
  • OpenEXR is great for visual effects, but like RAW, it is unbaked stuff, open to interpretation. If you really want to store data in separate channels, OpenEXR is acceptable. The major disadvantage is that EXR isn’t ubiquitous.
  •  
    Important disclaimer: Don’t go about archiving your precious work in TIFF just because I say so. If nothing else, I hope this article has stressed on you the importance of giving the subject a lot of thought. Consult professionals who are experts in the field of archival, and take their help. This article is only written as an information piece.

    Do you think my choice of archival format is right? If not, which format is better in your opinion, and why?

    Please support wolfcrow and buy from Amazon. It won't cost you extra.
    If this post helped you, please buy me a


    Share this article and help others:  

    May 8, 2013

    8 comments
    Sicofante
    Sicofante

    (Deleted by author. Wrong thread.)

    Sicofante
    Sicofante

    There are three outstanding errors regarding PNG:

    1. It's completely free from royalties. It actually was born because GIF started to collect them. It's a completely open format, free to use and not patent encumbered.

    2. While not in the spec, it can support "fully uncompressed" storage, although I can't see a use case for that (I've seen this mistake already made in another article of yours: mathematically lossless compression provides exactly the same information, bit for bit, than uncompressed, so there's absolutely no reason to go "fully uncompressed").

    3. Maximum bit-depth is 16 bits per component, which makes it 64 bit for a four channel picture (color+transparency), not 32. (I don't think archival masters should be saved with transparency, so I'd say either 24 or 48 bit is enough for non-floating point pictures).

    Regarding TIFF, it may contain patent encumbered bits which might not be free to use. Also, since TIFF supports a lot of encoding formats, it's hard to predict that it will be compatible with every application. That makes your choice pretty weird. While PNG is supported by every application (and there's no excuse not to, since it's completely open and free), the only reason I can think of for using TIFF is for saving in a floating point format, where anyway OpenEXR tends to be the industry standard and is as open and free as PNG, so, again, you''ll be able to open an OpenEXR file forever. BTW, the "baked" nature of OpenEXR is fully optional, not mandatory. You can use that format to store and archive "fully baked" pictures, which will be properly read for centuries. :-)

    In other words, why not choose PNG for non-floating point archival (which by the way is almost 3 times smaller than TIFF) and OpenEXR for floating point archival? Or if you want just one, go with OpenEXR, which supports almost anything under the sun. Just make sure you're archiving your OpenEXR masters with the information you want there, not app-dependant info.

    I'd also point out that the fact that a format is not "ubiquitous" becomes highly irrelevant when that format is fully open, documented and free (such as PNG and OpenEXR), because there will always be a way to open pictures in that format, no matter how many years from now. Take LibreOffice vs MS Office files as an analogy. Sure, MS ones are "ubiquitous" TODAY, but you don't know how that will be 20 years from now (as a matter of fact, some old MS Office files can't be opened by today's versions of MS Office itself...), while, no matter what, you will always, forever, be able to open an Open Document (LibreOffice) file.

    Just my two cents.


    macsnapper
    macsnapper

    Please explain how to save something as tiff video- I didn't know that was a video format?

    thanks

    Sareesh Sudhakaran
    Sareesh Sudhakaran moderator

    @Sicofante You've got some great info here. If it were up to me I'd use OpenEXR.

    But it's not up to me, or you. There is no guarantee that software will be completely open source tomorrow. The concept of open source, like the concept of freedom, is a great thing. But humans have been known to destroy great things for immediate and temporary gains. In fact, that's the only thing you can conclusively guarantee about humans!

    Why not PNG? It is restrictive, that's why. What if tomorrow there's a new form of compression, and mountains of PNG data have to be re-encoded to the new standard? In the long run, I feel compression by its very nature is not suitable for long-term archival.

    Sicofante
    Sicofante

    @Sareesh Sudhakaran @Sicofante Mmmhhh, I think you have a fuzzy idea of open source. There's indeed a guarantee that current open source of anything will be open source forever. Once a piece of code is put in the open with a proper open source license, there's no way to close it back. What can happen (and sometimes does) is that improvements or changes to newer versions aren't open, but you won't be restricted by these new eventually closed versions from using the original open ones. There can be derivatives that become close (with some particular licenses), but the oriignal, i.e, whatever you're using today to archive, is "uncloseable". The means to open a PNG file today can't be removed from the internet at all, and won't be. Even if a new compression method "du jour" appears and becomes standard practice in the future, old formats will always be usable.

    Whatever your worries are about a chance to format change, they apply in a much stronger way to proprietary formats or implementations. Nothing guarantees you that TIFF -especially with whatever particular compression scheme you're using today- will be around some years from now (as a matter of fact, it's less and less used everywhere, so its chances of surviving are slimmer than both PNG and OpenEXR). In other words, 20 years from now you might not be able to open your TIFF files. That simply can't happen to PNG or OpenEXR, because you "own" the source code to decode them the way they're saved today. Even it it becomes closed at some point (which is very unlikely, BTW) there's no way the code to open your files saved under the current status is removed from the internet.

    What exactly do you mean by restrictive when you refer to PNG? My only gripe is that it doesn't support floating point formats (which it might in the future), but, honestly, I find its 64 bits per pixel capacity future proof enough for material being worked on in the current and next decade. :-) (I'm planning to use it for archival in a ProRes workflow in Lightworks.)

    I'm hoping CinemaDNG to become an open standard soon enough (if I understand it correctly, it's in a process to become one). That would be definitely a great master and archival format.

    Sicofante
    Sicofante

    @Sareesh Sudhakaran @Sicofante You're simply wrong about open source and open standards, but since this is not your problem with PNG, I won't insist on the subject.

    I see your point about compression now. However, PNG's compression schemes are guaranteed to be used forever. Of course you are free to not believe that. However, that same logic would apply to formatted file formats (excuse the redundancy). TIFF is not a plain format to read or write. You need libraries to open TIFF files or at least proper documentation to create those libraries yourself whenever TIFF disappears completely from the scene (a point in time which is closer to what you seem to think). Your worries about any compression scheme becoming obsolete and "unopenable" apply exactly the same to uncompressed TIFF as much as any other. It's not a plain file format. If you really need absolute future proof files, your only choice is probably the old SGI RGB format... because it isn't actually a "format". It's the closest to an equivalent of plain text. It's just a pixel value after the other, written as binary integers, no formats involved at all.

    Your fears are technically unjustified IMO, but of course you're entitled to your choice.


    PS: I still believe you should update your table regarding PNG's properties.

    Sareesh Sudhakaran
    Sareesh Sudhakaran moderator

    @Sicofante If Windows 3.1 is made open source today, would you use it? Dead open source projects are a help to nobody.

    I don't support compressed TIFF, only uncompressed. Compressed PNG is restrictive in the same sense compressed JPEG is. 

    About 'owning' the source code, that's assuming you and I are alive! Archival extends beyond our lifetime, that's my definition of the term 'archival'.

    PNG is as widely accepted as TIFF, so your usage of it is following a certain philosophy of archival. Mine happens to differ in that respect. I do not accept compression as a valid feature of archival.