Invalid 24 bit BWF files imported by Avid without timecode

The problem

When you import 24 bit BWF files into Avid, you may encounter a pretty obscure problem: some of the files will be imported without the timecode, while others will be OK. This will never happen happen with 16 bit files, nor will it happen with stereo 24 bit files. And it will only happen with files produced by certain recorders. And I'm not sure whether it only happens on Macs (I used Avid Xpress Pro 5 on Mac). It is a rare problem, and it took me a while to figure out, so here is what I found. I had the problem with files originating on a Sound Devices 744T (fw 2.03), but the same problem seems to exist with the HHB Portadrive, and probably with others as well. (As far as I know, the Cantar doesn't have this problem. I don't know about others). (Update: the Sound Devices firmware update 2.15 fixes this.) The manufacturers lay the blame on Avid, like in this post on the HHB forum:

"There has been a problem with importing some Portadrive 24-bit wav files into Avid Xpress Pro. Avid would import them via Quicktime rather than directly and in the process all metadata (including most vitally the timestamp) would be lost. This is an Avid bug and will be fixed in the next release of Xpress Pro."

However, what I found is that it seems to really be a bug in the recorders which produce invalid WAVE files, as shown also by the fact that Avid sends the file to Quicktime instead of processing it itself. I guess Avid could fix their WAV reader, but not easily without breaking the import of some valid files. It would seem much more straightforward to update the firmware in these recorders so that they write valid WAVE files, especially since the change would be quite trivial.

The theory

In short, WAV files are supposed to have an even number of bytes. When you record 24 bit mono sound, you get 3 bytes per sample, and around half of the files will have audio containing an odd number of bytes. So these files would not be valid and confuse the routines reading them. (Before all the technical details, here is the opportunity to jump directly further down to something more practical or even straight to a possible solution.) Broadcast Wave files (BWF) are really Microsoft WAVE format files, with a "broadcast extension" containing timecode and other metadata information. And these .WAV files are really just a form of RIFF files. RIFF files were created by Microsoft and IBM a long time ago, and are based on another even older format: IFF files. (AIFF is another such descendent of IFF). These specifications have all been published at some point, but have also been changed and augmented since then, and a clear and current reference is hard to find. Basically, all these RIFF files are composed of various blocks called chunks, of varying sizes. Any WAV file contains at least a "data" chunk with the audio, and a format chunk describing the number of channels, sampling rate, etc. In addition, the BWF variety of WAV files also contains a "bext" chunk which is where the timecode is stored along with other information, and may also contain various other chunks (iXML, link, etc.). The best quotes I found to pinpoint this specifc problem of odd/even sized chunks are these: GFF Format Summary: Microsoft RIFF from an O'Reilly book:

"ChunkData contains data that is WORD-aligned within the RIFF file. If the data is an odd length in size, an extra byte of NULL padding is added to the end of the data. The ChunkSize value does not include the length of the padding."

and AVI RIFF File Reference from the Microsoft site:

"The data is always padded to nearest WORD boundary. ckSize gives the size of the valid data in the chunk; it does not include the padding, the size of ckID, or the size of ckSize."

So all these chunks must have an even number of bytes. If the data in the chunk happens to need an odd number of bytes like in the case of some 24 bit mono sounds, a null byte must be added to pad the chunk to an even number. But the reported chunk size is still expected to be the exact number of useful bytes, different from the real size when it happens to be odd. Weird, isn't it? According to the IFF entry in Wikipedia, this weirdness is due to a limitation of an ancient processor:

"Chunks must begin on even file offsets, as befits the origins of IFF on the Motorola 68000 processor, which couldn't address quantities larger than a byte on odd addresses. Thus a chunk following an odd-length chunk will be preceded by a pad byte."

No wonder many devices and software are programmed without taking this (may I say) odd detail into account. Especially since you will hardly find these quotes if you search for WAV or BWF, even though they apply to any RIFF file, including WAVE files.

Back to practice

You can recognize these problematic files by the fact that their size in bytes is an odd number (once you find how to get your file manager to show you exact sizes in bytes instead of MB). If the audio is not padded to an even number with a null byte, you end up with an invalid WAV file, which some programs will understand, and some not. If the audio happens to be at the very end of the WAV file, which seems to be the case with the files I have seen, some programs will either not mind about the missing null byte which is not part of the audio anyway, or not even notice (I guess that is what happens in the Windows version of Avid). However, if the audio data is followed in the file with another chunk like a bext or iXML chunk, you are likely to have many problems. The way Avid seems to handle it, is that it does see the file as invalid, and lets Quicktime have a go at it. Quicktime will read the audio, but it skips the bext chunk with the timecode, so Avid gets back the audio without the TC. You will also notice, during the import into Avid, the mention of QT instead of WAVE in the progress window. This is a clear sign that you will get the audio through Quicktime, and without timecode.

The solution

In the HHB forum thread mentioned earlier, they suggest using a little Mac OS9 program: HHBFix which can be downloaded from the HHB site. It did fix the specific files I had so that they imported correctly. However, it does it in a way which proved quite unreliable in some cases: it just blindly adds a null byte to the end of any odd-sized file. Obviously, this will only work if the audio data chunk happens to be the last one in the file. That will probably be the case in a file coming straight from a portable recorder, but there is no guarantee about it. If, before importing the file, some program added another chunk at the end, HHBFix will not do any good. I would also have another solution, but it currently uses Perl in a shell window, and a modified version of the File::Format::RIFF module. Since the people likely to need a solution are also unlikely to know what Perl or a shell window like Terminal are, I'm not sure it's really useful. The best solution would really be for manufacturers to correct the code in their firmware (update: Sound Devices seems to have done that quickly). Update: BWF Widget Pro (runs on Windows) added the ability to correct these invalid files since version 1.175.

Conclusion

Don't let this discourage you from using timecode. In most cases, you will never encounter this problem since it only happens with mono 24 bit sounds produced on some recorders and imported into a Mac version of Avid.