@Copyright by Fore June, 2006

Video File Formats

Introduction
In the previous Chapter, we have discussed how to play a video using data saved in the raw format of a file. In reality, video data are saved in a predefined format. There have been numerous video formats around and the data of most of them are saved in compressed form. In this Chapter, we shall give a brief discussion on some popular formats and do a case study on the .avi file format. Just as raw pixel data are often saved in .bmp format, and raw PCM sound samples in .wav format, raw video data are often saved in .avi format. We shall learn how to extract the raw video data from a .avi file so that we can play the video as we discussed in the previous Chapter or process them in our own way.
Video Storage Formats
Requirements of Internet Video File Format
There exists a large number of video file formats in the market not only because competing companies create their own formats, hoping to push out competitors and to make their formats standards, but also because there are legal needs of not overstepping competitors' so called intellectual property. The following sections examine some common characteristics between the popular file formats. We could summarize the requirements for video file format to be successful as follows. A video file format should be able to

store video and audio data,
provide fast, real-time playback on target viewing platforms,
provide efficient scrubbing ( fast-forward and rewind while previewing ),
store metadata ( e.g. copyright, authorship, creation dates ... ),
store additional tracks and multimedia data like thumbnails, subtitle tracks, alternate language audio tracks ...,
allow for multiple resolutions,
provide file locking mechanisms,
allow for video editing,
provide integrity checking mechanism, and
perform segmentation of audio and video portions into packets for efficient Internet transmission.

Common Internet Video Container File Format
A container file format is hierarchical in structure, and can hold different kinds of media ( audio, video, text .. ) synchronized in time. The following are some popular container formats, which can save various types of media.

AVI ( Audio Video Interleaved ) -- standard audio / video file format under Windows; not suited for streaming as it does not have any standard way to store packetization data.
MOV -- Apple's Quick Time format, better than AVI in synching audio and video; also supports Windows, and Linux.
ASF ( Advanced Streaming Format ) -- Microsoft's proprietary format ( .WMV, .WMA ), designed primarily to hold synchronized audio and video.
RM ( RealMedia ) -- Real's streaming media files; can be extended to hold all types of multimedia; supports Windows, and Linux platforms and many standards, including MPEG-2, MPEG-4, Real, H263.
MP4 ( MPEG-4 ) -- almost identical to MOV but MPEG-4 players can handle only MPEG-4 related audio, video and multimedia.
SWF, SWV ( Shockwave Flash ) -- for Flash movies, typically containing vector-drawn animations with scripting controls; also supports video codecs, JPEG still images, remote loading of SWF files, XML data and raw text.
NSV ( NullSoft Video ) -- by NullSoft ( a division of AOL ) for streaming.

Simple Raw or Stream Video Format
Simple raw storage or stream formats store the compressed data without extra headers or metadata. These are essentially live audio-video streams saved to disk. Below are some examples.

MPEG-1, MPEG-2 -- streams are composed of interleaved audio and video data arranged into groups of pictures.
MP3 -- encode audio; part of MPEG-1
DV ( Digital Video ) -- used by modern digital cameras and video editing software.
.263 -- video compressed with H.263 codec.
.RTP -- Real Time Protocol data.

Internet Playlist, Index, and Scripting Format
Index formats have pointers linking to other resources. The following are some of this kind.

MOV -- has several formats that do not contain actual video or audio data, but merely point to other files.
RAM ( RealAudio Metafile ) -- points to the URL of the actual media file ( RealMedia .RM or RealAudio .RA ).
ASX ( Active Streaming Index ) -- index files that work in Windows Media system and point to the content held in an ASF media file
SMIL ( Synchronized Multimedia Integration Language ) -- provides instructions to a media player on how to present a multimedia interface and what content to display

Case Study: AVI Files ( .avi )

AVI ( Audio Video Interleaved ) is a file format defined by Microsoft for use in applications that capture, edit and play back audio-video sequences. Just as raw pixel data are often saved in .bmp format, and raw PCM sound samples in .wav format, raw video data are often saved in .avi format. It is a special case of RIFF (Resource Interchange File Format) and is the most commonly used format for storing audio/video data in a PC. An AVI file can be embedded in a web page using a link like:

somedomain

In order that your Apache Web server is able to handle avi files, you need to add in the configuration file the following statement.

AddType video/avi .avi

AVI is often regarded as an obsolete video/audio file format as it lacks many contemporary and crucial features to support streaming and image processing. However, it has been extended by OpenDML to include some of those features.

RIFF FILE Format

The AVI file format is based on the RIFF (resource interchange file format) document format. A RIFF file consists of a RIFF header followed by zero or more lists and chunks; it uses a FOURCC ( four-character code ) to denote a text header. A FOURCC is a 32-bit unsigned integer created by concatenating four ASCII characters. For example, 'abcd' = 0x64636261. The AVI file format uses FOURCC codes to identify stream types, data chunks, index entries, and other information. The RIFF file format has the following form.

A RIFF header consists of 'RIFF' fileSize fileType (data) where
'RIFF' is the literal FOURCC code 'RIFF',
fileSize is a 4-byte value indicating the size of the data in the file including the size of the fileType plus the size of the data that follows,
fileType is a FOURCC that identifies the specific file type,
data consists of chunks and lists in any order.
A chunk consists of chunkID chunkSize chunkData where
chunkID is a FOURCC that identifies the data contained in the chunk,
chunkSize is a 4-byte value giving the size of data in chunkData not including padded values,
chunkData is zero or more bytes of data, padded to nearest WORD boundary.
A list consists of 'LIST' listSize listType listData where
'LIST' is the literal FOURCC code 'LIST,
listSize is 4-byte value, indicating the size of the list,
listType is a FOURCC code specifying the list type,
listData consists of chunks or lists, in any order

The following are sample data from and AVI file. The data are displayed in hexadecimal; the corresponding ASCII characters are printed on the right if they are printable otherwise a dot is printed. Some comments are shown at the far right to indicate what the data represent.

[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0123456789012345] 000000000: 52 49 46 46 DC 6C 57 09 41 56 49 20 4C 49 53 54 |RIFF.lW.AVI LIST|RIFF fileSize fileType LIST 000000016: CC 41 00 00 68 64 72 6C 61 76 69 68 38 00 00 00 |.A..hdrlavih8...|listSize listType avih structureSize 000000032: 50 C3 00 00 00 B0 04 00 00 00 00 00 10 00 00 00 |P...............|microSecondPerFrame maxBytesPerSec 000000048: A8 02 00 00 00 00 00 00 01 00 00 00 00 84 03 00 |................|totalFrames initialFrames streams suggestedBufferSize 000000064: 40 01 00 00 F0 00 00 00 00 00 00 00 00 00 00 00 |@...............|width height 000000080: 00 00 00 00 00 00 00 00 4C 49 53 54 74 40 00 00 |........LISTt@..| 000000096: 73 74 72 6C 73 74 72 68 38 00 00 00 76 69 64 73 |strlstrh8...vids| 000000112: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000128: 64 00 00 00 D0 07 00 00 00 00 00 00 A8 02 00 00 |d...............| 000000144: 00 84 03 00 10 27 00 00 00 00 00 00 00 00 00 00 |.....'..........| 000000160: 40 01 F0 00 73 74 72 66 28 00 00 00 28 00 00 00 |@...strf(...(...| 000000176: 40 01 00 00 F0 00 00 00 01 00 18 00 00 00 00 00 |@...............| 000000192: 00 84 03 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000208: 00 00 00 00 69 6E 64 78 F8 3F 00 00 04 00 00 00 |....indx.?......| 000000224: 01 00 00 00 30 30 64 62 00 00 00 00 00 00 00 00 |....00db........| 000000240: 00 00 00 00 0C 44 00 00 00 00 00 00 00 40 00 00 |.....D.......@..| . . 000017408: 4C 49 53 54 38 F9 56 09 6D 6F 76 69 69 78 30 30 |LIST8.V.moviix00|LISTlistSize listType indexBlock ( ix00 ) 000017424: F8 3F 00 00 02 00 00 01 A8 02 00 00 30 30 64 62 |.?..........00db| .... 00db ( uncompressed video frame ) . .

Table 1. Sample AVI data

As you can see from the above sample data, a two-character code is used to define the type of information in the chunk.

Two-character code	Description
db	Uncompressed video frame
dc	Compressed video frame
pc	Palette change
wb	Audio data

For example, if stream 0 contains audio, the data chunks for that stream would have the FOURCC '00wb'. If stream 1 contains video, the data chunks for that stream would have the FOURCC '01db' or '01dc'.

AVI RIFF Format

As shown in Table 1, the FOURCC 'AVI ' in a RIFF header identifies the file to be an AVI file. An AVI file has two mandatory LIST chunks, defining the format of the streams and the stream data, respectively. An AVI file might also include an index chunk, indicating the address of the data chunks of the file; it has the following form ( Table 2 ):

	RIFF ('AVI '
      		LIST ('hdrl' ... )
      		LIST ('movi' ... )
      		['idx1' () ]
     	)

Table 2. AVI RIFF Format

The 'hdrl' list defines the format of the data and is the first mandatory LIST chunk. The 'movi' list contains the data for the AVI sequence and is the second required LIST chunk. An optional index ('idx1') chunk can follow the 'movi' list. The index contains a list of the data chunks and their location in the file.

If we expand 'hdrl' and 'movi' in Table 2, we shall get a form shown below ( Table 3 ):

RIFF ('AVI '
      LIST ('hdrl'			//header length
            'avih'()
            LIST ('strl'		//stream length
                  'strh'()
                  'strf'()
                  [ 'strd'() ]
                  [ 'strn'() ]
                  ...
                 )
             ...
           )
      LIST ('movi'
            {SubChunk | LIST ('rec '
                              SubChunk1
                              SubChunk2
                              ...
                             )
               ...
            }
            ...
           )
      ['idx1' () ]
     )

The 'hdrl' list begins with the main AVI header, which is contained in an 'avih' chunk. The main header contains global information for the entire AVI file, such as the number of streams within the file and the width and height of the AVI sequence. This main header structure is shown below.

typedef struct _avimainheader {
    FOURCC fcc;			//'avih'
    DWORD  cb;			//size of structure, not including first 8 bytes
    DWORD  dwMicroSecPerFrame;
    DWORD  dwMaxBytesPerSec;
    DWORD  dwPaddingGranularity;
    DWORD  dwFlags;
    DWORD  dwTotalFrames;
    DWORD  dwInitialFrames;
    DWORD  dwStreams;
    DWORD  dwSuggestedBufferSize;
    DWORD  dwWidth;
    DWORD  dwHeight;
    DWORD  dwReserved[4];
} AVIMAINHEADER;

One or more 'strl' lists follow the main header. A 'strl' list is required for each data stream. Each 'strl' list contains information about one stream in the file, and must contain a stream header chunk ('strh') and a stream format chunk ('strf'). In addition, a 'strl' list might contain a stream-header data chunk ('strd') and a stream name chunk ('strn'). The stream header chunk ('strh') consists of an AVISTREAMHEADER structure shown below.

typedef struct _avistreamheader {
     FOURCC fcc;
     DWORD  cb;
     FOURCC fccType;	//'vids' - video, 'auds' - audio, 'txts' - subtitle
     FOURCC fccHandler;
     DWORD  dwFlags;
     WORD   wPriority;
     WORD   wLanguage;
     DWORD  dwInitialFrames;
     DWORD  dwScale;
     DWORD  dwRate;
     DWORD  dwStart;
     DWORD  dwLength;
     DWORD  dwSuggestedBufferSize;
     DWORD  dwQuality;
     DWORD  dwSampleSize;
     struct {
         short int left;
         short int top;
         short int right;
         short int bottom;
     }  rcFrame;
} AVISTREAMHEADER;

Table 3. AVI Main Stream Header

One can also express Digital Video ( DV ) data in the AVI file format. The following example shows the AIFF RIFF form for an AVI file with one DV data stream, expanded with completed header chunks.

00000000 RIFF (0FAE35D4) 'AVI '
0000000C     LIST (00000106) 'hdrl'
00000018         avih (00000038)
                     dwMicroSecPerFrame    : 33367
                     dwMaxBytesPerSec      : 3728000
                     dwPaddingGranularity  : 0
                     dwFlags               : 0x810 HASINDEX | TRUSTCKTYPE
                     dwTotalFrames         : 2192
                     dwInitialFrames       : 0
                     dwStreams             : 1
                     dwSuggestedBufferSize : 120000
                     dwWidth               : 720
                     dwHeight              : 480
                     dwReserved            : 0x0
00000058         LIST (0000006C) 'strl'
00000064             strh (00000038)
                         fccType               : 'iavs'
                         fccHandler            : 'dvsd'
                         dwFlags               : 0x0
                         wPriority             : 0
                         wLanguage             : 0x0 undefined
                         dwInitialFrames       : 0
                         dwScale               : 100 (29.970 Frames/Sec)
                         dwRate                : 2997
                         dwStart               : 0
                         dwLength              : 2192
                         dwSuggestedBufferSize : 120000
                         dwQuality             : 0
                         dwSampleSize          : 0
                         rcFrame               : 0,0,720,480
000000A4             strf (00000020)
                         dwDVAAuxSrc     : 0x........
                         dwDVAAuxCtl     : 0x........
                         dwDVAAuxSrc1    : 0x........
                         dwDVAAuxCtl1    : 0x........
                         dwDVVAuxSrc     : 0x........
                         dwDVVAuxCtl     : 0x........
                         dwDVReserved[2] : 0,0
000000CC     LIST (0FADAC00) 'movi'
0FADACD4     idx1 (00008900)

Utility Program for Reading AVI Files

In order to extract the data from an AVI file, we need to write a program that can understand the AVI format. However, rather than reinventing the wheel from the ground up, we make use of some existing open-source libraries and codes to help us do the job. We shall utilize the codes from MPEG4IP, which is an open-source package consisting of tools for streaming video and audio that is standards-oriented and free from proprietary protocols and extensions. MPEG4IP's development is focused on the Linux platform, and has been ported to Windows, Solaris, FreeBSD, BSD/OS and Mac OS X. It can handle MPEG-4, H.261, MPEG-2, H.263, MP3, AAC, and AMR file formats. When you download the package from its site and unpack it, you will find the utility program "avilib.c" and its header file "avilib.h" in the subdirectory ./lib/avi. You also need the header file "mpeg4ip.h" in the subdirectory ./include to compile "avilib.c".

We shall utilize the functions provided by "avilib.c" to read an AVI file. The main AVI structure is defined in "avilib.h" and is named avi_t. As shown below, avi_t basically covers all the crucial features of an AVI file.

typedef struct
{
   long   fdes;              /* File descriptor of AVI file */
   long   mode;              /* 0 for reading, 1 for writing */

   long   width;             /* Width  of a video frame */
   long   height;            /* Height of a video frame */
   double fps;               /* Frames per second */
   char   compressor[8];     /* Type of compressor, 4 bytes + padding for 0 byte */
   long   video_strn;        /* Video stream number */
   long   video_frames;      /* Number of video frames */
   char   video_tag[4];      /* Tag of video data */
   long   video_pos;         /* Number of next frame to be read
                                (if index present) */

   long   a_fmt;             /* Audio format, see #defines below */
   long   a_chans;           /* Audio channels, 0 for no audio */
   long   a_rate;            /* Rate in Hz */
   long   a_bits;            /* bits per audio sample */
   long   audio_strn;        /* Audio stream number */
   long   audio_bytes;       /* Total number of bytes of audio data */
   long   audio_chunks;      /* Chunks of audio data in the file */
   char   audio_tag[4];      /* Tag of audio data */
   long   audio_posc;        /* Audio position: chunk */
   long   audio_posb;        /* Audio position: byte within chunk */

   long   pos;               /* position in file */
   long   n_idx;             /* number of index entries actually filled */
   long   max_idx;           /* number of index entries actually allocated */
   unsigned char (*idx)[16]; /* index entries (AVI idx1 tag) */
   video_index_entry * video_index;
   audio_index_entry * audio_index;
   long   last_pos;          /* Position of last frame written */
   long   last_len;          /* Length of last frame written */
   int    must_use_index;    /* Flag if frames are duplicated */
   long   movi_start;
} avi_t;

To read an AVI file, we can first use the function AVI_open_input_file() of "avilib.c" to open the AVI file:

avi_t *AVI_open_input_file(const char *filename, int getIndex);

where filename points to the filename of the AVI file to be opened and getIndex determines if you need the indexing information of the file or not; a nonzero value indicates you want the information while a value of 0 indicates you don't. If opened successfully, it returns a pointer to an avi_t structure containing relevant information of the AVI file for further processing.

The following program, avi_to_raw.cpp is a sample program that makes use of the avilib functions to extract the video and audio from an AVI file. It is simple and straightforward and the code is self-explained. However, it has to be compiled with the avilib.c program which is also listed below. For example, you can use the following command to compile the executable avi_to_raw:

$g++  -o  avi_to_raw  avi_to_raw.cpp  avilib.c

You can execute it with the following format:

$./avi_to_raw  avi_file  [raw_video_output_file  raw_audio_output_file]

If the input avi_file is in uncompressed format, you will obtain the video raw data saved in raw_video_output_file. The program will also output other relevant data of the video and audio data. The following is a sample output.

---------------------------------------------------------
Video Frame Rate:          20   fps
Number of Frames:         680
File Duration:             34   sec
Video width:              320   pixels
Video height:             240   pixels
Compressor type:
Audio channels:             0
Audio sample rate:          0   Hz
Bits per audio sample:      0
Video starts at:        17420 ( 0x440c )
---------------------------------------------------------

The following lists the programs avi_to_raw.cpp; the links to avilib.c, and the required header files avilib.h and mpeg4ip.h are also provided. If you want to try it, simply copy-and-paste the code into your own file and download the other three files ( avilib.c, avilib.h, mpeg4ip.h ) to the same directory and compile them using the command mentioned above.

/*
  avi_to_raw.cpp
  Need to compile together with MPEG4IP avilib.c
  Compile: g++ -o avi_to_raw avi_to_raw.cpp avilib.c
  Execute: ./avi_to_raw infile [video_output_file audio_output_file]

  http://www.webkinesia.com/games/index.php
*/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <math.h>
#include "avilib.h"

using namespace std;

int main( int argc, char *argv[] )
{
  FILE *fpo = NULL;
  avi_t *avi = NULL;		//points to opened avi file
  char infilename[30], outfilename[30], *pc;
  double videoFrameRate;
  unsigned  numVideoFrames; 
  unsigned long fileDuration;
  long numBytes = 0, totalBytes = 0;
  unsigned long videoFramesRead = 0;
  unsigned long emptyFramesRead = 0;
  unsigned long short_frames_len = 4;

  if ( argc < 2 ) {
    printf("\nUsage: avi_to_raw infile [outfile1 outfile2]\n");
    exit ( 1 );
  } 
  strcpy ( infilename, argv[1] );

  avi = AVI_open_input_file( infilename, 1 );
  if (avi == NULL) {
    fprintf(stderr, "error %s: %s\n", infilename, AVI_strerror());
    exit( 2 );
  }
 
  videoFrameRate = AVI_video_frame_rate( avi );	//frame per second
  numVideoFrames = AVI_video_frames( avi );
  fileDuration = ( unsigned long ) ceil( numVideoFrames / videoFrameRate);

  printf("\n---------------------------------------------------------");
  printf("\nVideo Frame Rate:\t%5.0f \tfps", videoFrameRate );
  printf("\nNumber of Frames:\t%5d", numVideoFrames );
  printf("\nFile Duration:   \t%5ld \tsec", fileDuration );
  printf("\nVideo width:     \t%5d \tpixels", avi->width );
  printf("\nVideo height:     \t%5d \tpixels", avi->height );
  printf("\nCompressor type: \t%5s", avi->compressor );
  printf("\nAudio channels:  \t%5d", avi->a_chans );
  printf("\nAudio sample rate:\t%5d \tHz", avi->a_rate  );
  printf("\nBits per audio sample:\t%5d", avi->a_bits );

  printf("\nVideo starts at:  \t%5ld ( 0x%lx )", avi->movi_start, avi->movi_start );
  printf("\n---------------------------------------------------------\n");
 
  //extract video 
  if ( argc < 3 ) {				//construct raw video output file name
    strcpy ( outfilename, infilename );
    pc = strrchr ( outfilename, '.' );          //pc points to '.'
    if ( pc != NULL ) *pc = 0;
    strcat ( outfilename, ".rawv" );             //extenstion .rawv
  } else
    strcpy ( outfilename, argv[2] ); 
  if ( ( fpo = fopen ( outfilename, "wb" ) ) == NULL ) {
    fprintf(stderr, "error opening file %s for output\n", outfilename );
    exit ( 3 );
  }
  // get a buffer large enough to handle a frame of raw SDTV 
  u_char *buf = ( u_char * ) malloc ( 768 * 576 * 4 );
  if ( buf == NULL ) {
    fprintf(stderr, "Error allocating memory\n");
    exit ( 4 );
  }
  //move file pointer pointing to beginning of video data
  if (AVI_seek_start( avi )) {
     fprintf(stderr, "bad seek 0: %s\n", AVI_strerror());
     exit( 5 );
  }
  if (AVI_set_video_position( avi, 0, NULL)) {
     fprintf(stderr, "bad seek 1: %s\n", AVI_strerror());
     exit( 6 );
  }

  videoFramesRead = 0;
  while ( ( numBytes = AVI_read_frame( avi, (char *)buf ) ) > -1 ) {
    totalBytes += numBytes;
    videoFramesRead++;
    printf("frame %d - length %u total %u\r", videoFramesRead, numBytes, totalBytes);
    //eliminate short frames
    if ( numBytes > short_frames_len ) {
      if (  fwrite ( buf, 1, numBytes, fpo ) != numBytes ) {
        fprintf ( stderr, "Error writing %s\n", outfilename );    
        break;
      }
    } else {
      emptyFramesRead++;
    }
    if ( videoFramesRead >= numVideoFrames ) break;
  } //while
  // read error
  if (numBytes < 0) 
    fprintf(stderr, "read error\n" );
  if (emptyFramesRead) {
    fprintf(stderr, "warning: %u zero length frames ignored\n", emptyFramesRead);
  }

  fclose ( fpo );			//done with video

  if ( avi->a_chans == 0 ) {
    printf("\nno audio channel\n" );
    exit ( 7 );
  }
  //extract audio
  if ( argc < 4 ) {                             //construct raw audio output file name
    strcpy ( outfilename, infilename );
    pc = strrchr ( outfilename, '.' );          //pc points to '.'
    if ( pc != NULL ) *pc = 0;
    strcat ( outfilename, ".rawa" );       	//extenstion .rawa
  } else
    strcpy ( outfilename, argv[3] );
  if ( ( fpo = fopen ( outfilename, "wb" ) ) == NULL ) {
    fprintf(stderr, "error opening file %s for output\n", outfilename );
    exit ( 9 );
  }

  unsigned audioFrames = 0;
  totalBytes = 0;
  if (AVI_seek_start( avi )) {
    fprintf(stderr, "Audio bad seek 2: %s\n", AVI_strerror());
    exit( 10 );
  }
  if (AVI_set_audio_position( avi, 0)) {
    fprintf(stderr, "Audio bad seek 3: %s\n", AVI_strerror());
    exit( 11 );
  }
  while ((numBytes = AVI_read_audio( avi, (char *)buf, 8192)) > 0) {
    totalBytes += numBytes;
    if (fwrite ( buf, 1, numBytes, fpo ) != numBytes) {
      fprintf(stderr, "error writing %s\n", outfilename );
      break;
    }
    audioFrames++;
    printf("frame %d - length %u total %u\r", audioFrames, numBytes, totalBytes);
  }
 
  printf("\n");
  fclose ( fpo );	

  return 0;
}

For your convenience, you may read all the files off the web by clicking the following links.

avi_to_raw.cpp

avilib.c

avilib.h

mpeg4ip.h

Next >>