Video Compression

4. Spatial and Temporal Sampling

   We discussed in Chapter "SDL Sound" the concept of sampling and quantization. Sampling is the process of examining the value of a continuous function at regular intervals and quantization is the process of limiting the value of a sample of a continuous function to one of a predetermined number of allowed values. For a video, we need to consider two kinds of sampling, the spatial sampling and temporal sampling. ( Note: Most of the images used in this Chapter are downloaded from the Internet over a period of time and we forgot the sources of them. We would like to thank all the authors who created the images. )

   Spatial Sampling

   A video consists of a sequence of images. Sampling the analog signal output of a two-dimensional imaging device such as the Charge Couple Devices ( CCD ) array used in a camera at a point in time generates a sampled image or frame consisting of a set of values over a set of two-dimensional spatial points. In general, when we sample an image in space, we need to sample in two directions, along x and y. For instance, consider the following figure which shows the image of the planet Venus, as viewed in reflected microwave. The image is sampled by a rectangular grid. Sampling occurs at the grid points and the sampled image can be reconstructed by representing each sample at its corresponding grid point as a square picture element ( pixel ). In the example, each pixel is assigned a value between 0 to 255 corresponding to the level of reflected microwave energy. A grayscale image is formed by assigning each of the 0 to 255 values to varying shades of gray.

   

   
   Temporal Sampling

   Temporal sampling refers to sampling the image signal at various points of time. Each image sample at a time point is referred to as a frame. Typical video images are sampled at a rate of 10 to 20 frames per second ( fps ) at the low-end and 50 to 60 fps at the high-end. The concept of temporal sampling is shown in the following figure where images are examined at time t = 0, 1, 2, .... One can see from the figure that there are strong correlations between two adjacent frames.

   One can sample a video signal as a series of complete frames, referred to as progressive sampling or as a sequence of interlaced fields, referred to as interlaced sampling as shown in the following figure. A field consists of half of the data from one frame and the other half from an adjacent frame as shown in the figure.
   

   Interlacing is still widely used in TV video systems while progressive sampling is more popular in PC videos. Deinterlacing means to convert two fields back to two frames, which can be done by combining even and odd fields to form a new frame. Very often, as shown in the figure below, blurring may occur in the resulted images of deinterlacing especially when the regions considered have significant motion.

   
   • Color Spaces

     As stated in Wikipedia, a color model is an abstract mathematical model describing the way colors can be represented as tuples of numbers, typically as three or four values or color components. When this model is associated with a precise description of how the components are to be interpreted (viewing conditions, etc.), the resulting set of colors is called a color space.

     RGB Color Space

     In the RGB color space, red, green, and blue, referred to as primary colors are combined in various proportions to produce other colors, which are referred to as secondary colors. The left figure shown below is a cut-away color cube showing the mixing of red, green, blue to produce other colors. The middle figure represents the additive color mixing.

   The image shown at the right is an RGB image, along with its separate R, G, and B components. As one can see from the figure, the white snow consists of strong red, green and blue colors; the brown barn is made up of strong green with little red or blue, and the sky is composed of strong blue and moderately strong red and green. In a computer display, an image is naturally represented in the RGB color space. Color Cathode Ray Tubes ( CRTs ) and Liquid Crystal Displays ( LCDs ) show an image by emitting red, green and blue lights at each pixel with intensities proportional to the values of the components. When viewed from a distance, the separate red, green and blue lights merge to give the appearance of a 'true' color.

   • YCbCr Color Space

     The RGB color space may be best in describing computer graphics or displaying images. However, it is not a good representation when we want to process image information. This is because the human visual system ( HVS ) is less sensitive to color than to luminance ( brightness ). So if we need to discard some information carried by an image, we would first throw away some color information rather than that of luminance. The YCbCr and its variations, ( sometimes referred to as YUV ) color space allow us to describe an image using luminance and colors. In this representation, an image is composed of a Y component, which is the luminance ( luma ), and three color difference ( chrominance or chroma ) components, blue chroma C_b, red chroma C_r, and green chroma C_g. The components can be calculated from R, G, B using formulas like the following.

     Y = k_rR + k_gG + k_bB

     Cb = B - Y
     Cr = R - Y
     Cg = G - Y

     where k_x are weighing factors with k_r + k_g + k_b = 1

     Note that we can obtain Cg from Cb, Cr and Y. So in practice, only two of the three chroma components need to be saved. The figure at the right presents the same RGB image discussed above but along with the Y, Cb, and Cr components. You can see that even if we totally throw away the Cb, and Cr components, retaining only the Y component, we still have a fairly good idea about the features of the original image. In a more generic form, the conversion between RGB and YCbCr can be done using the following formulas: Y = krR + (1 - kb - kr)G + kbB Cb = ( B - Y ) / (2(1-kb)) Cr = ( R - Y ) / (2(1-kr))   R = Y + Cr(1-kr) / 2 B = Y + Cb(1-kb) / 2 G = Y - 2 [ Cb kb(1-kb) + Cr kr(1-kr) ] / (1 - kb - kr) ITU ( International Telecommunications Union ) recommendation BT.601 defines kb and kr to be 0.114 and 0.299 respectively. Substituting these values into the above equations, we obtain the following conversion equations: Y = 0.299R + 0.587G + 0.114B Cb = 0.564(B - Y),       Cr = 0.713(R - Y)   R = Y + 1.402Cr G = Y - 0.344Cb - 0.714Cr B = Y + 1.772Cb

   • YCbCr Sampling Formats

     We mentioned above that human eyes are more sensitive to luminance than to chromas and we can throw away some chroma data without degrading the image too much. In order to discard some chroma data in a convenient way, people group pixels together in groups of four. There are three commonly used formats known as 4:4:4, 4:2:2, 4:2:0 sampling.

     • 4:4:4 sampling means that the Y, Cb, and Cr components have the same resolution meaning that for every four Y samples, we have four Cb and Cr samples. In other words, a sample of each component exists at every pixel position as shown in the following figure, where black circles represent Y components, and blue and red circles represent Cb and Cr components respectively.

       For a true color RGB image, each color component ( R, G, and B ) of a pixel is 8-bit. When it is converted to 4:4:4 YCbCr format, we have the following information:

       Each sample has 8 bits.
       Each group ( 4 pixels ) has 3 x 4 = 12 samples.
       Total number of bits in a group = 8 x 12 = 96 bits.
       Average bits per pixel = 96 bits / 4 pixels = 24 bits / pixel.

     • In 4:2:2 sampling ( sometimes referred to as YUY2 ), for every four Y samples, we have two Cb and Cr samples; more precisely, Cb, Cr have the same vertical resolution as Y but half the horizontal resolution as shown below.

       In this case, we have

       each group has 4 + 2 x 2 = 8 samples,
       total = 8 x 8 = 64 bits, and
       average = 64 bits / 4 pixels = 16 bits / pixel.

     • In 4:2:0 sampling ( YV12 ), the name does not reflect what it means due to historical reasons; in this format, Cb and Cr each has half the horizontal and vertical resolution of Y; that is, for every four Y samples, we have one Cb and one Cr sample as shown below.

       So

       each group has 4 + 2 x 1 = 6 samples,
       total = 8 x 6 = 48 bits, and
       average = 48 bits / 4 pixels = 12 bits / pixel.

   We see that a true-color RGB image requires 24 bits / pixel to store but its corresponding 4:2:0 format only requires 12 bits / pixel. So when we convert an RGB image to the 4:2:0 sampling format, we have already compressed the image by a factor of two. Actually, such a process is called down sampling and is the first thing we need to do when we compress an image or a video.

5. Macroblock

   A PC image or frame with moderate size consists of a lot of pixels and require a large amount to storage space and computing power to process it. For example, an image of size 240 x 240 has 57600 pixels and requires 3 x 57600 / 2 = 86,400 bytes of storage space if 4:2:0 format is used. It is difficult and inconvenient to process all of these data simultaneously. In order to make things more manageable, an image is decomposed into macroblocks. A macroblock is a 16 x 16 pixel-region, which is the basic unit for processing a frame and is used in video compression standards like MPEG, H.261, H.263, and H.264. A macroblock has a total of 16x16 = 256 pixels.

   In our coding, we shall also process an image in the units of macroblocks. At the beginning, for simplicity and the convenience of discussion, we assume that each of our frames consists of an integral number of macroblocks. That is, both the width and height of an image are divisible by 16. We shall also use the 4:2:0 YCbCr format; a macroblock then consists of a 16x16 Y sample block, an 8x8 Cb sample block and an 8x8 Cr sample block. To better organize the data, we further divide the 16x16 Y sample block into four 8x8 sample blocks. Therefore, a 4:2:0 macroblock has a total of six 8x8 sample blocks; we label these blocks as shown in the figure below.

   

6. From RGB to YCbCr and Vice Versa

   Now let us write a simple program to convert from RGB to YCbCr and vice versa. Since the value of each RGB component is eight-bit and is always positive, it is natural for us to use data type unsigned char to represent it. The representation of a YCbCr component is a little more troublesome as real constants are involved in the conversion calculations. At the moment, let us simply use data type double to implement it. For demonstration purposes, the following piece of code converts a set of RGB values to YCbCr components.

   /* rgb2ybr0.cpp : simple program to demonstrate conversion from RGB to YCbCr and vice versa using ITU-R recommendation BT.601. Floating-point arithmetic is used. compile: g++ -o rgb2ybr0 rgb2ybr0.cpp execute: ./rgb2ybr0 */ #include <stdio.h> #include <stdlib.h> int main () { unsigned char R, G, B; //RGB components double Y, Cb, Cr; //YCbCr components //some sample values for demo R = 252; G = 120; B = 3; //convert from RGB to YCbcr Y = 0.299 * R + 0.587 * G + 0.114 * B; Cb = 0.564 * (B - Y); Cr = 0.713 * (R - Y); printf("\nOriginal RGB and corresponding YCbCr values:"); printf("\n\tR = %6d, G = %6d, B = %6d", R, G, B ); printf("\n\tY = %6.1f, Cb = %6.1f, Cr = %6.1f", Y, Cb, Cr ); //convert from YCbCr to RGB R = (unsigned char) (Y + 1.402 * Cr); G = (unsigned char) (Y - 0.344 * Cb - 0.714 * Cr); B = (unsigned char) (Y + 1.772 * Cb); printf("\n\nRecovered RGB values:"); printf("\n\tR = %6d, G = %6d, B = %6d\n\n", R, G, B ); return 0; }

   When executed, you should see the following output, which shows that the recovered RGB values are not identical to the original ones because of round-off errors.

   Original RGB and corresponding YCbCr values: R = 252, G = 120, B = 3 Y = 146.1, Cb = -80.7, Cr = 75.5 Recovered RGB values: R = 251, G = 120, B = 3

   Integer Arithmetic

   The above code illustrates how to make conversions between RGB and YCbCr. However, the code is not very practical in embedded systems as floating-point arithmetic has been used and when compressing an image, we have to apply the conversion to every pixel. If we change to integer-arithmetic, the calculation time can easily be shortened by a factor of two to three. Changing to integer-arithmetic here is quite simple because we can always approximate a real number as a fraction between two integers. For example, the coefficients for calculating Y from RGB can be expressed as

   0.299	=	19595 / 216
   0.587	=	38470 / 216
   0.114	=	7471 / 216

   So the trick to change to integer-arithmetic is to multiply the equation

   Y

   =

   0.299R + 0.587G + 0.114B

   by 2¹⁶, which becomes

   216Y

   =

   19595R + 38470G + 7471B

   Since R, G, B are integers, we can carry out the calculations using integer multiplications and then divide the result by 2¹⁶, which is the same as shifting right by 16. In summary, the following C statement shows the calculation of Y using integer-arithmetic.

   Y = (19595 * R + 38470 * G + 7471 * B ) >> 16;

   One should note that the sum of the coefficients is 2¹⁶ (i.e. 19595 + 38470 + 7471 = 65536 = 2¹⁶ ) corresponding to the requirement, k_r + k_g + k_b = 1 . If the values of R, G, and B are in the range [0, 255], then Y also lies within [0, 255]. This is because

   0 ≤ k_rR + k_gG + k_bB ≤ k_r255 + k_g255 + k_b255 = ( k_r + k_g + k_b ) 255 = 255

   and thus 0 ≤ Y ≤ 255. Therefore, Y can be represented by an 8-bit non-negative integer. One can also verify that

   -127 ≤ Cb ≤ 127
   -127 ≤ Cr ≤ 127

   Thus, Cb and Cr can be represented by 8-bit signed numbers. In our program, we can declare Y, Cb, Cr as short and if we need to save their values, we just need to save the lower eight bits of the short variable. The following program rgb2ybr1.cpp illustrates the use of integer-arithmetic to do RGB and YCbCr conversions. Note that in C/C++ language, when two short ( 16-bit ) variables are multiplied, they will be first promoted to int ( 32-bit ).

   /* rgb2ybr1.cpp : simple program to demonstrate conversion from RGB to YCbCr and vice versa using ITU-R recommendation BT.601. Integer-arithmetic is used. compile: g++ -o rgb2ybr1 rgb2ybr1.cpp execute: ./rgb2ybr1 */ #include <stdio.h> #include <stdlib.h> int main () { unsigned char R, G, B; //RGB components short Y, Cb, Cr; //YCbCr components //some sample values for demo R = 252; G = 120; B = 3; //convert from RGB to YCbcr Y = (short)((19595 * R + 38470 * G + 7471 * B ) >> 16); Cb = (short)( ( 36962 * ( B - Y ) ) >> 16); Cr = (short)(( 46727 * ( R - Y ) ) >> 16); printf("\nOriginal RGB and corresponding YCbCr values:"); printf("\n\tR = %6d, G = %6d, B = %6d", R, G, B ); printf("\n\tY = %6d, Cb = %6d, Cr = %6d", Y, Cb, Cr ); //convert from YCbCr to RGB, all constants have been multiplied by 65536 ( 2^16 ) R = ( unsigned char ) ( ( 65536 * Y + 91881 * Cr ) >> 16 ); G = ( unsigned char ) (( 65536 * Y - 22544 * Cb - 46793 * Cr ) >> 16); B = ( unsigned char ) ( ( 65536 * Y + 116129 * Cb ) >> 16); printf("\n\nRecovered RGB values:"); printf("\n\tR = %6d, G = %6d, B = %6d\n\n", R, G, B ); return 0; }

   When executed, you should see the following output.

   $ ./rgb2ybr1 Original RGB and corresponding YCbCr values: R = 252, G = 120, B = 3 Y = 146, Cb = -81, Cr = 75 Recovered RGB values: R = 251, G = 120, B = 2

   Again, some precision has been lost because of the right shift operation which is essentially a truncate operation. You may slightly improve your results if you add 2¹⁵ to the values before making a right-shift of 16; such an operation is essentially a round operation. Because of the rounding or truncating errors, some restored RGB values could lie outside the range [0, 255]; in this case we simply set those values to 0 or 255.

7. RGB and 4:2:0 YCbCr Transformation for Video Frames

   In this section, we discuss the conversion between RGB and 4:2:0 YCbCr formats for video frames. We process the data in units of macroblocks. As we discussed above, a macroblock has four 8x8 Y sample blocks and one 8x8 sample block for each of Cb and Cr. At this moment, we assume that a frame has an integral number of macroblocks. In the 4:2:0 YCbCr format, for each RGB pixel, we make a conversion for Y but we only make a conversion for Cb and Cr for every group of four RGB pixels. Each group is formed by grouping 4 neighbouring pixels ( see figure at the right ); for simplicity, when calculating the Cb, and Cr components, we simply use the upper left pixel of the four and ignore the other three. For convenience of programming, we define three structs, RGB for holding RGB values of a pixel, YCbCr for holding Ycbcr values of a pixel, and YCbCr_MACRO for holding the YCbCr values of a macroblock ( 16 x 16 pixels ) as follows.

   //defines an RGB pixel typedef struct { unsigned char R; //0 - 255 unsigned char G; //0 - 255 unsigned char B; //0 - 255 } RGB; typedef struct { short Y; //0 - 255 short Cb; //-127 - 127 short Cr; //-127 - 127 } YCbCr; //4:2:0 YCbCr Macroblock typedef struct { short Y[256]; //16x16 ( four 8x8 samples ) short Cb[64]; //8x8 short Cr[64]; //8x8 } YCbCr_MACRO;

   We put this definitions in the header file "common.h". We define the function rgb2ycbcr ( RGB &a, YCbCr &b ) to convert an RGB pixel a to a YCbCr pixel b; the function rgb2y( RGB &a, short &y ) converts an RGB pixel a to a Y component y. Now, we need a function to convert an entire RGB macroblock to a 4:2:0 YCbCr macroblock, which consists of four 8x8 Y sample blocks, one 8x8 Cb sample block and one 8x8 Cr sample block. The following function macroblock2ycbcr() does the job; the input can be thought of as a 16x16 two dimensional array, macro16x16[][] holding 16x16 RGB pixels; the output is a pointer to a YCbCr_MACRO struct defined above, which holds the converted Y, Cb, Cr sample values.

   /* Convert an RGB macro block ( 16x16 ) to 4:2:0 YCbCr sample blocks ( six 8x8 blocks ). */ void macroblock2ycbcr ( RGB *macro16x16, YCbCr_MACRO *ycbcr_macro ) { int i, j, k, r; short y; YCbCr ycb; r = k = 0; for ( i = 0; i < 16; ++i ) { for ( j = 0; j < 16; ++j ) { if ( !( i & 1 ) && !( j & 1 ) ) { //need one Cb, Cr for every 4 pixels rgb2ycbcr ( macro16x16[r], ycb ); //convert to Y and Cb, Cr values ycbcr_macro->Y[r] = ycb.Y; ycbcr_macro->Cb[k] = ycb.Cb; ycbcr_macro->Cr[k] = ycb.Cr; k++; } else { //only need the Y component for other 3 pixels rgb2y ( macro16x16[r], ycbcr_macro->Y[r] ); } r++; //convert every pixel for Y } } }

   Similarly, we can define a function, ycbcr2macroblock() to convert a YCbCr macroblock to an RGB macroblock. The following program, rgb_ybr.cpp contains all the functions we need to convert video frames from RGB to YCbCr and back.

   /* rgb_ybr.cpp Containing functions for converting RGB to YCbCr and vice versa. */ #include <stdio.h> #include <stdlib.h> #include "common.h" #include "rgb_ybr.h" /* Convert from RGB to YCbCr using ITU-R recommendation BT.601. Y = 0.299R + 0.587G + 0.114B Cb = 0.564(B-Y) Cr = 0.713(R-Y) Integer arithmetic is used to speed up calculations: 0.299 ~ 19595 / 2^16, 0.587 ~ 38470 / 2^16 ..... Input: an RGB pixel Output: a YCbCr "pixel". */ void rgb2ycbcr( RGB &rgb, YCbCr &ycb ) { //coefs summed to 65536 ( 1 << 16 ), so Y is always within [0, 255] ycb.Y = (short)((19595 * rgb.R + 38470 * rgb.G + 7471 * rgb.B ) >> 16); ycb.Cb = (short)( ( 36962 * ( rgb.B - ycb.Y ) ) >> 16); ycb.Cr = (short)(( 46727 * ( rgb.R - ycb.Y ) ) >> 16); } //just convert an RGB pixel to Y component void rgb2y( RGB &rgb, short &y ) { y = (short)((19595 * rgb.R + 38470 * rgb.G + 7471 * rgb.B ) >> 16); } //taking care of round off errors; ensure RGB values lie within [0, 255] int chop_rgb ( int x ) { if ( x < 0 ) { x = 0; } else if ( x > 255 ) { x = 255; } return x; } /* Convert from YCbCr to RGB domain. Using ITU-R standard: R = Y + 1.402Cr G = Y - 0.344Cb - 0.714Cr R = B = Y + 1.772Cb Integer arithmetic is used to speed up calculations. */ void ycbcr2rgb( YCbCr &ycb, RGB &rgb ) { int Y = ( int ) ycb.Y << 16; //same as multiply 65536 rgb.R = chop_rgb ( ( Y + (int)91881 * ycb.Cr ) >> 16 ); rgb.G = chop_rgb ( ( Y - (int)22544 * ycb.Cb - (int)46793 * ycb.Cr ) >> 16 ); rgb.B = chop_rgb ( ( Y + (int) 116129 * ycb.Cb ) >> 16 ); } /* Convert an RGB macro block ( 16x16 ) to 4:2:0 YCbCr sample blocks ( six 8x8 blocks ). */ void macroblock2ycbcr ( RGB *macro16x16, YCbCr_MACRO *ycbcr_macro ) { int i, j, k, r; short y; YCbCr ycb; r = k = 0; for ( i = 0; i < 16; ++i ) { for ( j = 0; j < 16; ++j ) { if ( !( i & 1 ) && !( j & 1 ) ) { //need one Cb, Cr for every 4 pixels rgb2ycbcr ( macro16x16[r], ycb ); //convert to Y and Cb, Cr values ycbcr_macro->Y[r] = ycb.Y; ycbcr_macro->Cb[k] = ycb.Cb; ycbcr_macro->Cr[k] = ycb.Cr; k++; } else { //only need the Y component for other 3 pixels rgb2y ( macro16x16[r], ycbcr_macro->Y[r] ); } r++; //convert every pixel for Y } } } /* Convert the 6 8x8 YCbCr sample blocks to RGB macroblock ( 16x16 ). */ void ycbcr2macroblock( YCbCr_MACRO *ycbcr_macro, RGB *macro16x16 ) { int i, j, k, r; short y; YCbCr ycb; r = k = 0; for ( i = 0; i < 16; ++i ) { for ( j = 0; j < 16; ++j ) { if ( !( i & 1 ) && !( j & 1 ) ) { //one Cb, Cr has been saved for every 4 pixels ycb.Y = ycbcr_macro->Y[r]; ycb.Cb = ycbcr_macro->Cb[k]; ycb.Cr = ycbcr_macro->Cr[k]; ycbcr2rgb ( ycb, macro16x16[r]); k++; } else { ycb.Y = ycbcr_macro->Y[r]; ycbcr2rgb( ycb, macro16x16[r] ); } r++; } } }

   After we have implemented the functions to convert an RGB macroblock to a YCbCr macroblock and vice versa, we can easily utilize these functions to convert an image frame from RGB to YCbCr and save the data. As 4:2:0 format is used, the saved YCbCr data are only half as much as the original RGB data. The following program encode.cpp, which consists of functions save_ybrblocks(), and encode() will do the job. Note that the standard C function putc() saves only the lower 8-bit of an integer and this is what we want; we can use getc() to read the byte back.

   /* encode.cpp Contains functions to convert an RGB frame to YCbCr and to save the converted data. */ #include <stdio.h> #include <stdlib.h> #include "common.h" #include "rgb_ybr.h" extern short width; //image width extern short height; //image height //save one YCbCr macroblock. void save_ybrblocks( YCbCr_MACRO *ycbcr_macro, FILE *fpo ) { short block, i, j, k, *py; //save four 8x8 Y sample blocks for ( block = 0; block < 4; block++ ) { if ( block < 2 ) py = ( short * ) &ycbcr_macro->Y + 8*block; //points to beginning of block else py = (short *)&ycbcr_macro->Y + 128 + 8*(block-2);//points to beginning of block for ( i = 0; i < 8; i++ ) { //one sample-block if ( i > 0 ) py += 16; //advance py by 16 ( length of one row ) for ( j = 0; j < 8; j++ ) { putc ( ( int ) *(py+j),fpo); //save one byte of data } } } //save one 8x8 Cb block k = 0; for ( i = 0; i < 8; ++i ) { for ( j = 0; j < 8; ++j ) { putc( ( int ) ycbcr_macro->Cb[k++], fpo ); } } //save one 8x8 Cr block k = 0; for ( i = 0; i < 8; ++i ) { for ( j = 0; j < 8; ++j ) { putc( ( int ) ycbcr_macro->Cr[k++], fpo ); } } } /* Convert RGB to YCbCr and save the converted data. width, height are global variables of image. */ void encode ( char *image, FILE *fpo ) { short row, col, i, j, r; RGB macro16x16[256]; //16x16 pixel macroblock; assume 24-bit for each RGB pixel YCbCr_MACRO ycbcr_macro; //macroblock for YCbCr samples RGB *p; //pointer to an RGB pixel static int nframe = 0; for ( row = 0; row < height; row += 16 ) { for ( col = 0; col < width; col += 16 ) { p = ( RGB *) image + ( row * width + col ); //points to beginning of macroblock r = 0; //note pointer arithmetic for ( i = 0; i < 16; ++i ) { for ( j = 0; j < 16; ++j ) { macro16x16[r++] = (RGB) *p++; } p += ( width - 16 ); //points to next row within macroblock ( note pointer arithmetic ) } macroblock2ycbcr ( macro16x16, &ycbcr_macro ); //convert from RGB to YCbCr save_ybrblocks( &ycbcr_macro, fpo ); //save one YCbCr macroblock } //for col } //for row }

   Combining these with the multi-threaded player we discussed in Chapter Raw Video Player, we can write a program to convert all the RGB frames to YCbCr frames and play back. The program typlayer.cpp is a sample program to illustrate this; in order not to distract readers from other housekeeping and related codes some parameters are hard-coded and assumptions are made to simplify the code. We assume that we read raw RGB video data from a file named "sample_video.raw" ( We have discussed in Chapter Video Formats how to extract raw video data from an AVI file. ) We also assume that the image size is 320x240 and hard-code this parameter in the program. When the program is executed, it reads RGB video data from "sample_video.raw", plays the video on the screen, convert the RGB data to YCbCr, and save the YCbCr data in a file named "sample_video.ybr". As you can see, the size of the file "sample_video.ybr" is only half as big as that of "sample_video.raw"i, which consists of raw RGB data. If we provide an extra parameter "y" to the program when executing it ( i.e. ./typlayer y ), the program then reads YCbCr data from "sample_video.ybr", converts the data to RGB and plays the video; you should see that the video playback looks idential to the playback using "sample_video.raw".

   /* typlayer.cpp Hard-coded program to illustrate the conversion from RGB to 4:2:0 YCbCr and vice vers. Assume that our video is 320x240 8-bit data. It converts raw RGB data of "sample_video.raw" to YCbCr data saved in "sample_video.ycb" and plays the video. Or if an argument "y" is provided, it reads YCbCr data from "sample_video.ycb", converts the data to RGB and plays the video. Compile:g++ -c typlayer.cpp -I/usr/include g++ -o typlayer typlayer.o rgb_ybr.o encode.o -lSDL -lSDL_image -lpthread Execute: ./typlayer     or     ./typlayer y See 'http://www.webkinesia.com/games/vcompress.php' */ #include <SDL/SDL.h> #include <SDL/SDL_thread.h> #include <stdio.h> #include <malloc.h> #include <stdlib.h> #include <assert.h> #include "common.h" #include "encode.h" #include "rgb_ybr.h" short ycb_flag = 0; //0: read RGB file, convert to YCbCr; 1: read YCbCr file to play video FILE *fpi, *fpo; short width = 320; //image width short height = 240; //image height unsigned long frameSize; //size of one frame unsigned long head=0, tail=0; //for synchronization of producer and consumer threads bool quit = false; //quit program if set to true char *buf[4]; //ring buffer to hold four frames of data //Consumer int player ( void *scr ) { SDL_Surface *screen = ( SDL_Surface * ) scr; while ( !quit || head < tail ) { if ( head >= tail ) { //buffer empty (data not available yet ) SDL_Delay ( 30 ); //sleep for 30 ms continue; } //consumes the data if ( !ycb_flag ) //convert RGB data to YCbCr and save it encode (buf[head%4],fpo); //encode and save there //if ycb_flag == 1, we just play the data in buf[] screen->pixels = buf[head%4]; SDL_UpdateRect ( screen, 0, 0, 0, 0 ); //update whole screen head++; SDL_Delay ( 40 ); //play ~ 20 fps, should use timer to calculate exact delay } //while return 0; } /* Get YCbCr data from file pointed by fpi. Put the four 8x8 Y sample blocks, one 8x8 Cb sample block and one 8x8 Cr sample block into a struct of YCbCr_MACRO. Return: number of bytes read from file and put in YCbCr_MACRO struct. */ int get_ybrblocks( YCbCr_MACRO *ycbcr_macro ) { short r, row, col, i, j, k, n, block, *py; short c; n = 0; //read data from file and put them in four 8x8 Y sample blocks for ( block = 0; block < 4; block++ ) { if ( block < 2 ) py = ( short * ) &ycbcr_macro->Y + 8*block; //points to beginning of block else py = (short *)&ycbcr_macro->Y + 128 + 8*(block-2); //points to beginning of block for ( i = 0; i < 8; i++ ) { //one sample-block if ( i > 0 ) py += 16; //advance py by 16 ( length of one row of macroblock ) for ( j = 0; j < 8; j++ ) { if ( ( c = getc ( fpi )) == EOF) //read one byte break; *( py + j ) = c; //put it in YCbCr_MACRO struct n++; } //for j } //for i } //for block //now do that for 8x8 Cb block k = 0; for ( i = 0; i < 8; ++i ) { for ( j = 0; j < 8; ++j ) { if ( ( c = getc ( fpi )) == EOF ) break; if ( c > 127 ) c -= 256; //convert from 8-bit unsigned to sign ycbcr_macro->Cb[k++] = c; n++; } } //now do that for 8x8 Cr block k = 0; for ( i = 0; i < 8; ++i ) { for ( j = 0; j < 8; ++j ) { if ( ( c = getc ( fpi )) == EOF ) break; if ( c > 127 ) c -= 256; //convert from 8-bit unsigned to sing ycbcr_macro->Cr[k++] = c; n++; } } return n; //number of bytes read } /* Convert a YCbCr frame to RGB frame. */ int decode_ybrFrame ( char *image ) { short r, row, col, i, j, k, block, *py; int n; RGB *p, macro16x16[256]; YCbCr_MACRO ycbcr_macro; for ( row = 0; row < height; row += 16 ) { for ( col = 0; col < width; col += 16 ) { n = get_ybrblocks( &ycbcr_macro ); if ( n <= 0 ) return n; ycbcr2macroblock( &ycbcr_macro, macro16x16 ); p = ( RGB *) image + ( row * width + col ); //points to beginning of macroblock r = 0; for ( i = 0; i < 16; ++i ) { for ( j = 0; j < 16; ++j ) { *p++ = macro16x16[r++]; } p += ( width - 16 ); //points to next row within macroblock } } //for col } //for row return n; } int decode_frame ( char *image ) { int n; n = fread ( image, 1, frameSize, fpi ); return n; } //Producer int decoder ( void *data ) { int n; char *image; while ( !quit ) { if ( tail >= head + 4 ) { //buffer full SDL_Delay ( 30 ); //sleep for 30 ms ( not the best method, better use //semaphore, let consumer wake you up continue; } //produce data image = buf[tail%4]; // n = fread ( buf[tail%4], frameSize, 1, fp ); if ( !ycb_flag ) n = decode_frame ( image );//decoded data put in image[] else n = decode_ybrFrame ( image ); if ( n <= 0 ) quit = true; else tail++; } //while return 0; } int main( int argc, char *argv[] ) { SDL_Surface *screen; frameSize = width * height * 3; SDL_Thread *producer, *consumer; SDL_Event event; int status; char *key; if ( argc > 1 && ( strcmp ( argv[1], "y" ) == 0 ) ) { //play from YCbCr file ycb_flag = 1; fpi = fopen ( "sample_video.ybr", "rb" ); //input of YCbCr data if ( fpi == NULL ) { printf("\nError opening file \"sample_video.ybr\" for input\n" ); return 1; } } else { //play from RGB file and convert to YCbCr ycb_flag = 0; fpi = fopen ( "sample_video.raw", "rb" ); //input of RGB data if ( fpi == NULL ) { printf("\nError opening file \"sample_video.raw\" for input\n" ); return 1; } fpo = fopen ( "sample_video.ybr", "wb" ); //output of YCbCr data if ( fpo == NULL ) { printf("\nError opening file \"sample_video.ybr\" for output\n" ); return 1; } } //initialize video system if ( SDL_Init( SDL_INIT_VIDEO ) < 0 ) { fprintf(stderr, "Unable to init SDL: %s\n", SDL_GetError()); exit(1); } //ensure SDL_Quit is called when the program exits atexit(SDL_Quit); //set video mode of 320 x 240 with 24-bit pixels screen = SDL_SetVideoMode(width, height, 24, SDL_SWSURFACE); if ( screen == NULL ) { fprintf(stderr, "Unable to set 320x240 video: %s\n", SDL_GetError()); exit(1); } for ( int i = 0; i < 4; ++i ) { buf[i] = ( char * ) malloc ( frameSize ); assert ( buf[i] ); } head = tail = 0; consumer = SDL_CreateThread ( player, screen ); producer = SDL_CreateThread ( decoder, ( void * ) "decdoing" ); while (!quit) { status = SDL_WaitEvent(&event); //wait indefinitely for an event to occur //event will be removed from event queue if ( !status ) { //Error has occured while waiting printf("SDL_WaitEvent error: %s\n", SDL_GetError()); quit = true; return false; } switch (event.type) { //check the event type case SDL_KEYDOWN: //if a key has been pressed key = SDL_GetKeyName(event.key.keysym.sym); printf("The %s key was pressed!\n", key ); if ( event.key.keysym.sym == SDLK_ESCAPE ) //quit if 'ESC' pressed quit = true; else if ( key[0] == 'q' ) //quit if 'q' pressed quit = true; //same as "if ( event.key.keysym.sym == SDLK_q )" } SDL_Delay ( 100 ); //give up some CPU time } SDL_WaitThread ( consumer, NULL ); //wait for child threads to finish SDL_WaitThread ( producer, NULL ); return 0; }

   For your convenience, we list below the links to all the programs discussed and the Makefile required to compile typlayer. The raw RGB data file "sample_video.raw" is also provided; when you click on the link to download it, simply name it as "sample_video.raw" ( without and .zip extension ); the file is not zipped.

   common.h
   rgb_ybr.h
   encode.h
   rgb_ybr.cpp
   encode.cpp
   typlayer.cpp
   Makefile
   sample_video.raw