The first step in developing a new video codec is building the tools that enable deep inspection and visualization of codec state.
These tools provide essential scaffholding for diagnosing visual artifacts and efficiency anomalies which can otherwise be difficult
to detect and root cause. During the development of Cairo I built four tools: analyze, convert, inspect, and player.
Convert and player are minimalist programs that demonstrate proper usage of the evx interface. These were created shortly after the development of Cairo as part of a tutorial series, and to allow users to quickly test out Cairo with their existing media. As the name suggests, convert will convert most standard video files (thanks ffmpeg!) to a simple evx file format. Player is a simple video player for these evx files.
Inspect allows you to step through the execution of the encoder and visually examine its internal state. Inspect can render a number of combined tables including the prediction state, block table, quantization table, per-block image variance, sub-pixel motion table, and more. Inspect can be very useful in tracking down hard to find bugs.
Lastly, analyze is a robust visualization tool that enables the codec developer to deeply inspect and step through the execution of their pipeline. This tool relies on functionality in convert, inspect, and player, but adds significantly more analysis tools and debug helpers. During the development of Cairo I relied heavily on this tool to measure performance and efficiency, detect and root cause visual artifacts, and assess the overall impact of a change. For more information about this tool, check out its project page.
Convert, inspect, and player are available in both source and binary formats (sorry, limited platform support for now!). Use convert to create
evx files from your own media, and then play these files using player.
Note that if you convert using a compressed source (e.g. an h.264 compressed video) then quality and efficiency will likely suffer in the output. When possible, try to convert from raw uncompressed sources for the best results.
Analyze tool (12.4 MB) (coming soon!)
If you don't have any test content of your own but would still like to try out player, download the following sample evx files.
Inspect allows you to examine the internal state of the encoder by viewing images of block state. These images, combined with additional analysis, enable the developer to step through the life of each pixel in the analyze tool. The following example images are taken from the inspect tool running the Big Buck Bunny test video.
Source Image View
Press 1 to enable in the inspect tool
The incoming raw image to encode. This is compared against the destination image for a quick visual check on compression quality.
Block Table View
Press 3 to enable in the inspect tool
Red identifies intra (versus inter), while green indicates motion and blue identifies skip blocks. Thus, a black macroblock implies an inter predicted block without motion vectors.
Quantization Table View
Press 4 to enable in the inspect tool
This image identifies the quantization levels for each macroblock in the image. The lighter the pixel the higher the quantization, and thus the greater the information loss and efficiency.
Sub-pixel Motion Table View
Press 5 to enable in the inspect tool
Identifies the macroblocks in the image that used sub-pixel motion compensation. Green indicates half-pixel interpolation was used, while blue indicates the use of quarter-pixel interpolation.
Variance Table View
Press 6 to enable in the inspect tool
Identifies the pixel variance for each luminance macroblock in the image. Block variance is used to perform variance adaptive quantization.
Output Image View
Press 2 to enable in the inspect tool
The final reconstructed image. The peak signal to noise ratio (PSNR) is computed using the destination and source images.
Similar to the public release of Cairo (upon which these packages are based), convert and player are reference implementations
that demonstrate the efficiency of Cairo, but lack performance optimizations.
The optimized production version of Cairo will receive a later release because it's both harder to read and full of hardware and platform dependencies that make it less suitable for students.
For those interested in checking out optimized builds of Cairo, check out the everyAir project that features Cairo on PC, Mac, and iOS. For those interested in tuning the public Cairo source release I would suggest a few low hanging fruit:
Add multi-threading to process images in slices.
Add vector instructions to improve convert.cpp, transform.cpp, and motion.cpp.
Optimize transform.cpp using Chen et al.'s fast DCT algorithm (or similar).
Move convert.cpp to the GPU (YUV to RGB right before presentation).
There are of course more involved optimizations (e.g. porting entirely to the GPU), but most of these are already in the production version of Cairo and are less straightforward than the items listed above. If you absolutely need the extra performance then please drop me a line and we'll see if we can save you a bit of time :-)