An important part of the codec development process is having the right tools that allow us to carefully analyze the performance of the codec over time. During the initial development of cairo I built four indispensable tools: convert, player, inspect, and analyze. In addition to their functionality, these tools also serve as simple code tutorials for using the cairo API as well as reading and writing evx files.
Cairo defines a new video file container and data format, bundled within .evx files. All subsequent variations of my experimental codecs (e.g. chennai, capri) use this format as well. The player tool is a simple viewer utility that parses evx files and plays back their content. The convert tool uses FFmpeg to convert existing and popular containers and video formats into the cairo file format.
Convert and player are available in both binary and source code formats from the links below:
Want to test out the player tool without having to convert your own content to evx? Try the following sample evx files:
Big Buck Bunny Clip
1280x720 @ 29.97 Hz
High Quality, 40 MB, 1.0 Mbps
Medium High Quality, 22 MB, 0.56 Mbps
Medium Quality, 15 MB, 0.38 Mbps
Snow Fall Clip
960x540 @ 29.97 Hz
High Quality, 19 MB, 0.55 Mbps
Medium High Quality, 12 MB, 0.35 Mbps
Medium Quality, 9 MB, 0.27 Mbps
Sony 4k Demo Clip
1280x720 @ 30 Hz
High Quality, 35 MB, 1.95 Mbps
Medium High Quality, 20 MB, 1.15 Mbps
Medium Quality, 12 MB, 0.72 Mbps
Microsoft Wildlife Clip
640x480 @ 30 Hz
High Quality, 22 MB, 0.8 Mbps
Medium High Quality, 15 MB, 0.55 Mbps
Medium Quality, 9 MB, 0.35 Mbps
The inspect tool allows you to step through the execution of the encoder and visually examine its internal state. Inspect can render a number
of combined tables including the prediction state, block table, quantization table, per-block image variance, sub-pixel motion table,
and more. Inspect can be very useful in tracking down hard to find bugs.
Inspect is available in both binary and source code formats from the links below:
The following example images are taken from the inspect tool running the Big Buck Bunny test video:
Source Image View
Press 1 to enable in the inspect tool
The incoming raw image to encode. This is compared against the destination image for a quick visual check on compression quality.
Block Table View
Press 3 to enable in the inspect tool
Red identifies intra (versus inter), while green indicates motion and blue identifies
skip blocks. Thus, a black macroblock implies an inter predicted block without motion vectors.
Quantization Table View
Press 4 to enable in the inspect tool
This image identifies the quantization levels for each macroblock in the image. The lighter the pixel the higher the quantization,
and thus the greater the information loss and efficiency.
Sub-pixel Motion Table View
Press 5 to enable in the inspect tool
Identifies the macroblocks in the image that used sub-pixel motion compensation. Green indicates half-pixel interpolation was used,
while blue indicates the use of quarter-pixel interpolation.
Variance Table View
Press 6 to enable in the inspect tool
Identifies the pixel variance for each luminance macroblock in the image. Block variance is used to perform variance adaptive quantization.
Output Image View
Press 2 to enable in the inspect tool
The final reconstructed image. The peak signal to noise ratio (PSNR) is computed using the destination and source images.
Lastly, the analyze tool is a robust visualization utility that enables the codec developer to deeply inspect and step through the execution of their pipeline. This tool relies on functionality in convert, inspect, and player, but adds significantly more analysis support and debug helpers. During the development of cairo I relied heavily on this tool to measure performance and efficiency, detect and root cause visual artifacts, and assess the overall impact of a change.
The analyze tool allows the user to load an image, video file, or a live desktop capture, and then process the content using cairo and analyze the results.
The left video view in this screenshot shows the current source frame, which in this case is an h.264 encoded 1280x720 frame. The center image is togglable, and currently shows the state of cairo's most recent prediction frame. The rightmost image is the cairo output of the source frame.
Note that while our source is in a compressed format for this example, in practice only raw content formats are used for experimentation.
In this example, the tool read an h.264 source frame (49 KB compressed), decompressed the source frame into a raw image (2.7 MB), and then compressed the raw image using the cairo codec to produce a re-encoded output image (20 KB). The re-encoded image has a reasonable PSNR of 39.33 and a mean squared error of 7.58. Note that while our source is in a compressed format for this example, in practice only raw content formats are used for experimentation.
As mentioned, the center image can be changed to examine the internal state of cairo. This includes views of the source, color converted source, all prediction frames, motion block data, transformed frame, quantized frame, and more.
In the screenshot above we examine the motion block table for the current frame. Each block is color coded by type to indicate the form of motion compensation. For example, green blocks indicate a skip, teal indicates a motion-skip, and blue indicates a general motion predicted block.
The analyze tool wasn't designed to be a general purpose image and video viewer, so it lacks many common features such as audio playback. It does however have a number of features
that are useful to codec developers:
Content Visualization
Load any video or image format supported by FFMpeg and play, pause, and skip around the content. Save out individual frames or sections of the cairo encoded output.
Inspect individual blocks or pixels to ensure they have the correct values.
Zoom and Filter
The analyze tool is built on Imagine, so you can zoom in to inspect a detail, or apply
an array of filters to distort the content in useful ways.
Encode Order Control
Re-order, repeat, or skip frames as they are fed into cairo to quickly run experiments on the content. The encoder and decoder can also be reset or reverted to a particular snapshot.
Control of the Pipeline
In addition to allowing you to inspect each stage of the cairo pipeline, the analyze tool also enables you to modify and re-try parts of the codec. For example, you can
change block types, set new quantization matrices, change the transform, update the quality, or toggle entire stages of the pipeline.
For more information about my codec projects, check out the compression section of this site. For articles about compression including much of the logic found in cairo, check out my blog.
