Sunday, October 11th, 2015
Several of the most popular virtual and augmented reality experiences involve video.
These videos come in a variety of formats including combinations of stereoscopic 3D,
360° panoramas and spherical views. Unfortunately, these formats place significant
strain on our processors, memory, and network bandwidth due to their increased requirements for
resolution, framerate and latency.
To cope with this trifecta of video compression challenges, we need a video codec that capitalizes
on the potential for increased levels of self similarity and inter-frame correlation within these kinds
of videos.
The following is a list of basic mixed-reality centric features that could be integrated into a modern
video codec to help achieve higher compression efficiency for VR content, while lowering processor and
bandwidth costs.
Sunday, July 5th, 2015
Rate control, in the context of video streaming, is the continual process of measuring a user's network connection quality and adjusting the video quality parameters
to ensure a smooth viewing experience. It's difficult to get right, as it must account for several dynamic factors including the encoder and decoder processing
performance and unforeseeable fluctuations in network conditions.
Furthermore, due to the pipelined nature of streaming and the bullwhip effect of congestion, recovery systems typically need to over-compensate
for fluctuations by abruptly stepping quality down to a level that the network can sustain. Some of these abrupt changes are more noticable than
others, due to the nonlinear relationship between bitrate and perceived quality.
Sunday, June 28th, 2015
I've been lecturing to university students about video compression. The slides are available here, but one section in particular seems to stand out to the audience: an abstract but approachable introduction to the Fourier Transform. This mathematical operation is fundamental to video compression, but it can be a bit tricky to understand (and teach!).
So let's assume we own a bar, and it's full of rowdy patrons who are all demanding piña coladas. As it turns out, we have a large pre-made batch of the drink that's ready to serve, but it unfortunately contains a very expensive rum. We'd like to serve the drink, but without the expensive rum in it.
Fortunately, we have a magical device, called the DCT-1000, which can separate the piña colada mixture into its constituent ingredients.
Saturday, June 27th, 2015
Cairo is a simple video codec that I developed in early 2011 during the later stages of everyAir.
The purpose of this codec was to experiment with compression features and learn about their individual
impact on efficiency while developing a simple framework for future work.
Video codecs contain a lot of features such as subpixel motion estimation, differential coding, quantization,
deblocking, rate control, entropy coding, and many more. Each feature must be adaptively configured to perform well despite
abrupt changes in the source video. Balancing these features with overall frame quality and size can be a daunting task,
and codecs that do it well will outperform those that don't.
It's been over four years since Cairo was completed and I thought it might be interesting to dust it off and visualize the
benefits of some of its features. Specifically, I was interested in comparing the effects of its generic motion compensation,
sub-pixel motion compensation, differential coding, and quality controlled quantization.
Saturday, April 18th, 2015
Denoising is a very useful process that's used in video compression, computer vision, video capture pipelines, and more.
Anywhere that noise may find its way into an image, denoising filters are there to help clean things up. There are many
different kinds of noise, and fortunately there are equally many ways of correcting it.
Modern video codecs include denoising filters to mitigate some of the artifacts produced by the encoding process.
The Cairo codec is no exception, and uses a simple denoise filter during
image decoding to smooth out noise that may find its way into a frame.
As mentioned, there are many different kinds of noise, but I'd like
to focus this article on one particular type: salt and pepper noise. This kind of noise presents as speckled light and dark
pixels that can give a frame a grainy look and degrade the clarity of the image. Sometimes these artifacts are desirable,
as they can mimic film grain or other visual qualities, but when they are introduced by the codec, it's best to rectify them.
Friday, March 20th, 2015
Balls Away is a fast paced action game for iOS that pits you against a horde of evil balls bent on the destruction of a small rabbit colony. Leveraging the Vision 3 game engine, Balls Away features a rich 3D world with randomly generated levels and intense gameplay.
For more information about this game, visit its project page. For more information about the game engine that powers Balls Away, check out the Vision 3 project page.
