Video Quality Graceful Degradation

Sunday, July 5th, 2015

Rate control, in the context of video streaming, is the continual process of measuring a user's network connection quality and adjusting the video quality parameters to ensure a smooth viewing experience. It's difficult to get right, as it must account for several dynamic factors including the encoder and decoder processing performance and unforeseeable fluctuations in network conditions.

Furthermore, due to the pipelined nature of streaming and the bullwhip effect of congestion, recovery systems typically need to over-compensate for fluctuations by abruptly stepping quality down to a level that the network can sustain. Some of these abrupt changes are more noticable than others, due to the nonlinear relationship between bitrate and perceived quality.

This is a vital component of video streaming services like Amazon Instant Video, Netflix and Hulu because they enable viewers to use their services uninterrupted despite wildly changing network conditions that may be impacted by all kinds of factors (e.g. the number of users on the network, what those users are doing, capacity outages, etc).

Standard Behavior

Surprisingly, there does not seem to be a general consensus on the best way to handle the graceful degredation of a video stream, as each of the aforementioned streaming services offer significantly different user experiences in this area. As an avid user of these services I'm often painfully aware of their shortcomings, so I've put together a short list of what I would consider the minimum set of graceful degredation features and behaviors that all large scale video streaming services should offer.

 

Buffering Percentage

This is extremely important and it's glaringly noticable when absent. Any time the video must be paused in order to buffer the user must be given a percentage or progress indicator so that they are assured of active progress plus an estimate of when video playback will resume.

When the app or website offers a buffer bar instead of a precentage then the bar should indicate to the user the amount of buffering needed in order to avoid further interruptions at current video and network quality levels.

 

Multiple Quality Levels

Most services use a precoded matrix of videos at various quality levels for different network levels and resolutions. When a change in quality is needed, the service is able to simply switch to a different video feed. Alternatively, for live streaming solutions the video encode parameters may be adjusted to dynamically adjust the video quality.

This ensures that the video stream remains fluid and intact if the network quality falls within a survivable level.

 

Audio Only

When network quality degrades so severely that even the lowest quality video setting is unacceptable, the stream can switch to a mode commonly known as storyboard. In this mode, the user is visually alerted that their stream is suffering and the video is presented at a very low framerate (0.25 to 1 frame per second).

The regular audio track accompanies the visual storyboard as this enables the user to continue to listen to the content and decide whether they wish to pause playback until the network conditions improve.

 

Degrade Quickly, Recover Slowly

Once the decision has been made to lower the video quality level, it's important to drop to a sufficient level that the network can realistically sustain. Prompting the user multiple times to buffer, or repeated drops followed by rapid improvements to video quality should be avoided.

 

Remain Responsive

The interface should remain responsive even when network quality has significantly suffered. Regardless of whether the system is buffering, the user should always be able to pause, seek, and halt the stream.





Cherry On Top

In addition to the features above, here are some nice to haves that can significantly improve the user experience. These features were offered in the first release of everyAir and received positive feedback.


 

Configurable Quality

Allow power users to set a maximum quality level. The 8k display on my 4.7" iPhone 8S doesn't need to be powered by 8k video, particularly over LTE. Given the small display size I would prefer a simple 720p stream that saves my bandwidth.

This is a common feature in streaming websites, but is often missing from TV and native mobile apps.

 

Drop to Black and White

A black and white stream can save up to 33% of bandwidth versus full color, and many users won't even notice the sudden change. If this doesn't seem like a significant improvement, consider that converting low quality streams with obvious blocking artifacts to black and white can salvage a third of the visual quality without an increase in the cost.

In user testing for everyAir we found that viewers were much more sensitive to large blocking artifacts and choppy framerates than they were to a loss in color.





Wrap Up

Did I miss a critical feature? Feel free to drop me a line and I'll add it to the list :-)