Broadband TV Conference Part 2: How to Measure Streaming Video Quality

Introduction:

This second article on the 2014 Broadband TV Conference summarizes a presentation by OPTICOM’s CEO on streaming video quality measurements. We think that topic will be very important for many players in the OTT streaming video and connected TV markets. In particular, we believe it’ll be quite valuable for adaptive bit rate OTT and mobile video streaming providers, in order to measure and then attempt to improve the Quality of Experience (QoE) of their customers.

Perceptual Quality Measurement of OTT Streaming Video TV Services, Michael Keyhl, CEO of OPTICOM

How do you measure streaming video quality? Very few seem to have good metrics on video Quality of Experience (QoE) for viewers, even though it impacts many participants in the OTT, SD/HD video content delivery business. The stakeholders involved in QoE for video subscribers/consumers include: content provider, OTT provider, pay TV providers (cable, satellite, telco), network operators (especially for mobile video consumption), device makers, video codec providers and mobile apps companies including Internet videos in their apps.

Michael Keyhl, CEO of OPTICOM addressed this important topic in a very enlightening Broadband TV conference session. Germany-based OPTICOM develops algorithms for measuring video quality and licenses that technology to test equipment, video analytics and other  OEM partner companies.

Mr. Keyhl said that existing standardized video quality measurements barely suffice when considering OTT streaming video. Fundamentally, all traditional objective testing standards are based on analyzing short video sequences -of only a few seconds in length. The Mean Opinion Scores are quite low (below 5) for OTT video quality measured that way.  Michael said  that “snapshots of 10 second videos are inadequate to assess re-buffering and long term streaming behavior.” Hence, there’s a need for new type(s) of subjective testing methods and procedures.

In an attempt to greatly improve perceptual video testing standard for streaming video services (including ABR), OPTICOM created Perceptual Evaluation of Video Quality- Streaming (PEVQ-S).  It was described as an “advanced framework algorithm for full-reference picture quality analysis in video streaming environments.” The rules (but not implementation) have been standardized by ITU-T as J.247: Objective perceptual multimedia video quality measurement in the presence of a full reference. Related follow on work on video quality measurements is taking place in the Video Quality Experts Group (VQEG) which produces inputs to various ITU Study Groups for recommendations they’re developing.

As opposed to the lightweight “No Reference” video quality testing type, Full Reference testing is more processing intensive, but offers the highest accuracy and is standardized by ITU. It’s based on differential analysis – comparing the degraded video signal with the original source video with access to the reference/studio source video. OPTICOM‘s PEVQ/ITU-­T J.247 is the standard for Full Reference Video Quality Measurement as noted above.

The different types of Adaptive Bit Rate (ABR) video streaming methods are illustrated in the chart below. As you can see, there are many combinations and permutations for video quality measurements.

Note: In Adaptive Bit rate (ABR) video streaming, the transmitted bit rate, resolution and other aspects of each “media segment” varies according to bandwidth and resources available at the client (receiving device). Video quality significantly depends on the client behavior, such as negotiating bit rate with the video server depending on dynamically allocated bandwidth, streaming protocol and re-buffering. The Media Presentation Description (MPD) is used to convey that information from client to server.

OPTICOM’s PEVQ was said to be validated for many different types of video- not just OTT ABR- using subjective testing. The validated video formats were based on ITU-R recommendation BT.500 – originally named “CRT TV Quality Testing (SD)” and ITU-T Recommendation P.910 – “Multimedia (QCIF, CIF, VGA) and IPTV (HD 720/1080) Testing.”

Based on a fundamental requirement analysis to understand adaptive streaming artifacts, the design of a novel test method was described. A four layer OTT quality model was presented with these four layers (top to bottom): Presentation, Transmission, Media Stream, Content.

Michael said an OTT Video Quality Measurement technique needs to have the following characteristics/attributes:

• be related to content quality as a reference;
• accurately score encoding and transcoding artifacts =Media Stream Quality;
• measure and compare the picture quality for different frame sizes and frame rates = Media Stream/Transmission Quality;
• continuously track the different bit rates and evaluate how smooth the video player is able to interact with the video server in a congested network= Transmission Quality;
• take into account the player and endpoint device characteristics as well as the viewing environment = Presentation Quality.

The architecture of OPTICOM’s novel approach to measuring streaming video quality was said to be able to “overcome the limitations of standardized perceptual video metrics with regard to adaptive streaming of longer video sequences, while maintaining maximum backward compatibility (and thus accuracy) with ITU-T J.341/J.247 for short term analysis.”

An end-to-end functional block diagram of streaming video source/destination measurement using PEVQ-­S is shown in the illustration below.  The four OTT quality layers are shown at the bottom of the figure.

Conclusions: 

1. There’s a clear need for a streaming video quality measurement (VQM) technique which accurately evaluates video subscriber/consumer QoE
2. QoE concepts must be completely reformulated and we must reinterpret video quality in the context of multi-screen use scenarios.
3. Currently, there is no standard for subjective and/or objective VQM of ABR video streaming
4. PEVQ-­S is proposed to resolve that problem, based on advancing existing standards, while maintaining maximum backward compatibility and validated accuracy.
5. PEVQ-­S is well suited to evaluate all 4 OTT Quality Layers (from bottom to top): Content, Media Stream, Transmission, and Presentation
6. PEVQ-­S allows for analysis of common ABR protocols and formats, various video codecs at various bit rates. It can analyze video at different frame sized and frames per second.
7. PEVQ-­S is licensed by OPTICON to leading OTT, Middleware, and Test & Measurement vendors.  It will soon be built into many such products. OPTICOM says they have over 100 licensed OEM customers.

OPTICOM’s Demo:

OPTICOM had a demo at the conference where they measured ABR video quality under various simulated reception conditions. We certainly could detect a difference in quality during different time periods of the stream. The quality of each of the video segments were measured and recorded.

We think that such measurements would be especially useful for mobile OTT video streaming where RF reception varies depending on the wireless subscribers location and physical environment.

End Note:

Time and space constraints do not permit me to highlight all the excellent sessions from this two day conference. Such a complete report is possible under a consulting arrangement. Please contact the author using the form below, if interested:

[contact-form to=’[email protected]’ subject=’I am interested in Alan Weissberger%26#039;s consulting services’][contact-field label=’Name’ type=’name’ required=’1’/][contact-field label=’Email’ type=’email’ required=’1’/][contact-field label=’Comment’ type=’textarea’/][/contact-form]