We all know the modern “first world problems” of our time: You want to watch a film on a streaming service, but it takes a lot of time for the video to load – or it is played in poor quality. Sometimes there are interruptions, so-called buffering. In the worst case, the video stops completely and no longer loads, leaving you frustrated.

This is what experts refer to as the user experience (UX), and preferably, the experience should be a good one. But what does that actually mean, and what does it depend on?

“The term UX summarises several factors: the perceived video quality, the time between request and start of the video, buffering, or even unexpected stops during streaming,” explains video coding expert Prof. Dr. Matthias Narroschke from RheinMain University of Applied Sciences in Wiesbaden. All these factors together can lead to a better or worse user experience.

In the BMBF project “End-to-end optimization of the user experience for OTT video services (VidUX)”, Narroschke and his colleague Prof. Dr. Wolfgang Ruppel want to improve the user experience for video playback. To this end, they have partnered up with the streaming provider Joyn GmbH who can immediately test the scientific findings “in the field”, i.e. on its users, and directly implement the results into its services.

The goal of the project is to develop an algorithm that estimates the subjective user experience as accurately as possible – and then calculates the best possible playback quality under the existing conditions. One such condition is, for instance, the available data rate on the respective end device. If the device cannot provide a sufficient data rate, the system automatically switches to a lower quality level. Currently, videos are played in a few predefined data rate levels depending on the internet connection, regardless of their content or complexity. If the available data rate on an end device is too low, the video is either stopped or the optical quality has to be adjusted manually by the user.

Research is focusing on the user experience

This is where the VidUX project comes in. Via the so-called VMAF value (video multi-method assessment fusion), the subjective quality of a video can be represented on a scale from 0 to 100. The VidUX team has determined which VMAF values are required for an optimal user experience. “A VMAF value of at least 95 corresponds to the original video quality when played back on a 4K TV. For playback on a current smartphone, a lower VMAF value of about 92 is required for original quality," says project manager Narroschke.

The team carried out trial encodings and standardised subjective tests in the lab that were able to represent an optimal UX taking into account various parameters. For instance, the researchers were able to determine under which circumstances the best playback quality can be achieved and at what point users perceive noticeable quality gradations in the playback of a video. Research in VidUX thus focuses on the subjective user experience – in combination with minimised data rates and maximum quality. “In our experiments, we also found out that the UX for videos with a VMAF value below 50 is so low that it no longer makes sense to play this video, as the quality is unacceptable for most users”, says Prof. Narroschke.

Minimising the data rate to maximise quality

By minimising the data rate with maximum possible quality, the costs for storage and transmission are also minimised, thus increasing economic efficiency of streaming services. In addition, the likelihood of buffering and stuttering videos is reduced. Ideally, users should hardly notice any qualitative differences in playback: The nuances in video quality are so tiny that switching from one quality level to another is almost imperceptible.

Put into practise, all this means that a user starts a video, and the system analyses the necessary data rate and requests the video in a quality that is always just below the available data rate. This ensures that the video produces the best possible UX.

However, Prof. Ruppel says it is extremely difficult to produce the desired quality in encoding. “Each video has different complexities, so we need a powerful algorithm that can analyse these complexities and suggest suitable encoding parameters.” Prof. Narroschke adds: “In this project, we have developed such an algorithm based on artificial intelligence and neural networks. Our students, whom we prepare intensively for these modern techniques in our courses, also contributed to the development.”

Standardisation of the procedure as a long-term goal

VidUX was originally scheduled to run until the end of 2022, but as is so often the case in research, unpredictable developments have affected the project. In particular, the large-scale subjective assessments in the lab could not take place as planned due to the pandemic. Thanks to an extension of six months, however, the VidUX team is optimistic that it will be able to complete the work successfully.

“Quality-oriented encoding for video streaming is a trending research topic worldwide, and we were one of the first institutions to publish on it”, says Prof. Narroschke. The streaming provider Joyn can evaluate the success of the approach developed in the VidUX project via user feedback and thus provide important insights to further improve the algorithm for quality-oriented video streaming in.

So, what is the next step envisaged by the project team? The main goal of the project is, of course, to have the technology implemented across the board by the project partner Joyn. However, in the long run it might also be adopted by other streaming providers.