Even for those with the fastest internet connections, the problem of occasional glitches and stalls in streaming media is very real. Too often, those annoying burps and stutters can destroy the experience of the programming we watch—especially music.
In the earliest days of streaming media, such artifacts were an everyday occurrence. But today the issue has gained far more importance since video and audio streaming are now the predominant internet application—making up three quarters of all traffic. If streaming media is to truly replace broadcast, cable and satellite distribution, its image and sound quality needs to become far more trouble-free and reliable.
Researchers have been studying the problem using machine learning to improve video streaming algorithms. They are trying to figure out how to teach a computer to design new algorithms to reduce these nasty artifacts. This is complex stuff since the internet is not a monolithic entity. It is an ever-changing web of interconnections ranging from fast fiber-optic delivery to slower wireless connections in rural areas.
ALGORITHM-BASED
One effort to improve streaming media quality is Puffer, a research study at Stanford University launched in 2019 that is using machine learning to not only improve picture quality, but to predict how the capacity of an internet connection changes over time.
Puffer research is focused on three types of algorithms. The first are congestion-control algorithms that decide when to send each piece of data—also known as a packet—through the network. The second is throughput forecasters, which predict how long it will take to send a certain amount of data over an internet connection in the near future. And the third are adaptive-bitrate (ABR) algorithms that decide what quality of video to send to try to give the user the best picture quality that won’t lead to a stall or rebuffer.
To understand the challenges of video streaming and measure the behavior of ABR schemes, the Stanford team built Puffer, a free, publicly accessible website (https://puffer.stanford.edu/) that live streams over-the-air channels from NBC, CBS, ABC, PBS, CW and Fox. Puffer operates as a randomized controlled trial; sessions are randomly assigned to one of a set of ABR or congestion-control schemes.
Study participants include anyone in the public who wishes to sign up. Users are blinded to algorithm assignment, and the researchers record client telemetry on video quality and playback. And, in case you are wondering, a Stanford Institutional Review Board determined that Puffer does not constitute human subject research.
Puffer works well in the Chrome, Firefox, Edge and Opera browsers, on a computer or an Android phone or tablet. However, it does not work on Apple’s iPhones or iPads or in the Safari browser. This is because Puffer uses the Web Media Source Extensions (MSE) to stream video. This standard is not supported by all browsers, and, in particular, not by Apple.
The researcher’s reasoning for streaming live television is to collect data from enough participants and network paths to draw conclusions about the performance of algorithms for ABR control and network prediction. Live television is an evergreen source of popular content that had not been broadly available for free on the internet. The study benefits, in part, from a law that allows nonprofit organizations to retransmit over-the-air television signals without charge.
The Stanford team wants to create a learned ABR algorithm that performs well over the wild internet. Their best answer is to train the algorithm on data from the real deployment environment and use an algorithm whose structure is sophisticated enough, yet also simple enough, to benefit from that kind of training.
So far, Puffer has streamed more than 38.6 years of video to 63,508 users across the internet. Sessions are randomized in blinded fashion among algorithms, and client telemetry is recorded for analysis. The team has found the Fugu algorithm has out-performed other schemes, both simple and sophisticated, on objective measures (SSIM, stall time, SSIM variability) and increased the duration that users chose to continue streaming.
THE LAST FRONTIER?
Puffer was launched in 2019 by Francis Y. Yan, then a Stanford doctoral student in computer science, and a group of other Stanford researchers. Dr. Yan is now a senior researcher in the Mobility and Networking Research Group at Microsoft.
Yan said the Puffer study has found that the confidence intervals on the performance of AVR algorithms are much bigger than the team first realized. He said a simple buffer-based AVR control published more than six years ago, performs surprisingly well. And, he found, it is possible to learn an AVR algorithm to outperform existing ones. But the learning has to be in situ, meaning in place on the actual deployment environment.
Puffer plans to operate as long as feasible and invites researchers to train and validate new algorithms for ABR control, network and throughput prediction, and congestion control on its traffic. The study on AVR algorithms is seeking to answer the question: What does it take to learn an AVR algorithm that performs consistently well over the real internet?
In an important way, the Stanford and other studies focused on removing annoying artifacts from internet streaming are the last frontier to making distribution over the internet as solid and reliable as more mature broadcast technologies. l
