Fox and AWS have laid out many new details about their development work on what the two companies are calling a groundbreaking AI-based replay system. The system was first developed for FIFA World Cup Qatar 2022 and has since been expanded for use during broadcasts of Major League Baseball (MLB), the United States Football League (USFL) and the FIFA Women’s World Cup.
The Recap Feature and Catch Up With Highlights was built with Amazon Media Replay Engine (MRE), the companies said.
In a blog post, Fox and AWS noted that with the digitization and continued advancement of broadcast delivery technology, sports fans have come to expect a dynamic viewing experience. Whether streaming a game live or recording to view on-demand, audiences are increasingly looking for standout elements in the coverage of their favorite teams, from compelling graphics and data insights to timely replays and highlights. When the games in question are some of the most anticipated in sports, the stakes are even higher, the companies explained.
To meet those audience expectations, Fox built and deployed its new AI-powered Catch Up With Highlights feature ahead of its US broadcast coverage of the FIFA World Cup Qatar 2022.
Leveraging Amazon’s open source Media Replay Engine (MRE) framework for automating video clipping and replay generation, alongside Amazon Web Services (AWS) infrastructure and solutions, Catch Up With Highlights is an end-to-end solution that can ingest broadcast streams, identify valuable match moments through AI-trained computer vision, and then encode delivery streams with metadata across user devices. With AWS already the foundation of Fox Corporation media workflows, the development team was able to stand up the complex new feature in about eight weeks, the companies said in a blog post.
“Over the last five years our team has had tremendous success with unlocking the power of data through the building of advanced and intelligent platforms and products,” explained Fox’s vice president of data products and machine learning (ML) strategy, Phil Martin, who led the Catch Up With Highlights development and first deployment. “In addition, we’ve really rallied around the idea of revolutionizing fandom. We believe that sports should be enjoyable and accessible for everyone, which is why we've designed our platform to make it easy to engage with and enjoy. Our Catch Up With Highlights feature is a great example of this philosophy. We understand that the level of play at events like the World Cup is exceptional, and we wanted our feature to reflect that standard. To achieve this, we developed a cutting-edge AI technology to create a highlights engine that can dynamically retell the game's story in real-time, highlighting key moments and events as they unfold. This ensures that our users can stay up-to-date with the action and experience the excitement of the game, no matter where they are. Prior to and through this journey, AWS has been a trusted partner of ours and together we were able to take another step forward in revolutionizing fandom with data all while on the world’s biggest sporting stage.”
Upon launch, the feature ran seamlessly with very little human input, the companies said. Martin and his team built Catch Up With Highlights to continually optimize the highlight recap video as the game progressed, retelling the game story upwards of 20 times per match with event-driven updates. This approach ensured that the viewers would be able to access a comprehensive recap of the game to date, available via the FOX Sports app on connected TVs and mobile devices. Incremental improvements were rolled out based on performance, such as adding a thumbs-up or down; and event enhancements, including shots saved and penalty kicks. Ultimately, 731 recap videos were published autonomously across 61 matches.
The Catch Up With Highlights workflow includes AWS Lambda event-driven serverless compute, and computer vision with Amazon Rekognition. Sports data is housed in an Amazon DynamoDB NoSQL database, then piped into MRE. Live video is quickly delivered to viewers via AWS Elemental MediaConnect, with videos stored in Amazon Simple Storage Service (Amazon S3), and Amazon ElastiCache for caching live clips, the companies said.
“Soccer is often referred to as ‘the beautiful game’ because of the flowing and artistic style of play and objectively it’s a simple sport: score more goals than the opposition,” Martin explained. “However, training models and developing logic around a game that lacks set patterns is anything but simple. To further compound the complexities, it was our first time publicly releasing a platform feature that ingests multiple concurrent streams, truncates clips for inference, contextualizes and optimizes each segment, and transcodes and delivers directly into our CMS in a matter of moments of an event occurring.”
Additionally, the team had limited reference information from the event organizers. Instead, they relied on audio cues and camera angles to help train the computer vision model. Clips were auto-assembled with polished elements, like transitions and a fade to black, to ensure a high-caliber viewing experience for fans. Now, with a proven, scalable approach, Martin looks forward to replicating the Catch Up With Highlights playbook for other sports broadcasts.
Martin concluded, “We are extremely bullish on this space and already expanding to other sports properties. Fans can experience the feature today during broadcasts of Major League Baseball (MLB), the United States Football League (USFL) and the FIFA Women’s World Cup. Catch Up With Highlights is coming to football broadcasts this fall and more personalized content is on the horizon. The possibilities are endless and we’re just getting warmed up.”
