Live-Event Testing: Protect Your Investment
The initial thrill of subscribing to a streaming media platform centered around the ability to access on-demand programming, anywhere at any time. And while that’s still how most subscribers access streaming content, the demand for live programming has increased in recent years, especially as more cord-cutters become subscribers.
From sports to music concerts and even reality TV specials, subscribers are demanding live experiences. In fact, PwC expects this demand to help push the streaming market past $60 billion in revenue over the next few years, with a CAGR of 7.2% through 2027.
Here’s the problem: Consumers carry their expectations of on-demand programming — and digital experiences as a whole — to live programming. They expect seamless streaming experiences, despite the fact that live event broadcasts come with unique challenges. For media and entertainment companies investing billions into streaming live events, the stakes are incredibly high. Let’s discuss what it takes to deliver high-quality live streaming events.
Big bucks, big expectations
Investing in live event streaming is a significant commitment. Consider the massive streaming rights deals taking place for sporting events — the dollar figures are astronomical. ESPN’s streaming deal with the NBA is expected to top $2 billion annually, and Netflix’s recent agreement with the NFL will see it pay hundreds of millions of dollars for just two games a year.
However, the initial investment that streamers make in rights deals often dwarfs, by many orders of magnitude, the continuing investment in app development and delivery. In some cases, entire live streams are left to a small team or even a singular person who must ensure high quality. As a result, some media companies face struggles with in-market coverage and device compatibility during their live event broadcasts. Even when these organizations are able to staff employees in key locations, covering all necessary geos and device combinations, particularly for events happening on nights and weekends, remains a challenge.
Subscribers will come to a platform for their content offerings, but are just as likely to churn when the experience is poor. Antenna’s State of Subscriptions 2024 report offers some grim news for streamers fighting tooth-and-nail for profitability:
- churn has nearly tripled in four years (pg 4);
- total cancellations equaled more than $140 million in 2023 (pg 4);
- serial churners, who cancel services more than twice in a two-year span, are on the rise (pg 15);
- yet non-serial churners still make up the majority of cancellations (pg 18).
Attempting to retain subscribers can sometimes feel like trying to plug holes in a dam with a half-used roll of painter’s tape. But success means strengthening the dam itself, giving subscribers a reason to stick around. Great content helps, but programming is dynamic and changes week to week. Outstanding subscriber experiences, however, can serve as a brand differentiator.
Webinars
Delivering Value from High-Stakes Live Streaming Events
Join industry experts Ken Isaacson and Ross Curley in this webinar to learn how to take a comprehensive approach to digital quality for live-streaming programming.
Limitations across all organizations
Success and profitability lie in robust testing and monitoring, which help guarantee return on that significant upfront investment. In-market coverage and compatibility with various devices and operating systems are critical, but often challenging to achieve. We typically see media organizations take one of two approaches, both of which leave blind spots.
Immature organizations:
Less-mature organizations often try to bandage digital quality challenges with low-investment resources. For example, many of these organizations will leverage existing QA resources, which causes burnout among internal staff who must now test during off hours, or hire part-time employees for major events, who might lack the skill to perform the task.
These organizations, which often have a small QA footprint, really have no means of monitoring and testing live events on their own. And when you consider that the battle for streaming media occurs over a global marketplace, those challenges are magnified many times over, both in terms of scale and localization.
Mature organizations:
Mature organizations often have dedicated digital network operations centers that monitor live events, such as sports and news, around the clock. These individuals are often skilled in operations or digital quality disciplines, knowing how to identify and remediate issues when they pop up.
However, even with this investment in infrastructure and personnel, there are constraints to the geographic reach. For example, one of our clients, a major media company, wanted to pilot live-event testing in Africa. Naturally, this provided a number of localization challenges on a continent where more than 2,000 languages are spoken. Even supporting just three languages was a challenge for the company due to lack of in-market resources. On top of that, they experienced challenges around playback control, closed captions and more, necessitating broader help to ensure live streams.
Common challenges in live-event testing
Functional, usability and payment defects can occur at many junctures of a streaming customer’s journey. The best defense against those challenges is with a thorough testing approach, but the practical limitations mentioned above typically manifest in four key areas.
High turnover and low skill sets:
Organizations often hire part-time employees to monitor live streams. These individuals might be aspiring network operations professionals, or they might simply be high school graduates looking for gig work. These employees typically have low skill sets compared to professionals with a digital quality or operations background, requiring extensive training to bring them up to speed. High turnover and complicated or inconsistent night/weekend schedules add to the challenge, making it difficult to maintain a skilled testing team over time.
Device/OS sprawl:
The media landscape is uniquely fragmented with its myriad of laptops, tablets, phones, smart TVs and OTT devices and their respective operating systems. Ensuring compatibility and quality across all these platforms is a daunting task, especially when you consider that some devices might only be available in certain markets. Device labs can’t keep up with this ever-growing assortment of devices.
Localization:
Language barriers, region-specific content and regulatory guidance all vary wildly market to market. An approach that might work like a charm in one market could fail remarkably in another. We constantly see media companies struggle to manage the complexity of delivering content and ads to diverse linguistic audiences — and that’s just the tip of the iceberg for localization challenges, as cultural insensitivities and many other flaws can damage the brand’s in-market reputation.
Playback issues:
The sooner you can find an issue, the sooner you can remediate it. But, during live streams, many organizations simply don’t have the mechanisms in place to collect data on playback issues, buffering, incorrect metadata or audio sync problems. These defects severely impact the viewer experience.
A multi-faceted approach to live-event testing
Large and small organizations can follow some proven practices to improve the quality of their live streaming events and ads. However, it takes a comprehensive, multi-stage approach to ensure high-quality subscriber experiences.
Pre-event testing:
Conducting rounds of testing before major events is crucial. For example, one major sports league invests heavily in pre-season testing, deploying dozens of testers to ensure the system scales properly during live events and minimizes latency. By running these systems through their paces early, they can avoid friction when the season begins, which is always a popular time to stream.
Pre-event testing offers a low-stakes but realistic testing environment. Facilitate pre-event testing through crowdtesting to identify and resolve issues before they become revenue-bleeding defects. Deploy crowdtesters that mimic real user personas in real locations to help ensure that every potential functional or localization issue is uncovered in real-world environments.
Real-time monitoring:
Ensuring a seamless live event experience requires real-time feedback during the event itself. No amount of upfront prep work can mitigate the need for this critical look into the live subscriber experience. If a defect reaches the point of attracting public complaints on social media, that has a real effect on a brand’s reputation when not promptly addressed.
Crowdtesting provides a structured system for real-time issue reporting and centralized management. Testers from different geos provide immediate and actionable insights, providing documentation of issues for quick troubleshooting and resolution.
Ad validation and insertion:
Subscribers must be happy, and so must advertisers, who also have many options to exert in a crowded marketplace. Ensuring ads display correctly in the right markets at the right times is vital for maintaining advertiser trust and revenue. Media companies often face challenges with ad insertion, especially during live events. These ad experiences, which might involve different systems coordinating with each other, require manual human validation — who knows if the coach will call a timeout 10 minutes or 15 minutes into the game?
Through crowdtesting, organizations can validate ad delivery and insertion points to help ensure that ads are shown accurately and in the appropriate regions. This not only helps streamers meet advertiser agreements, but it also enhances the overall viewer experience by both preventing disruptions and establishing local relevance with recognized regional brands and advertising practices.
E-Books
Content and Ad Validation in Streaming Media
Learn why delivering relevant, localized, and interactive content and ads is a necessity in retaining viewer attention — and subscriber dollars.
Leveraging the global crowd:
Learn what many of our media industry clients already have: the crowd helps unlock the challenges of live-event testing. Days, nights, weekdays, weekends, Applause’s million-plus global community of experts is always available to ensure the quality of your live streams.
Applause ensures comprehensive coverage across various devices, operating systems, geographic locations and test cases. This extensive reach allows for in-market testing in real-world scenarios, uncovering edge cases and defects that might otherwise be missed, including during live events, when seemingly anything can go wrong.
With rapid feedback from our digital community, streamers can remediate issues quickly and get back to providing an exceptional subscriber experience. Moreover, Applause offers access to specific knowledge and skill sets that might be outside of an internal team’s expertise, providing clear guidance on how to reproduce defects and verify that issues have been remediated.
Scalability is another significant advantage of our crowdtesting solutions. Applause can scale testing efforts up or down as needed, which is particularly useful during peak times or for large-scale events. This flexibility not only ensures thorough testing but also proves to be cost-efficient by avoiding idle resources and wasted time hiring and training part-time employees. Media companies trust Applause for localized in-market testing during major events, managing device/OS compatibility, localization issues and real-time monitoring effectively. This approach helps maintain high-quality standards and protect brand reputation, even when facing the complexities of live event broadcasting.
Learn today how Applause can help ensure successful streaming events, now and in the future.