Testing Lessons from the COVID-19 Vaccine Rollout
When Massachusetts opened up COVID-19 vaccines to a broad group of residents last Thursday, people who met the new requirements quickly went online to schedule their shots.
This spike in traffic caused the vaccine appointment portal to crash. So did the state’s health and human services telephone hotline.
Residents called the crash “a nightmare,” “frustrating” and “unforgivable.” Other government agencies suffered similar failures with their online vaccine portals, including the state of Minnesota and Florida’s Pinellas County. Once the Massachusetts vaccination site recovered, many users found the lengthy — and often redundant — data entry process painful.
What went wrong?
Stuart Madnick, director of the cybersecurity center at the MIT Sloan School of Management, told The Boston Globe that the state system was most likely cobbled together too quickly to permit adequate testing of software and servers. “A new system is likely to have flaws in it,” Madnick said. “When you try to rush things out, that’s one of the likely outcomes.”
Leaders from the company managing the Massachusetts COVID-19 vaccination portal said the site did not scale adequately to handle such a dramatic surge in traffic. Nearly one million state residents met the new vaccine requirements; only 60,000 appointments were available through the portal. Neither the website nor the call center was prepared.
The right tests could have prevented mass frustration
Frustrated vaccine-seekers said that much of the aggravation could have been avoided. They’re right: load, stress, and soak testing would have helped developers better understand how the site would perform in real-world scenarios. Rigorous QA processes include non-functional tests to identify a multitude of failure points, such as inability to withstand high volume, errors on certain devices or networks, bugs and faulty code.
Simple tests could have revealed how both the vaccine portal and hotline would respond to heavy traffic. With a clear understanding of limitations, extra servers or load balancing could have let the portal’s operators scale server capacity to meet spikes in demand. Companies rely on Applause to assess readiness for high volume in a range of scenarios, such as concert ticket sales, financial market fluctuations and live pay-per-view sporting events. All too often, downtime means lost revenue. Investing in testing prior to launch reduces the risk of lost sales.
Simple tests could have revealed how both the vaccine portal and hotline would respond to heavy traffic.
User experience testing would have uncovered opportunities for improvement as well. One of the biggest complaints about the Massachusetts vaccine portal was that users were required to enter the same information every time they tried to book an appointment at a different time or location. Simple usability or UX tests would have revealed the opportunity to streamline the appointment-scheduling process by having users save profiles with the relevant information, rather than starting from scratch each time.
What companies should learn from the vaccine portal crashes
You can't rush a product to the finish line and expect it to work when thousands of people access it simultaneously — unless you’ve thoroughly tested. Whether you're a government agency or a highly recognizable brand, you must deliver exceptional digital experiences — or risk loss of revenue and reputation.
To prevent bugs, crashes and angry customers, organizations can adopt a holistic testing strategy. A holistic approach assesses different dimensions of digital quality, such as functionality, usability, accessibility and more — all to ensure the best possible user experience.
In a recent blog post, Applause’s John Kotzian explained how the right testing environment can help organizations avoid costly errors. Many companies can’t sufficiently replicate real-world scenarios in house or a lab, so they turn to a testing partner they can trust to make sure they’re executing a holistic strategy. Outsourcing certain types of tests allows internal teams to focus on additional logistical challenges, such as preparing the call center to better handle a surge.
Taking time up front to make sure things work smoothly pays off, saving costs and building customer confidence. Make sure you have a healthy testing plan in place. Learn to develop a holistic testing strategy from this ebook.
