How In-the-Wild Testing Could Have Prevented the Iowa Caucus Setback

The Iowa caucus was expected to answer questions about the viability of 2020 Democratic presidential candidates. However, the process resulted in more questions than answers. Why? Because the mobile app used by the Iowa Democratic Party to calculate and transmit results for the Iowa caucus experienced technical difficulties at the most crucial juncture.

The worst part is that it could have easily been prevented. How? In-the-wild testing performed by real people. We see it constantly in the business world, where the pace of the market pressures companies to release buggy, low-quality software faster than they should in order to meet consumer demands.

How can political parties — or any company releasing an app — avoid similar issues in the future? Here’s what we at Applause, the worldwide leader in real-world testing, think.

The way an app functions in a sterile lab environment is often vastly different from how it functions in real-world scenarios.

Where Iowa’s problems lie

The way an app functions in a sterile lab environment is often vastly different from how it functions in real-world scenarios. There are numerous variables to account for in the real world, such as network connectivity, load volume and device configuration. Failure to account for these variables can result in devastating consequences — as demonstrated by the Iowa caucus failure.

The Iowa caucus app “had not been properly tested at a state-wide scale,” according to the New York Times. In addition, “there were concerns that the app would malfunction in areas with poor connectivity, or because of high bandwidth use, such as when many people tried to use it at the same time.”

In Iowa, the app was recording data correctly, but only sending out partial data, according to NBC — a problem deemed a "coding error" by the Iowa Democratic Party. The app was reportedly rushed through development in just two months, according to the New York Times.

The missing element in Iowa was in-the-wild testing performed at scale. There is no way that the Iowa Democratic Party could replicate the circumstances of the Iowa Caucus on their own in advance, with over 1,700 unique caucuses each reporting their results from different devices and in different environments. Given how quickly the Iowa Democratic Party found the coding error after the caucuses, it’s likely that testing would have immediately identified the error long before any Iowans entered the caucus. This is where Applause could have helped to ensure the app worked as intended in the conditions where caucus precinct captains would actually use the app.

The Iowa caucus app “had not been properly tested at a state-wide scale,” according to the New York Times.

Test in the real world to anticipate problems

It is a monumental challenge for an organization to test its own app extensively within real-world conditions. Applause helps organizations overcome this challenge by leveraging its global community of testers to test the app whenever and wherever you need, at a moment’s notice.

Applause customizes testing teams to have your ideal end users test the app simultaneously in the same, or similar, conditions that it will ultimately be used in. In this case, Applause could’ve set up the Iowa Democratic Party with real-world testers throughout the state of Iowa to ensure the app functions properly under stress and in extreme conditions (e.g., poor WiFi areas, busy community centers, etc.).

Applause testing is all done on demand, enabling organizations to quickly collect feedback and immediately address any found issues before they reach production. Had the Iowa Democratic Party leveraged in-the-wild testing ahead of time, they likely would have identified the errors well before the Feb. 3 caucuses.

Just as importantly, by not testing with real people, the Iowa Democratic Party missed out on the unbiased insight that in-the-wild testing produces, which internal teams often miss in lab simulations.

Ensure you get the device coverage you need

People use apps on a variety of different devices, often on varying operating systems using different WiFi networks. How that app performs under each configuration can vary greatly. Working on their own, organizations rarely have access to enough real device configurations to reach a viable device coverage mark.

Applause enables organizations to test across the spectrum of device configurations with real devices owned by the selected testers. This not only enables you to scale your device coverage instantly, but it eliminates the need to maintain a stocked device lab, reducing overhead costs in the process. With 1,765 Iowa precincts reporting caucus results, the Iowa Democratic Party clearly needed to test the app on thousands of different device, browser and OS combinations. By not doing in-the-wild testing, the party left the app’s viability to chance and became a national story for the wrong reasons.

With Applause, that chance becomes certainty. The upfront investment would’ve saved the Iowa Democratic Party days and weeks of national embarrassment.