What went wrong with the Iowa app

The chaos that upended the Democrats’ Iowa caucus bears all the hallmarks of a collision between poorly vetted computer code, inadequately trained volunteers and party officials who waved off warnings from outside experts.

It most likely wasn’t the work of hackers, homeland security and cybersecurity officials said Tuesday — though some cautioned that it's too soon to say even that much for certain. But security researchers expressed bafflement at the flubs and miscues in deploying the app, calling it a cautionary tale in how not to launch technology that’s critical to Americans’ faith in democracy. And they said they hope the lesson sticks.

“Experts have been unanimous in warning of the dangers of Internet and mobile apps in voting,” said Matt Blaze, a professor of computer science and law at Georgetown University and a leading voting security expert. “While the risks and threats in a caucus are somewhat different from those in a general election, what happened in Iowa is a perfect example of why this technology is so perilous.”

Here’s what we know about the app, which left multiple candidates declaring victory while President Donald Trump’s supporters chuckled at the Democrats’ dysfunction.

Where did the app come from?
The Iowa Democrats commissioned it to help volunteers send the results of Monday night’s low-tech caucusing — a series of in-person gatherings around the state — to a central office so they could be posted online. The app was also supposed to help precinct chairs do the math to help figure out which candidates met the caucus’ viability thresholds, according to a complicated tallying formula that the Democrats were also debuting Monday.

The software was the handiwork of Shadow Inc., a maker of digital campaign tools for Democrats that received more than $63,000 from the Iowa Democratic Party in November and December. On Tuesday, the company expressed "regret" about the confusion without disclosing what went wrong.

"Importantly, this issue did not affect the underlying caucus results data," the company wrote on Twitter. "We worked as quickly as possible overnight to resolve this issue, and the [Iowa Democratic Party] has worked diligently to verify results."

Shadow also received $58,000 last year from the Nevada Democratic Party — which announced Tuesday that “we will not be employing the same app or vendor used in the Iowa caucus” — and had done text messaging, software and other work for clients including Joe Biden’s and Kirsten Gillibrand’s (D-N.Y.) presidential campaigns, according to federal election records.

Shadow has ties to a nonprofit group, Acronym, that has expressed ambitions about boosting the Democratic Party’s digital innovation and countering President Donald Trump’s dominance of online advertising and free social media. Acronym’s leadership includes Tara McGowan, a veteran of the digital advertising operation of the Democratic super PAC Priorities USA Action, and David Plouffe, a former Obama campaign adviser who has joined its board of directors in September.

Acronym declined to comment Tuesday.

Didn’t anyone vet the app ahead of time?
The state party has said it consulted with cyber experts at the Department of Homeland Security, Harvard University and elsewhere about how to safely report the results, according to a party official who had insisted on being anonymous. “We are confident in the security systems we have in place,” Iowa Democratic Chairman Troy Price told POLITICO last week.

But a DHS official told POLITICO on Tuesday that it had not vetted or tested the app. And acting Homeland Security Secretary Chad Wolf said on Fox News that his department’s Cybersecurity and Infrastructure Security Agency offered to test the app "from a hacking perspective," but "they declined.”

Is this the first time the parties used an app?
No — in 2016, both the Democrats and the GOP used an app devised by Microsoft to record the results in Iowa. And the Republican Party used an app from a different company — which it would not identify — on Monday night, with no apparent distress.

Mitt Romney’s (R-Utah) 2012 White House campaign had its own brush with disaster when an app called Orca, which it used for turning out voters, broke down on Election Day in November, his former chief digital strategist Zac Moffatt recalled Tuesday.

Moffatt, who was in Iowa on Monday as a caucus observer, had this take on what went wrong this time: “I’m assuming there was a combination of bad tech, bad adoption, bad education, and then people on social media showing [the app’s] vulnerabilities.”

So what happened on Monday?
The Iowa Democratic Party has refused to release any details, though on Tuesday it blamed the problems on an unspecified “coding error.” But the trouble at the caucus started early on Monday, as POLITICO reported at the time, when volunteers said they were “struggling” to download or use the app just a few hours before the voting started.

Many precinct volunteers tried to use the party’s telephone hotline to call in the results instead — only to face hours of waiting on hold.

Those reports stunned the cyber experts, who said they bore all the signs of a rush to deploy inadequately tested software without fully training the users.

“The modest sum of money paid to the app provider over a couple months, the absence of any meaningful testing and the poor system performance on caucus night shows that ‘you get what you pay for,’” said Eddie Perez, global director for technology development at the OSET Institute, which advocates for open-source election systems.

“As a friend wrote, Macy's doesn't [roll] out a new cash register the day before Black Friday — you test it out in a limited market and figure out the problems without the pressure,” said Jeremy Epstein, a voting security expert who is the vice chairman of the Association for Computing Machinery's U.S. Technology Policy Committee. “Writing reliable software is very hard, and I'd say this is a case in point.”

Duncan Buell, a computer science professor at the University of South Carolina, likened Monday’s caucus to previous elections in which election-related software in counties in Kansas and Indiana couldn’t handle the traffic on Election Day.

“One ought not be testing for the first time the ability of the system to handle a full load in a live event,” Buell said. “That's just dumb.”

Is anyone saying, ‘I told you so'?
Plenty of people could — Buell and other cybersecurity researchers had raised alarms in the weeks before Monday about the party’s plans to use an app installed on volunteers’ personal smartphones for such a critical task, noting that phones have often proven easy prey for hackers. They also criticized both the Democrats’ and Republicans’ refusal to identify the maker of their apps or describe how they work — saying the parties were making a classic mistake in supposing that “security through obscurity” could keep a determined attacker from exploiting holes in the software.

The Iowa Democratic Party said in a statement Tuesday that “in preparation for the caucuses, our systems were tested by independent cybersecurity consultants." But that didn’t satisfy any of the doubts.

“Do they name who did the testing?” asked Dan Wallach, a computer science professor at Rice University. “Can we read their report? Has anybody from the testing been quoted by the press, even? If not, it doesn't mean much.”

Does anybody think hackers are to blame?
None of the experts said they had seen any evidence that a cyberattack was to blame for the failures of the app or the telephone hotline. Neither did DHS, which said in a statement that "we have no reporting of any malicious cyber activity."

Then again, some experts said it’s impossible to know for sure.

At the moment, “there is no way to distinguish between an inadvertent error and a malicious attack,” said one cyber researcher, who spoke on condition of anonymity in order to offer a candid opinion. “Even if we can identify a specific coding error that can explain everything that happened, we can never be certain that the error wasn’t introduced with malicious intent. This probably was just incompetence, but there’s no way to be certain.”

And regardless of whether any hacking occurred, Monday’s turmoil fuels the kinds of conspiracy theories that undermine people’s faith in the electoral process — as seen in tweets from people like Trump’s son Eric, who wrote shortly before midnight: “Mark my words, they are rigging this thing.”

“Even if the app wasn’t hacked, without sufficient transparency about the app and its developer, darkness is a ripe environment for disinformation campaigns,” said Perez, from the OSET Institute. “When the public doesn’t have solid information about the tools that elections depend on, it’s that much easier to raise questions about the integrity of the democratic process. Trust is built on the combination of communication + transparency — and the state party fell short in both of those areas.”

Virginia Sen. Mark Warner, the top Democrat on the Senate Intelligence Committee, echoed that warning in a statement. “As we’ve seen in the past, foreign actors like Russia and China won’t hesitate to latch onto this kind of content in order to add to the domestic discord and distrust in our elections,” he said.

Are there any positive lessons to draw from this debacle?
Several of the experts were optimistic that Monday could put the kibosh on any stabs at allowing people to cast actual votes through apps or online. They noted that at least in Iowa, the in-person caucus process should leave a paper record of how people voted — a safeguard that would be lacking in an online vote in a traditional primary.

Indeed, the Iowa Democrats had initially intended to use a telephone-based “virtual caucus” on Monday to people to vote remotely, in the name of increasing participation. But it scrapped those plans last year amid warnings about security risks.

“Despite the delay, we're all going to learn what the correct results are, because the results from individual sites are on paper and will eventually be reported and cross checked,” said J. Alex Halderman, a computer science professor at the University of Michigan who has tested many voting systems. “It would have been far, far worse if this was an app-based voting system — then we might never know for sure what the results were.”

Tim Starks, Martin Matishak and Zach Montellaro contributed to this report.