The US Election 2020 and the dangers of “peeking” too early on experiment results
It’s 08:41 a.m. on 6 November 2020 and I have a browser tab open to a map of the US showing the 2020 election results.
Right now, I’m thinking about the democratisation of experiment processes across an entire company. Part of the principles of democratisation is to provide everyone with the ability to view real-time experiment results.
Getting back to the US Election for a moment: I’m one of the millions of people around the world who are monitoring the results of the election with tense anticipation. Each new update brings us the latest number of votes counted per candidate, along with what percentage each candidate has, as well as what is left to count.
It’s a unique situation. The outcome becomes more (or sometimes less) apparent with each update. And these updates are happening over the course of days, rather than hours.
For me, this is a fascinating experiment in the democratising data and results— it’s a slow reveal of the ultimate fate of the world (am I being overly dramatic? I don’t think so).
The reason I find this so fascinating (and terrifying) is because I’m very aware of the dangers involved when it comes to reading the results of anything when it is slowly drip-fed over a period of time. It reminds me of monitoring A/B experiments — more specifically, it reminds me of monitoring the results of experiments where there are high stakes involved.
When test results are “peeked” at too early (particularly by those without a good understanding of statistics) it can lead to huge misunderstandings. This is because early views can present a radically different picture from the final result. When this happens, it can lead to a loss of faith in the experimentation process as a whole.
As of right now, Donald Trump, the Republican party, and many Americans claim their voting system is rigged because (in part) they can’t understand how a radical lead they first enjoyed is now being eroded away. They point to this as evidence of some sort of wrongdoing.
Yes, with the 2020 US Election Results, there is a clear reason why the results showed a “flip” — Republicans encouraged voters to vote on the day, while Democrats encouraged voting by mail. Mail ballots were counted later, therefore providing a false early view — hence the so-called “red mirage”.
Putting that explanation aside, however, it’s still important to remember that other effects are also at play. There is often a lot of “noise” when viewing results early in realtime because of statistical effects like regression to the mean. You see this all the time when you monitor the results of an A/B experiments. The results could favour one variation group at one point and then flip to favour the other group as more data is gathered.
Add to this the stakes. If they are high (whether it is an election or an important experiment), people get invested in any early results they see. After all, they saw their experiment win for a moment. Once seen, this can no longer be unseen and it colours everything they see from that point onwards.
So, what does this tell us about democratising experiment dashboards? Am I saying it’s a bad idea? Well, no.
Sure, there are dangers in allowing people to peek at ongoing test results, but well-defined processes can fix this — no, statistical significance isn’t enough. And while rules, processes and governance can hopefully avoid early peeking problems, a greater understanding of statistics can help foster faith in the process.
Now, don’t get me wrong. I’m not saying that everyone should become trained statisticians. Instead, I think everyone should have some basic understanding about why statistics is necessary and why early test reads can be misleading.
I’m actually in the process of writing an article, explaining these basics so I’ll share that soon. For now, you’ll have to excuse me as I’ve just noticed that Biden has edged ahead in Georgia…
I’m Iqbal Ali. Former Head of Optimisation at Trainline. Now an Optimisation Consultant, helping companies achieve success with their experimentation programs. I’m also a graphic novelist in my spare time. Here’s my LinkedIn in case you wanted to connect.
