Fifteen Eighty Four

Polls are already big news – and they’ll only get bigger as we doom scroll our way through another appalling election cycle.  Is Trump really up in Michigan? Is Biden really hemorrhaging support among young people?

For all the attention we pay to polls, it is crazy how little we actually know about how they work.  Back in the day we knew a lot: if you get answers from a random sample of the population, you’ll get unbiased estimates that get more accurate as the sample size goes up.  But now?  The New York Times may contact a random sample, but only 1 in 100 answer.  And the New York Times is one of the best in the business.  There are hordes of low-budget pollsters who troll for answers with pop-up ads on the internet, add a bit of weighting fairy dust and bingo: another headline.

So even as polling lives on it is time to face the fact that random sampling is dead. When 1 in 100 people respond, that’s not a random sample.  It just isn’t. Recognizing this has tons of implications.  The biggest is that the tiny response rates open up huge the door for bias to sneak in.  Suppose that the people who like Biden are just a bit more likely to respond, even after account for demographics.  Suppose you’re polling a crucial battleground state like Wisconsin in 2020 and for whatever reason, the Biden supporters among white working-class voters were more excited about answering (or more likely to home during covid).  Just a little bit of that and you could end up with a poll that showed Biden up by 17% in Wisconsin in late October 2020.  Biden won in Wisconsin by less than 1%.

Oops.

There is so much to unpack about polling today – and it’s all good fun.  I wrote a whole book: Polling at a Crossroads: Rethinking Modern Survey Research  on the topic.  I show how pollsters lock in on a way to do things… until they screw up a presidential election as they did in 1936 and 1948.  Was 2016 another inflection point?  It’s complicated because the polls were fine in some ways and terrible in others in 2016. After that polls predicted elections well in 2018, were terrible in 2020 and then got better in 2022. I explain where the field is and how it continues to hurtle away from the random sampling paradigm we relied on not too long ago.

Most importantly, though, is that I want to figure out ways to improve polls.  There are a ton of ideas out there – some of them developed in other fields and many of them quite intuitive.  In the second half of my book I go through them.  I show for example that a theory-based new approach to survey design and analysis showed strong signs that Trump was underperforming in polls relative to actual support in the Midwest before the 2020 election.

What do you think?

• Do you trust polls?
• Is random sampling really dead?
• Can we do anything to make polls better?
• Are we going to be shocked (again!) the day after the election?