RUTHFULLY YOURS

Last week’s New York Times/Siena College polls showing Vice President Kamala Harris leading in three swing states led some Republicans and Democrats to ask the same question: Is this real?

At the center of this question is whether the Times/Siena polls have enough voters who supported Donald J. Trump in 2020. The Trump campaign released a memo arguing that our polls would have actually showed a Trump lead if we had weighted the results properly.

The Trump campaign’s critique focused on something pollsters refer to as “recalled vote.” In the polls of Pennsylvania, Michigan and Wisconsin, respondents recalled backing President Biden over Mr. Trump by six points, 52 percent to 46 percent, even though Mr. Biden actually won these three states by an average of about 1.5 points. The Trump campaign used this data point to say Mr. Trump would have led if the poll had the “right” number of Trump 2020 supporters.

This isn’t an absurd argument. In recent years, many pollsters have embraced recalled vote in exactly the way the Trump campaign describes: as an accurate measure of how people voted in the last election, which can then be used to evaluate the partisan balance of the sample. As an idea, it makes logical sense.

But over the longer run, recalled vote hasn’t usually been very reliable. Oddly enough, this is one of the first things I remember learning when I started getting interested in polling methodology in the fall of 2004. Back then, it was George W. Bush who led the polls, and it was the Democrats trying to prove that the polls were skewed. Among other things, they argued the polls had too many Bush ’00 voters, based on recalled vote.

A CBS News/Times poll at the time, for instance, found Mr. Bush and John Kerry tied, but Bush ’00 voters outnumbered Al Gore ’00 voters by six points. Mr. Gore won the popular vote in 2000; if the poll had the “right” number of Gore supporters, Democrats thought, Mr. Kerry would have led — perhaps by a lot. Plausible, right?

Needless to say, the polls leading up to Mr. Bush’s 2004 re-election were not badly underestimating Mr. Kerry, as the recalled vote implied. This is not an isolated occurrence: Asking how people voted in the last election has been unreliable dating all the way to the earliest polls.

There are at least three factors at play: A surprising number of people don’t seem to remember how they voted; people seem likelier to remember voting for the winner (Biden in 2020; Bush in 2000); and they seem likelier to report voting when they did not.

Despite this history, more pollsters have been using recalled vote in recent years. For some pollsters with lower-quality samples, recall vote is a blunt instrument that can hammer even the worst poll into the ballpark. Indeed, if you take the last year of Times/Siena poll respondents from Manhattan — which Mr. Biden won by 74 points — you can produce an entirely plausible Trump +3 result for a national poll by weighting it to the 2020 result nationwide.

Even serious pollsters have used recalled vote in recent years. Some believe that the measure has become more accurate during our polarized era, which certainly seems plausible. It’s also possible that it’s more accurate for some new survey methodologies, like panels (which attract highly engaged voters) or even just online polls in general.

But the Times/Siena data doesn’t support the idea that recalled vote has become much more useful. This year, about half of respondents without a record of voting in the 2020 election told us they actually did vote. Only about 92 percent of those who did vote will tell us whom they supported — and there’s no way to be sure that the 92 percent are telling the truth, either.

While some pollsters — even good pollsters — say weighting on recalled vote would have made their polls more accurate in prior elections, the same cannot be said here: Times/Siena polls in 2020 and 2022 would have been less accurate if they had been weighted by recalled vote to match the election result.

It’s especially striking that recalled vote wouldn’t have helped Times/Siena polls in 2020, a year when our surveys (along with virtually all other surveys) fairly significantly underestimated support for Mr. Trump. In fact, the Times/Siena polls would have underestimated Mr. Trump by even more if they had been weighted by recalled vote to match the actual results of the 2016 presidential election. That’s because the Times/Siena polls had “too many” Trump ’16 voters, a surprising finding but one consistent with the long history of people who voted for the losing candidate but who often later say they supported someone else.

This cycle, Times/Siena polls yet again show “too many” respondents saying they supported the winner of the last election: Mr. Biden. On average, Times/Siena respondents have recalled voting for Mr. Biden by about three points more than the actual results of the last election.

Is it possible that Times/Siena polls have been underestimating Mr. Trump all along? Absolutely. They certainly did in 2020, after all. Nonetheless, our polls so far this cycle routinely provided very positive results for Mr. Trump, even when there seemed to be “too many” Biden ’20 voters. Arguably our best result for Mr. Trump this cycle — a February poll showing him ahead by six points among registered voters — also found Mr. Biden with a 12-point edge on recalled vote, compared with his actual 4.5-point victory.

Despite this history, there’s still no guarantee that the Times/Siena data is correct. Even an unequivocally bad methodological decision — say, calling only landlines — can still occasionally yield a more “accurate” result if it cancels out a different kind of bias in another direction (say, Mr. Trump’s supporters being less likely to respond to a poll). Weighting on recalled vote is clearly far more debatable, but even if it is every bit as problematic it could still yield accurate results this cycle for the same reason. No one ever knows which polls — or which polling methodologies — will appear “right” or “wrong” until the election results begin to arrive.

The post The Polls Have Shifted Toward Harris. Is It Real, or Something Else? appeared first on New York Times.