Will The Polls Overestimate Democrats Again? – FiveThirtyEight

ILLUSTRATION BY EMILY SCHERER

As Democrats prospects for the midterms have improved theyre now up to a 71 percent chance of keeping the Senate and a 29 percent chance of retaining the House, according to the 2022 FiveThirtyEight midterm election forecast Ive observed a corresponding increase in concern among liberals that the polls might overestimate Democrats position again, as they did in 2016 and 2020. Even among commenters who are analyzing the race from an arms-length distance, there sometimes seems to be a presumption that the polls will be biased toward Democrats.

The best version of this argument comes from Other Nate (Nate Cohn, of The New York Times). He pointed out in a piece on Monday that states such as Wisconsin and Ohio where Democratic Senate candidates are outperforming FiveThirtyEights fundamentals index like how the state has voted in other recent elections were also prone to significant polling errors in 2020. Cohns analysis is worth reading in full.

Here, Im going to present something of a rebuttal. Not necessarily to Cohns specific claims, but rather to the presumption I often see in discussion about polling that polling bias is predictable and necessarily favors Democrats. My contention is that while the polls could have another bad year, its hard to know right now whether that bias will benefit Democrats or Republicans. Peoples guesses about this are often wrong. In 2014, for example, there was a lot of discussion about whether the polls would have a pro-Republican bias, as they did in 2012. But they turned out to have a pro-Democratic bias instead.

Theres one important complication to this, however. Our model actually assumes that current polling probably does overstate the case for Democrats. Its just not necessarily for the reasons people assume.

As I mentioned, the Deluxe version of our forecast gives Democrats a 71 percent and 29 percent chance of keeping the Senate and House, respectively. But the Deluxe forecast isnt just based on polls: It incorporates the fundamentals I mentioned earlier, along with expert ratings about these races. Furthermore, it accounts for the historical tendency of the presidents party to perform poorly at the midterms, President Bidens mediocre (although improving) approval rating and the fact that Democrats may not perform as well in polls of likely voters as among registered voters. As the election approaches, it tends to put more weight on the polls and less on these other factors, but it never zeros them out completely. (In this respect, it differs from our presidential forecast.)

By contrast, the Lite version of our forecast, which is more or less a polls-only view of the race, gives Democrats an 81 percent chance of keeping the Senate and a 41 percent chance of keeping the House. It also suggests that theyll win somewhat more seats: There are 52.4 Democratic Senate seats in an average Lite simulation as compared with 50.8 in a Deluxe simulation, or 212 Democratic House seats in an average Lite simulation versus 209 in a Deluxe simulation. Notably, this corresponds to current polls overstating Democrats position by the equivalent of 1.5 or 2 percentage points. Put another way, we should think of a race in which the polling average shows Democrats 2 points ahead as being tied.

Thats not quite the same thing as saying that the polls are systematically biased, though. Polls reflect a snapshot of what is happening today, and Democrats might indeed do very well if the election were held now instead of in November. In states like Ohio, for instance, theyve enjoyed a significant advertising advantage thanks to superior fundraising, but that will probably even out to some extent by Election Day.

Meanwhile, Biden and Democrats have also been on something of a winning streak lately, between a series of policy accomplishments, inflation trending downward and the political backlash to the Supreme Courts unpopular decision to overturn Roe v. Wade. But a worse-than-expected inflation report this week and a narrowly averted rail workers strike, which could have caused substantial supply chain disruptions, are reminders that uncertain real-world events wont necessarily continue to play out in Democrats favor.

Its also the case that in individual races, information besides the polls can help make a more accurate prediction, even when you have a lot of polls. For example, the partisan lean of a state still tells you something. Lets say the polling average has the Democrat ahead by 10 points in a state where the fundamentals put the Republican up by 2. Empirically, the best forecast in a race like this uses a blend of mostly polls and some fundamentals (exactly how much weight is given to the polls depends on how many polls there are and how close it is to the election). And you might end up with a forecast that has the Democrat winning by 7 or 8 points rather than 10 points, for instance. In that sense, in races such as Wisconsin and Ohio where there is a significant divergence between polls and fundamentals, Democrats probably should have concerns.

What I resist, though, is the implication that it can be presumed that the polls have a predictable, persistent, systematic bias toward Democrats. Is Rep. Tim Ryan going to underperform his current polls in Ohios Senate race? Well see, but more likely than not, the answer is yes. But is it just a thing now that polls always overrate Democrats?

Im skeptical. Here are seven reasons why:

Our historical database of polls shows that theres not much in the way of consistent polling bias. Two cycles of a pro-Republican bias in 1998 and 2000 were followed by a Democratic bias in 2002. A fairly sharp Republican bias in 2012 reversed itself, and the polls were biased toward Democrats in both 2014 and 2016.

Weighted-average statistical bias in polls in final 21 days of thecampaign

Bias is calculated only for races in which the top two finishers are a Democrat and a Republican. Therefore, it is not calculated for presidential primaries. Pollsters that are banned by FiveThirtyEight are not included in the averages. So as not to give a more prolific pollster too much influence over the average, polls are weighted by one over the square root of the number of polls each pollster conducted in a specific category.

Historically, the correlation between the polling bias in a given cycle and the bias in the previous cycle is either essentially zero or slightly negative, depending on whether you define previous cycle as two years ago or four years ago.

Pollsters get a lot of crap from people, but one nice thing about their job is that they regularly get to compare their results against reality. Sure, its possible for a pollster to get unlucky because of sampling error if you survey 500 people, sometimes youll draw a sample showing the Republican winning even if the Democrat is really up by 5 points. For the most part, though, pollsters can and do consider changes to their methodology based on errors in past elections.

And precisely because pollsters are subject to public scrutiny and there are relatively objective ways to measure their performance, they have strong financial and professional incentives to scrutinize their methods for potential sources of error and fix them if they can. Its the same incentive that a professional golfer has to fix his swing: If hes consistently hitting every shot to the left side of the fairway, for instance, at some point hell make adjustments. Maybe hell even overcompensate and start hitting everything to the right side instead.

Even if pollsters dont change their methods, the market will change the polling landscape on its own, at least to some degree. Pollsters who performed well in previous elections will get more business, and those who performed poorly will lose it.

For instance, weve seen relatively few traditional gold standard polls sponsored by major media organizations this cycle, perhaps because those polls tended to have a Democratic bias in 2020. Thats a shame, because most of these polling organizations have good long-term track records despite some recent problems. But it does mean that polling averages are more weighted toward Republican-leaning firms that have done comparatively well in recent election cycles, such as Rasmussen Reports and Trafalgar Group. This is especially true for FiveThirtyEights polling averages, which weight polls in part based on their historical accuracy. Groups like Rasmussen, for instance, get more say in the polling average than they did in 2020 because their rating is now higher.

As you can see in the table in the first point, polls did not have a systematic Democratic bias in 2018. That seems relevant, considering that was the most recent midterm.

Polls have also generally not had a Democratic bias in other elections in the Trump era when Trump himself was not on the ballot. They didnt have one in the Alabama Senate special election in 2017, for instance, or the Georgia Senate runoffs in January 2021, or in last years Virginia gubernatorial race.

There have also been some races where Democrats have overperformed their polls, such as in last years California gubernatorial recall election and in the 2017 governors race in Virginia. But these errors dont tend to get as much attention from the media as those that underestimated Republicans.

It may be that Republicans benefit from higher turnout only when Trump himself is on the ballot. A certain number of voters were willing to walk over glass to vote for Trump: Would they do the same for J.D. Vance, Mehmet Oz, Ron Johnson or Blake Masters? Evidence from non-Trump elections in the Trump era suggests maybe not. I tend not to buy the so-called shy Trump theory, or that voters are reluctant to state their preference for Trump. But it may nonetheless be hard to reach Trump voters, who may be more socially isolated, or who may be irregular voters who are screened out by likely voter models.

Democrats have had a lot of success in elections since the Supreme Courts Dobbs decision and importantly for our purposes, theyve done as well or better than polls predicted in these races:

I couldnt find any polls for the special elections in New Yorks 23rd Congressional District or Nebrakas 1st Congressional District, also held since the Dobbs decision.

Ironically, polls conducted before large parts of the country were shut down in March 2020 in response to the COVID-19 pandemic were more accurate than those conducted immediately before Election Day in 2020. Take the FiveThirtyEight polling average on March 1, 2020. It showed Biden up by 4.1 percentage points nationally, very close to his eventual 4.5-point popular vote margin. Our polling averages also correctly showed a very close race in states such as Pennsylvania and Wisconsin.

This may be because the pandemic profoundly affected who answered the polls. Specifically, Democrats were more likely to be in jurisdictions that implemented stay-at-home orders, and liberals were otherwise more likely to voluntarily limit their social interactions. Having more time at home on their hands, they may have been more likely to respond to polls. Thats less of a concern this year, with few voters treating COVID-19 as a high priority and few government restrictions in place.

Elections have consequences, and theyre relatively infrequent events. So the second-guessing and recriminations tend to linger for a while.

But that doesnt change the fact that peoples concerns about the polls stem mostly from a sample of exactly two elections, 2020 and 2016. You can point out that polls also had a Democratic bias in 2014. But, of course, they had a Republican bias in 2012, were largely unbiased in 2018, and have either tended to be unbiased or had a Republican bias in recent special elections.

True, in 2020 and 2016, polls were off the mark in a large number of races and states. But the whole notion of a systematic polling error is that its, well, systematic: It affects nearly all races, or at least the large majority of them. There just isnt a meaningful sample size to work with here, or anything close to it.

Again, that doesnt mean you should expect the polls to be spot-on. It may be that were living in a universe with larger polling errors than before in response to declining response rates. And there are some decent reasons to suspect that Democrats wont perform as well in November as they would in an election right now. Still, Ill stick to my usual advice: Prepare for the polls to be wrong in either direction.

See more here:

Will The Polls Overestimate Democrats Again? - FiveThirtyEight