Ranking the Pollsters

Trump Surges Ahead! Again! No, Really, For Real This Time!

The headlines and talk shows are filled with deep, thoughtful sounding people explaining why the GOP is flocking to Trump in recent polls. Theories range from his aggressive stance towards ISIS, a growing fear for border security, or Carson's recent fumbling of foreign policy. 

The truth is much simpler. Trump's support hasn't changed. 

The real culprit is that most pollsters tend to release their numbers on a monthly basis, though not all at the same time. Right now we have Bloomberg (Trump +4) and PPP (Trump +7), with Quinnipiac (Trump +1) and NBC/Wall Street Journal (Carson +6) polls as the last ones to compare to. If they keep to schedule, we should see CNN (Trump +5) and ABC (Trump +10) polls coming out soon, which are likely to continue the perceived trend of "Return of the Trump" before the next CBS/NY Times (Carson +4) and NBC polls causes pundits to wonder if Trump's decline could spell the beginning of the end of his campaign.

I can't be sure which pollster is right (yet), but let me show you some trends on Trump's versus Carson's poll numbers:

21 vs   5 (08/02)   21 vs 16 (09/21)   24 vs 20 (11/17)    Bloomberg

29 vs 15 (08/30)   27 vs 17 (10/04)   26 vs 19 (11/17)    PPP

28 vs 12 (08/25)   25 vs 17 (09/21)   24 vs 23 (11/02)    Quinnipiac

19 vs 10 (07/30)   21 vs 20 (09/24)   23 vs 29 (10/29)    NBC/WSJ

24 vs   9 (08/16)   24 vs 14 (09/19)   27 vs 22 (10/17)    CNN/ORC

24 vs   6 (07/19)   33 vs 20 (09/10)   33 vs 22 (10/18)    ABC/WaPo

If we look at the individual pollsters, Trump is holding steady since August. Carson has been building support, though that may have slowed recently. Using this data, we can predict that the CNN and ABC numbers will be very similar to the October 17 & 18 numbers, with Trump holding steady and Carson gaining around 2-4 points. However these numbers are still noisy, so any individual poll might be off by as much as five points. The trend and the aggregate is much more stable. Looking too closely at any one poll is like blindly touching the trunk of an elephant and deciding you’ve caught a snake.

The Hidden Romney-Trump Connection

This effect of differing pollsters is of course not new to this election. A look at the last election is illustrative, and can perhaps serve as a starting point for interpreting these latest polls.

In 2012, I tracked all of the general election polls for the President and created an excellent predictor for the results of the national popular vote, as well as each of the individual states. Looking at the final month of predictions, corrected by the actual results in each state, I computed the bias (consistent error) and standard deviation (noise level) of each major pollster. The results are shown below, with a positive bias indicating a consistent over-prediction for Romney.

2016 Bias       2012 Bias          2012 S.D.    Pollster

 -3.3 Trump      -2.3 Romney     2.2             ABC/Washington Post

                      -1.3 Romney      5.1             Marist College

 -3.1 Trump     -0.7 Romney      4.0             PPP

 -0.7 Trump     -0.3 Romney      3.4             CNN/ORC

  0.3 Trump      0.8 Romney      4.3             Quinnipiac

  1.9 Trump                                               Bloomberg

                       1.2 Romney      3.4             IBD/TIPP

  7.7 Trump      1.3 Romney      2.6             NBC/Wall Street Journal

                       1.3 Romney      3.6             Fox

This shows that even in 2012, the pollsters were all over the place. However, the three most accurate were CNN/OPC, PPP, and Quinnipiac, and their bias mostly cancelled each other out. ABC/WaPo was way off in one direction while NBC/WSJ was off in the other (though not quite as extreme). If we’re brave enough to carry that lesson forward, we would expect to see the best three pollsters in the middle again, flanked on either side by ABC/WaPo and NBC/WSJ. In fact, that’s not too different from what we see across the past three months, with Quinnipiac and CNN/ORC sitting in the middle with NBC and ABC sitting at the two extremes.

The really interesting thing is the strong correlation between a bias towards Romney, and an apparent bias towards Trump. The complete ordering of most to least biased is preserved. This would lend one to believe that something consistent is playing out in the pollsters and their underlying models of the electorate. Perhaps correcting the pollsters is more tractable than it looks. However, determining how to capture that correlation, and how that relates back to the actual election will be the trick. I’ll keep you posted.