POLLS, POLLS, POLLS – What Do They Really Tell Us?

by Ken Grossberger, PhD

It’s the silly season (again) and all the punsters and pollsters are making predictions as to who is leading and who will win. How do they know? The answers they use are in the almighty polls, but what do those polls really tell us?

The consensus is that voters begin to seriously focus on the election after Labor Day, so the early polls may provide an indication, but we need to look deeper to understand what the polls may mean. Polls are not necessarily predictive but are designed to provide qualitative data on popular sentiment. Gauging a number of polls over time yields trends, which are retrospective, not prospective. In other words, political polls look at the present and the past, not the future. Yet many play the prediction game because we think that what is probable today might tell us what’s going to happen tomorrow. Not necessarily.

Political polls are public opinion surveys, and thus we have to be concerned with the data collection methods as well as the data parameters. For example, in order to collect data many polls use a technique known as Random Digit Dialing (RDD) to ensure all voters in a population have an equal chance of being called. The survey staff must be properly trained and supervised, and samples should be large enough to be statistically meaningful or they may suffer from small sample size validity issues (which may lead to results that are true, when they are false, called Type II error in statistics).

For instance, a recent Reuters/Ipsos poll (04/15/24) showed that 833 registered voters preferred Biden to Trump by 41% to 37%. The margin of error (MOE, the statistic that indicates how far off a survey may be) was 4% with 22% of those polled having no preference/might not vote/favoring a third party candidate. So what does this really tell us? The media would report that Biden has 4 point lead, but is that accurate? Does this poll have any realistic predictive value?

My observations are as follows: any sample of less than 1,000 respondents is too small to have much statistical power, and the results are within the margin of error as well.  Also, 22% undecided is too large a group to give us confidence in the determination as to which candidate may be leading. This is just the tip of the statistical iceberg. Many polls suffer from considerable statistical issues:

• Registered voters vs. likely voters: polls that measure registered voters only are weak as they measure people who may not vote; likely voter responses are more meaningful
• Timing: polls from last month may be out of date in an age of fast paced news cycles
• Sample size: it’s doubtful that a sample of less than 1,000 voters could be representative of the voting population of the United States
• Undecided responses: the Reuters/Ipsos poll mentioned above had 22% choosing something other than Biden and Trump, with only a 4% difference between the two with seven months to go to the election – this makes it very difficult to draw any meaningful  inferences
• Margin of error: rates over 4% are usually too large to provide confidence in the result, and leads within the MOE means no one has a lead
• Outlier polls: a result that is far from the average of the other polls – the media on either side loves to quote polls that show their guy in the lead, even if it is way outside the average

So to revisit the Reuters/Ipsos poll, a result that seems to favor Biden: it’s a small sample of registered voters only, a result within the MOE, with a large amount of undecided. If you’re looking for a prediction, don’t bet any real money using this poll.

Another issue is using the average of polls, a method which is supposed to give us a solid basis as to who is leading whom. Perhaps the most popular is the RealClear Politics (04/15/24) average of polls (widely quoted by the media) which is an unweighted average, meaning it’s a simple average of polls with different sample sizes. Here is a recent report:

POLLSTER	DATE	SAMPLE	MOE	TRUMP (R)	BIDEN (D)	SPREAD
RCP Average	3/21 – 4/7	—	—	45.5	45.3	Trump+0.2
Morning Consult	4/5 – 4/7	6236 RV	1.0	44	43	Trump+1
Reuters/Ipsos	4/3 – 4/7	833 RV	4.0	37	41	Biden+4
I&I/TIPP	4/3 – 4/5	1265 RV	2.8	40	43	Biden+3
Emerson	4/2 – 4/3	1438 RV	2.5	46	45	Trump+1
Rasmussen Reports	3/31 – 4/2	1099 LV	3.0	49	41	Trump+8
Data for Progress (D)**	3/27 – 3/29	1200 LV	3.0	46	47	Biden+1
NPR/PBS/Marist	3/25 – 3/28	1199 RV	3.7	48	50	Biden+2
Forbes/HarrisX	3/25 – 3/25	1010 RV	3.1	50	50	Tie
FOX News	3/22 – 3/25	1094 RV	3.0	50	45	Trump+5
Quinnipiac	3/21 – 3/25	1407 RV	2.6	45	48	Biden+3

But when the samples are weighted by the number of voters (a sample of 1,438 is given more “weight” than a sample of 833), Trump’s lead doubles. Also, note how many of the polls are “RV” or registered voters, and that some of the polls are older, meaning from prior news cycles. The age of a poll is an issue as in just one month Biden may have any number of mumbled gaffes, Trump may insult another dozen people, and another war or two may break out. All of these sway public opinion and affect the polls.

Anyone who draws firm conclusions from these polls is engaging in a neurotic level of wishful thinking. But there is hope, many surveys over a period of time (longitudinal, in statistical language) generate more confidence than one-off polls (cross-sectional). There are no simple answers, we have to do the work. For example, most recent polls show Trump with a lead outside the MOE in the swing states over a period of time, and we therefore can have some confidence in that result. In recent history polls tend to tighten as we get closer to election day, and much can happen. We will see what events move the arc of the current trajectory.

References:

RealClear Politics. 04/15/24. RCP Poll Average. RealClearPolitics – Live Opinion, News, Analysis, Video and Polls. RealClearPolitics – Live Opinion, News, Analysis, Video and Polls

Palmer, Ewan. 04/15/24. Donald Trump’s Polling Numbers Are Dropping. Newsweek. Donald Trump’s Polling Numbers Are Dropping (newsweek.com)