When it comes to survey research, one of the most critical aspects of the entire process is the design of the questions which comprise the survey. Without questions, there is no survey. However, not all questions are created equal. In some instances, asking a “bad” question is worse than not asking one at all.
What makes a “bad” question, you ask? Occasionally, it is obvious when a question is worded poorly. For example, if a pollster were to ask, “How well is Politician X performing as Governor,” they are already biasing respondents towards thinking that Politician X will perform well. What about those who do not expect Politician X will perform well?
A better way to phrase this question would be: “How do you believe Politician X is performing as Governor?” With this wording, the pollster does not lead the respondents to assume the Governor is guaranteed to perform well.
In other cases, it is much less apparent if a question isn’t designed well. Another potential misstep surveys can fall victim to is that of “social desirability bias.” When asking about things that most people believe they should be doing but aren’t (or vice versa), respondents are likelier to lie to the person asking the question. According to the Pew Research Center:
“[R]espondents understate alcohol and drug use, tax evasion and racial bias; they also may overstate church attendance, charitable contributions and the likelihood that they will vote in an election. Researchers attempt to account for this potential bias in crafting questions about these topics.”
A perfect example of the “social desirability bias” in action can be found in two questions from OH Predictive Insights’ August edition of the Arizona Public Opinion Pulse (AZPOP). One question asked, “When you are out in a public place, how often would you say you have worn a mask or other type of face-covering?” and another asked, “When you are out in a public place, how many people around you would you say have been wearing a mask or other type of face-covering?
If there were no such thing as social desirability bias, one would not expect a meaningful difference between the answers to the two questions. However, this was not the case.
It is by no means surprising that the more “socially desirable” answer has the highest percentage of respondents. People may be embarrassed or ashamed to admit they do not always wear masks, whereas there is not nearly as much social pressure to acknowledge that people around you aren’t wearing masks.
Here at OH Predictive Insights, we know the importance of survey data reporting best practices. We’re embedded deep in the heart of the great state of Arizona. We live, work, and play here, and as native Arizonans, we know the types of issues the voting public face on a day to day basis. This makes us ideal for your Arizona survey needs, as we know how to find the right people and ask them the right questions to provide the most accurate and insightful results for your organization. Turn to OHPI for Arizona survey results that you can trust.
If you liked this blog post and would like to see more, please subscribe to our newsletter!