Leading, and loaded, questions: And other ways to bias questions and polls.

      “When did you stop beating your spouse?” Ludicrous, but such questions happen. By phrasing the question such that some type of foundational axiom is assumed, you bias the situation and make it very difficult for the person to answer. In a poll, the answer, “I have never beat my spouse” is highly unlikely to be part of the limited choice of answers. In a direct interview, it is easy to edit the situation to make the reader or viewer assume that there is some basis for the question. A firm denial may make the responder sound defensive. Perhaps the best response, in a verbal exchange, would be to invoke humor — respond with “that’s the question I was planning to ask you!”.

     Such questions can be called leading questions, loaded questions, presupposition, or begging the question. In the court of US law, this situation of embedding an accusation within a question, is called “leading the witness”. The same thing can occur with inexperienced counselors and can be part of the process for implanting false memories.

Some additional leading questions examples (derived from Google searching):

• “You enjoyed the movie, didn’t you?”

• “How much do you love our new product?”

• “Wouldn’t you agree that our service is the best in the industry?”

  Any question that requires a “yes/no” response — that is not referring to factual data — is almost always leading. “Does 2 come after 1” is a factual question with a binary response. “Do you approve of Politician A’s desire to improve schools” is a question with two embedded biases — it states there is a politician’s desire that may, or may not, exist and it gives a vague directive that few would disagree with (“desire to improve schools”).

It is too easy to manipulate from either direction — the question or the responses.

     Have you ever encountered a poll (or a test, for that matter) where you said “Where is option E”? For polls, that is often deliberate. For tests — well, they probably don’t do it deliberately but it is probably done because of too little, or too much, experience.

What is the color of the sky?
a) Red
b) Yellow
c) Blue
d) Green

     The creator of the test PROBABLY meant the answer to be C (blue) but have you ever seen a sunset, or sunrise, that contained red, yellow, or orange? Very likely. It would have been better to rephrase the question to “What color is the sky, when not obscured by clouds, most of the time?” By the way, it is not unusual for the sky to appear green near tornadoes.

     I have seen something very similar to the following in a poll. I found it amazing that anyone would not see it as just a way to implant false information — but, apparently, many don’t see it that way. Note that it misleads on BOTH the question and the response.

How do you feel about the Democrats’ open door policy for immigration?

a) I think it is wonderful
b) I think it is terrible
c) I think all Democrats should feel ashamed
d) The policy should be squelched immediately

     Of course, Democrats have never had an “open door policy for immigration”. But the question indicates that they have had such a policy and, unfortunately, many people do not do any research on their own — so they believe it. Embedding lies within a question is a time-honored (though not honorable in any other way) method for politicians to mislead. There should be an “e” response:

e) I think the creator of this question should be seated at a blackboard and be required to write “I will not lie” 500 times.

     I am sure that there are questions with embedded lies for other political party situations — but I have seen the above sent in an email. As mentioned in a recent newsletter, it is always dangerous to just believe what you read or have been told. Research, get multiple viewpoints and data sources. What you hear is NOT necessarily what exists. Polls can indirectly lead people to accept lies as truth, and a bad poll question can lead many a recipient astray.

Watch out.

Thanks for reading Ideas & Interpretations! This post is public so feel free to share it.

Ideas & Interpretations is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Polls and Surveys: Not often a many-splendored thing

     As mentioned in my 300th blog, this blog is transitioning to a substack newsletter format. If you would like to continue getting my blogs/newsletters, please click here and sign up for a free subscription (if you have money to spare, you are welcome to get a paid subscription). Thank you for reading these. I hope they are of interest, and will continue to be of interest.

     It seems that election season is usually the time for a plethora (so many that a person gets sick of them) of polls. Surveys happen all the time, for many reasons, but polls usually happen for elections. Why? Polls are aimed at the future. What do you want to happen in the future? Do you want a nuclear power station in your town? Do you want to vote for Candidate A or do you want to vote for Candidate B (occasionally the poll will be written such that a choice for “neither” is allowed)?

     A survey is more of a gathering of information about what is happening right now. Do you put on your left sock first or your right sock first (Japan loves surveys of this nature)? How do you feel about the economy? Do you think that your local new pattern of “100-year” storms every four years is caused by climate change? How many pets do you have at home?

     Like weather forecasts, it is often true that polls don’t get it right (surveys don’t also but that’s for a different subset of reasons). In fact, many times polls are used in the opposite manner. They are manipulated to create a desired output, recognizing that there is a “winner effect” where people have a tendency to change their minds in order to be part of the “winning” side.

     Even when a poll is not deliberately manipulated to give a desired output, there are problems.

1. The questions must be created such that there is no inherent bias or assumption

2. The polls must be filled out (or answered) by a representative group of people

3. The people answering must have full anonymity — otherwise they may not give their honest answers.

     It is more difficult than one might think to keep bias or assumptions out of questions. “How often do you beat your spouse?” is rather blatant (though I suspect it has occurred on more than one poll). “How bad do you think the economy is?” has the same inherent assumption (that the economy is bad) as the spousal question but may not be as obvious.

     Many polls are multiple choice. But desired responses do not often align completely with one of the choices. The potential answers often have their own bias and assumption. If none of the answers matches what is wanted as the reply then there is a choice — don’t answer or choose the “closest”. But if your desired answer is the opposite of all of the listed possible answers then no answer is all you can do — and the poll will not reflect your opinion.

     It is very hard to get a representative number of people. For elections, they not only need to represent the variety of people voting in appropriate percentages but they must also be people who are going to vote. Younger generations often do not use phones for audible conversations so a phoned poll will be out-of-balance with leaving out many younger people (hopefully active voters). Although becoming less a problem as the technology continues to be more established, older people may be left out of polls if the polls are done via online methods. So, do both. But then the “middle” group may be over-represented.

     Legal voters don’t necessarily all feel comfortable in thinking about more complex issues in languages other than their home language. But polls (and surveys) cannot be created in ALL potential languages and polls are more unlikely to be answered if there is any struggle to complete them.

     We have many different physical, economic, ethnographic, and other areas of the country. Each has their own history and perspective so all must be represented in the polls. Very difficult.

     Finally, anonymity. In many countries, honest answers can be dangerous if the answers can be attributed to specific people. But even in honest democracies, attributed answers may result in unpleasant results (such as being bombarded with pleas for election money) if the person can be connected with the response. The answer may not be private within the house (or other physical location) and there may be intimidation or pressure to answer in a way not truly desired.

     Surveys can have the same types of problems but, since they deal with current existing issues, can change without causing the “chicken and egg” situations.

     The bottom line is that polls can only be treated as potential approximations. Their results should not be used to guide a person’s actions or expectations. If the poll results reinforce your opinions then be pessimistic and if they are contrary to your opinions then try even harder.

Polling: Problems and attractions

 

     I cannot claim that I have recent information but, about 30 years ago, when I was studying Japanese culture -- trying to decide to make the leap and start learning Japanese (one of my favorite Aunts was born in Japan and that has always attracted me to the country and its people) -- I came across an interesting tidbit. It seems that, at least at that time, minutely detailed polls were conducted with people. Things like "do you brush your upper teeth first or your lower teeth first?" or "Do you sleep on the left side of the bed or the right side of the bed?" Many readers would go to the poll results of previous polls before checking out the rest of the paper. It is not unlike enjoying the horoscopes first, or the crosswords, or the comic page of a newspaper (back when printed media held a primary position). The difference was the importance placed. It mattered to the readers as to what the answer was. Were they doing it "right"? In Japanese culture, the group comes first.

     Within the US, and many countries, polls are associated with political matters. Popularity, or unpopularity, of politicians. Popularity, or unpopularity of issues and possible approaches to those issues. It is not unusual for the polls to "say" one thing in an election and, following the election, have the results vary substantially from what the polls forecast.

     For political candidates, the questions on the poll are fairly direct and simple. Do you, or do you not, favor Politician X. In the case of issues, however, it is much like a general survey. How well are the questions phrased? Are there assumptions that lie behind the question. "Do you think that we are doing the correct response to situation A?" This question assumes that situation A is as stated. If situation A is different from that which is stated then the entire question becomes invalid. Given Situation Z do you prefer option G, H, J, or K? What happens if Situation Z is unlikely (or impossible)? What if you don't like G, H, J, OR K? Once again, any results from the poll question becomes unusable.

     Let's assume that the question is either simple and straight-forward OR it is formulated in such a manner such that the situation is true and the possible solutions cover the possibilities. What is the next problem in getting good results from a poll (or survey)? Audience. A survey, or poll, is supposed to be getting samples of answers from a small audience that can be applied to a much larger population.

     So, is the sample representative of the larger population? It doesn't have to be representative for ALL people. If the poll/survey concerns something related to college students, then it has to represent all college students. If it concerns only homemanagers, then it only has to represent all who do the multiple-job position of a homemanager. But the sample must be an accurate representation of the population. If you multiply the sample by a hundred, thousand, or a million then the sample should expand out into a close copy of the full population. What are possible problems:

• Getting in touch with a representative population. Once upon a time, this could be done via door-to-door polling or telephone polling. Not anymore. The population connects with the rest of the world via many different potential avenues. And, by leaving out all of the people who communicate only via SMS, you may no longer be representative. Or by leaving out all of the people who communicate only via a particular social app, or game chat conduit. Not only does this make it much harder to communicate with your representative group but it makes the poll/survey much more complicated, and expensive, to do correctly.
  
  
• Getting people to respond. Our landline rings 10 times a day. We only answer if we recognize the caller (and occasionally that is spoofed and the caller is someone unknown). Some days, we never answer the phone. A message can be left -- but usually they do not leave a message. The same situation exists in other types of connections. Even if you locate the appropriate people, they have to be willing to respond.
  
  
• Will they give a truthful answer? For some topics and questions, the respondent won't want to tell you what they really feel. They feel that it is the "incorrect" answer and may be bashful, or ashamed, to tell you. But, when they have anonymity, they will answer as they really feel. Closed, anonymous, survey answers can avoid this -- if they are trusted to remain anonymous -- but that makes the survey/poll more complicated, and expensive, once again.

     More than ever, polling is harder to do accurately. So, how much faith should be placed in a poll? Given the lack of an alternative, they may be better than nothing but don't expect them to be fully accurate. In addition, people like to be "part of the winning team" and, if they think an answer is more popular and acceptable, they may choose answers based on what they think is the "winner". Whether they will continue to support that opinion is unknown.

     Polls are meant to be an indication of the thoughts of a larger population. But improper use of polling methods can actually be used to influence the thoughts of the larger population. In a parallel sense with the group mindset of the polling of the Japanese population as mentioned at the beginning of this blog, the poll results can be used as indicators to the larger population as to what they "should" think. Thus, polling can work both directions. The sample can reflect the opinions of the larger population and the larger population may allow the smaller sample to indicate the opinions that they should have.

     Be careful with the interpretations of poll results and carefully examine the questions and answers.