Questionnaire Design 101: Everybody fails.

Post Reply New Topic

I love questionnaire design. Creating meaningful surveys that people know how to answer is a passion of mine. But I get frustrated (sometimes really frustrated!) when I read other questionnaires, often written by highly trained professionals, and they make rookie mistakes!

This is the most common mistake I see, it happens in nearly every psychology-related survey I come across: Using "frequency terms" in the questions then asking for a scaled response.

Example: I sometimes find it difficult to see things from the "other guy's" point of view.
Response: On a scale of 1-5, from Does not describe me well, to Describes me very well.

Okay. So say I'm a regular person, who occasionally has some trouble taking the other person's perspective. I would think this is fairly normal. How do I answer the question? To say that I sometimes find it difficult to see things from the "other guy's" point of view describes me very well, does it not? So I should give this question a score of 5.

But I know that's not what they mean. They want me to answer 5 if I ALWAYS or NEARLY ALWAYS have this problem. Why didn't they word the question like that then? A better question would be: I find it difficult to see things from the "other guy's" point of view. Remove the "frequency term", i.e. sometimes, and the question allows for accurate responses through the entire 1-5 scale. Even better, for questions where it is important to focus on the frequency, make the Likert scale a frequency scale, e.g. from This never/rarely applies to me, to This always/often applies to me.

Have a look at some questionnaires and you will see how common this is. It appears in the AQ, and to a lesser extent in the SQ and EQ. I just looked at a test on Empathy (where my above example came from) and found this mistake in about half of the questions.


If you feel like having a rant about questionnaires too, feel free to add your personal "survey pet hates"

One thing that stands out to me regarding psychological surveys is how subjective many of the responses are - often to the extent that the question is effectively meaningless for evaluation purposes. I've done a few EQ tests and I used to rate myself reasonably good at reading people's body language and facial expressions. However, one day I did an EQ test that had visual questions - there were faces to look at and photos of small groups of people and I had to select what particular people were feeling from a small list. My abysmal score on that objective test showed that my responses to the other (subjective) tests was extremely incorrect. In short it turns out I have a very low EQ and am very poor at understanding facial expressions and body language. I simply did not know that "normal" people could read so much information from those things.

It reminds me of the situation when I was 18 and thought my eyesight was perfect until a friend was passing around her new spectacles during class and I tried them on and was stunned to discover I could see much better with them. This resulted in me visiting the opticians and wearing spectacles ever since! So much for subjective evaluation of one's abilities.

The problem I have with those is when they make it all black and white. There is no sometimes. It's either yes or no. As if you can't have the problem off and on?

Indeed, when the response options are just "yes" and "no", that's pretty frustrating. If there's a scale, then I think it's better to leave "sometimes" out of the question, and put it in the response options instead, e.g. Always, Usually, Sometimes, Rarely, Never. Something like that. Even that type of measure has its flaws though.

I saw one recently that seemed to be trying to avoid this problem, by wording it "People often describe me as...". That's a tough question for the sort of people who might not have a lot of friends, or who don't talk about this sort of stuff! Is the answer "no" just because nobody has explicitly told you how they would describe you? You'd have to guess what they MIGHT say, and that just takes us back to subjectivity.

It probably doesn't matter much from the designer's point of view, but I hate questionnaires that ask you to choose between "strongly agree" and "somewhat agree", but then score them exactly the same! AQ test, I'm looking at you. I probably would have spent 20 minutes less on that test if it didn't make that distinction.

Many questions just tend to be too vague for me. The standard advice that accompanies them is "don't think about them too much, just go with your gut feel". OK, but what does that really mean? The answer is completely different depending on how I interpret the question and two (or more) interpretations have occurred to me. (I suppose one must have occurred to me first, but it happened so quickly that I didn't register which one.) Should I just pick randomly? Would the results mean anything if I did that? Then again, maybe all those tests are normed on people who do answer based on their initial reaction, so it's actually by thinking about questions that I ensure that the result is inaccurate!

Related to that, often they'll ask many very similar questions and I know that's done to detect people answering randomly. The problem is, they're similar, but not the same. Example:

I am a worthwhile person. (agree/disagree)
then, 50 questions later:
I certainly feel worthless at times. (agree/disagree)

I look at that and feel almost sure that they expect the opposite answers, but the true answer is "agree" to both. So I stare at the question for 3 minutes wondering: "will I trip up some validity indicator if I agree here or do the designers of the test realise that it's not quite the same question?"

(+1) "I strongly agree"

One of the big problems I have is that they use complicated words. Like they use a word the average person may not really know; it's a pain having to look up definitions while taking a quiz/test. Or they use a word that the average person uses quite differently than it's official medical/psychology meaning. Or they don't elaborate on context; like they'll use a word or phrase or something that most people would use when applied to a certain type of thing or situation but it could also be used in a general context & I'm not sure if they're talking about how I am in general with most things or with a certain type of situation & it can vary.

I did some research using the AQ, and scored it the way they recommend, and also by distinguishing between "strongly" and "somewhat". I liked the results I got using the second method better.

My pet hate is when they ask multiple versions of the same question
e.g.
Q2 "I like eating banana's"
Q7 "I dislike eating bananas"
Q21 "I consider myself to like eating banana's"

And also really subjective questions, where I am not sure how they are supposed to be interpreted or answered.

I hate when they give you a choice or a comparison to make & neither of the options apply to you or you like them all equally or dislike them equally. For example this popular Myers Briggs Personality test click link ask "Often you prefer to read a book than go to a party" & my choices or Yes or No. I do not like either. I think about the question briefly & I think it's trying to find out if I am introverted or extroverted; Would I rather go out with a buch of people or stay in doing my own thing; I'm an introverted type & would defaintly rather stay in doing my own thing so I answered it Yes