What's missing from research and assessments: Feedback

Post Reply New Topic

I'll contain this to talking about ASD research but it really applies to all psychological research.

Studying autism is great, in my opinion. I'm a huge fan of gaining knowledge through scientific research, even just for its own sake if not to improve the lives of fellow human beings. But I'm not such a huge fan of poorly conducted research, which leads to misleading results, leading to false perceptions out in the general community.

I've studied psychology, which included units on questionnaire development. There are some really basic rules when writing up a survey that get broken ALL THE TIME. I'm yet to find a survey that doesn't break at least one of the simple rules. This is endlessly frustrating. But there's another aspect that I don't think was covered in my course, and I think it's potentially an important feature to add to quantitative surveys, as well as qualitative assessments, to help give more meaning, understanding, and hopefully more validity to the results...

In-depth participant feedback.

For example, you answer a bunch of survey questions, putting your responses into a neat little package of 1 to 5's. The researcher collates all your data and makes assumptions based on your and everyone else's results. You feel happy to have helped out, but are thinking "I'm not sure my answer to that question really reflects how it is for me/what I'm trying to say/what I think you want to know". I don't know about you but I have that reaction OFTEN.

Another example, you participate in an interactive assessment where you are asked to complete a bunch of activities. There's one you are really struggling to complete, but the observer doesn't ask you WHY. They observe that you struggled with it, give you a low score or something, and then go away and come to their own conclusions as to why you "have a deficit compared to NTs" in that skill. You walk away feeling that you have more to tell, that there are thoughts and feelings going on in your head that they don't realise, but which would be important to consider.

So if there are any psychology students about to do their honours or PhD thesis and are looking for a topic, this is something I personally would love to study if I was able: What insight do you gain by adding comment boxes beside each survey question, or by asking your participants "why" more often?

Sure, it takes a LOT more time and effort to add this element to your research. But if it gets us NEW INSIGHT with which to build our autism theories, isn't that pretty important to know?


Feel free to comment on any aspect of this. In particular though I'd be interested to hear why this might not be a good idea, or why we might not gain anything from it so would be wasting our time and resources. Do you think that researchers currently get the right information from the standard tests they use? (e.g. the AQ, the ADOS, the WAIS...don't get me started on the Eyes Test )

Feedback is important, even if most is useless, because there might be small scrap of information that generates next new idear of what new question to answer or eggsperimental condition to test.

Yes, exactly. If someone has the time and funds to wade through the feedback, there could be some real gems to discover.

That's why I think it would be a great thesis topic - you don't need to apply for funding or anything, and you'll have at least 6 months to work on it. If something really interesting is found, publish it and hopefully a research company will pick it up and expand on it.

The key I think would be to look for trends (and lack of) in the feedback. Are a whole lot of participants telling you something you didn't consider before? Pay attention to it. Is everyone giving you completely different feedback to each another on a particular question? That question might be crap, get rid of it.

I do autism/neuroscience research, and I ask participants for feedback after the eggsperiment, basically whatever they want to tell me about what was going on inside their minds during the task, but it would be useful to have a post-eggsperiment questionnaire with a few specific feedback questions, but sometimes, I want the participants to remain naive to most aspects of the eggsperiment in case I need them back for followup, so in that case, feedback questions must not give away anything about the eggsperiment.

That's amazing, I have never had anyone ask me for feedback and I participated in a LOT of psych studies at uni.

Do you have any examples you can give of anything really interesting you learned, that you wouldn't have learnt if you didn't ask for feedback?

I won't say specifics about pilot study, but I based design of one of my new eggsperiments on feedback I got from two autistic participants who were just trying out my new task for fun while they came to lab to do another unrelated study.

I've had very little exposure to this sort of thing. However, I do remember when I've done many of the questionares from WP links;

They often ask more than one question in the question.
They often ask a question then make a statement at the same time. The statement is an assumption of a correct anser, but is often not the correct answer which causes confusion how to answer.
They often ask for feelings or similar.
Many questions are too broad, there are too many if's and but's to make a single statement about the subject.

I think, they should ask as small a question as possible. I don't care if I answer a lot of questions, but please make them as small and specific as possible. Where the only possible answers are Yes or No.
Avoid any question were "sometimes" is a valid answer.

For example.
(do) Have you ever taken drugs?
(don't) Do you use drugs?

This example is double bad because which drugs? legal ones or illegal ones? what is the question?

Yeah, those things you are talking about are the basic "rules" of questionnaire design that most questionnaire designers fail at.

If you had a comment box next to each question, you would be able to bring this up. You could explain why you chose the answer you did, so the researcher could determine if everyone was answering it the same way, or if they were all quite confused by it and answering in different ways.

Unfortunately when someone undertakes undergraduate research, they normally have to use the pre-existing "validated" surveys, and can't even tweek them a bit to help improve them. But the research done to "validate" these surveys rarely involves asking participants for qualitative feedback. It's more like "90% of the respondents who answered the question in this way have been determined to have ASD so the question must be working".

Ironically, I took a series of psychological questionnaires for a program I'm involved in last week because they wanted my feedback on how it was run. All of the faults the OP and GregCav mentioned appeared in those questionnaires.

This only makes my blood boil if I see two questions asking the exact same thing in completely different wording.

Ex: I feel like I can call on a friend to support me emotionally when I need it.

(10 questions later)

I feel like no one is around to support me emotionally when I need it.

Last edited by AScomposer13413 on 31 Oct 2013, 8:56 am, edited 1 time in total.

That's a good example question for a different reason as well. There are so many reasons why I might answer "no":

- I don't feel that my friends are supportive
- I don't feel comfortable calling on my friends for support
- I couldn't call on a friend, but I could call on a parter, family member or colleague
- I don't have any friends
- I don't need emotional support
- I don't want emotional support

There are probably more. So which of those is the researcher going to assume is your reason? Shouldn't they confirm your reason? I think they should.


(Two questions worded differently is supposed to be for checking that you are answering honestly and not just picking numbers at random or using other methods of not filling out the survey properly.)

Only an aspie researcher would ask themselves "I wonder if the respondents actually interpreted the questions and the answers the way I intended?"

... which is why we need more aspie researchers! Yes, I think in-depth feedback would be great for improving the research methodology. Obviously that would make things more difficult: you need more participants and much more time. There's also some chance that you'll realise your research is completely useless in its current form and I suspect that's a risk many researchers don't want to take.

Would feedback actually contribute to research results, though? For example, are the reasons autistic participants give for their difficulties useful to researchers? That's a harder question. I'm glad you said it needs research, because the naive assumption that more information is better is certainly not always true. Another problem I can see is that free-form feedback would be hard to quantify and aggregate and therefore hard to analyse.

I have a little bit of personal experience with this. The closest I've come to psychology research is software usability studies. This is basically where software developers/designers observe people unfamiliar with their software using it to accomplish some pre-determined tasks. The idea is to see what people have difficulty with and improve the user interface to make it easier to accomplish those tasks. Subjects are also asked to "think out loud" as they do this, but most of them don't do that very well. At the end there is a small survey, too, but this is a very minor part of the whole thing and the reason is simple: it's next to useless. What the users tell you is often inconsistent with how they actually behave, so we more or less ignore what they say and concentrate on what they do. They may say "this part was easy/hard" - but did they accomplish the task successfully or give up in frustration? How long did it take them? What did they try first? That's the stuff that matters. (People often found it hard to accept that we really don't care about what they have to say, so we had to go to some pains to explain to them that the study is not a survey. )

I think that researchers are unlikely to get big theoretical insights from participant feedback, but asking participants about what is going in their minds instead of assuming what is going on is good idear, and usually this is done verry merry berry quickly after an eggsperiment, and so far, has helped me in determining which eggsperimental conditions to try and refining direction of which questions to ask and how to prioritize questions and eggsperiments than to understand something big about autism or brain.

And yes, what participants tell you afterwards is often contrary to their measured behavior or performance during eggsperiment task.

Other people have said that participants often see patterns that don't eggsist in their eggsperiments.

One example that comes to mind is Theory of Mind. I've read a few anecdotes on this forum that suggest that people with ASD don't necessarily have a weakened ToM, and depending on how you think about it, perhaps a strengthened one.

E.g. Ask an NT and AS participant a ToM question about what some character in a scenario was feeling. The NT participant says "she was happy", the AS participant says "I don't know". Write that up in your research report as the AS participant clearly having a weakened ToM, because they were unable to attribute a feeling to that character based on the information given. OR...ask the NT participant why they gave that answer, and ask the AS participant why they don't know. The NT participant says "it was just obvious!" or "because she was smiling!" or "because she got what she wanted!", the AS participant says "because I'm not her, there's no way I could know for sure what she feels unless she tells me. But if I had to guess, I'd say she was happy.".

I could be wrong, but if I were that researcher, I'd see that as valuable information that would affect the way I analysed the results.

The researchers usually wouldn't ask questions in such an open-ended way, but instead design an eggsperiment that tests for one or few things and also make sure that participants know what the researchers want them to try to do during the task, such as by doing practice trials before the task.

Also, the researchers know that hfa adults and older children have basic theory of mind that they can deploy on social cognition tests.

you can't plug extended feedback into a chart...

most people who do the surveys have a goal in mind...

these same people would usually be too lazy to go through all the feedback....

if it's a large sample who could go through all the feedback....

much cheaper to keep things simple...

(much of above partially already covered)

not that I am opposed to this idea...I think it would improve on these tests/assessments

I think it would be more accurate with a feedback element to the tests/assessments. I often question the accuracy of such tests because there is always so much more going on than is marked down on such assessments/tests.

Yeah, what I'm suggesting is more about figuring out whether the (quantitative/original) data are telling you what they think they are telling you, not so much about analysing the feedback data and reporting on it.