June 13th, 2005
Frequently I use this blog to criticise surveys that are badly designed, ill conceived, or perhaps being used for ulterior motives (usually to sell products). I’ve published several papers on ‘how to’ do survey research, so perhaps it might seem obvious that:
a. I know my onions when it comes to surveys
b. that gives me the right to complain about other people’s questionnaires.
However, even experienced researchers don’t always get it 100% right.
Recently I’ve designed a survey on bullying in Higher Education. This research is ongoing, and so far has received a lot of responses.
The process of putting the research together was painstaking – several pilots of the survey at different stages of development, checking with existing research, and confirming the survey was as accurate as possible.
In spite of this, a couple of survey respondents have pointed out a few areas where I could have done better.
On one question I asked about sexuality. I gave the options ‘straight’, ‘gay’, ‘bi’ and ‘other’. I forgot, of course, that many lesbians prefer to categorise themselves as ‘lesbian’ rather than ‘gay’. Now why I forgot it I don’t know since only the other week I complained about a colleague doing exactly the same thing.
On another question I asked about being ‘registered disabled’ and rightly people have pointed out that in order to keep up with current legislation (not to mention being accessible) I should have said ‘do you consider yourself to have a disability?’
The miswording of these two questions fortunately hasn’t prevented people from answering. But they’re not exactly welcoming to a group of respondents whose views are much needed in research – and who may already feel excluded. Not asking questions in the right way simply compounds that exclusion. And there’s no excuse for this.
Given the survey is in mid-flow, I can’t alter the questions now. But I can acknowledge I could have done better, and I can apologise for not being as sensitive as I should have been.
When research is published we usually see the ‘cleaned up’ version, meaning problems and oversights in a study are not reported. As you can see from some of my entries in this blog, when academics decide to unpick someone’s work they don’t exactly hold back. Often academics don’t admit to any oversights because they don’t want to give critics any more ammunition.
Still, if we’re not transparent about our research practices we don’t learn, and nor do other would-be researchers. So I’m coming clean here.
The next time you see me stamping my feet in this blog about a shoddy survey, remember that even experienced survey designers can stuff up sometimes. But when they’re ethical, they’ll at least hold up their hands and admit their mistakes.
That’s what good research is about. Realising that even in the most meticulous piece of work things can still get missed – and then learning from your mistakes.Tweet