May 27th, 2009
West Yorkshire police had a surprise when they implemented a staff diversity survey recently. 11657 staff were asked to report on their gender, sexuality, ethnicity, and religion.
They were also asked if they already had undergone, or were in the process of having, gender reassignment.
Apparently 300 participants who answered that particular question claimed they’d had (or were having) a sex change.
Newspaper reports quote a police spokeswoman admitting these findings were “wide of the national average”. She added “there were suggestions that individuals may have not understood the question. Due to the questionnaires being anonymous, the results couldn’t be followed up with individuals. After careful consideration we felt it was appropriate to exclude the data.’’
Cue hilarity in the media, blogs and forums. And lots of jokes about truncheons.
But actually, once you’ve stopped smirking, this is an important case study for anyone interested in survey design.
One of the first rules of completing any survey is to be specific about what information you need to collect. And to know what you’ll use that information for. There’s no point in collecting data that you won’t be able to use. It seemed from reports of this survey, data was being collected but it wasn’t really clear what it was being collected for.
In this case the survey was attempting to measure diversity in the workplace. So asking about gender isn’t unreasonable. Given that many Trans people may well struggle with discrimination or harassment in the workplace then it is sensible to find out if any of your staff fall into categories who might require support.
You do need some insight into your potential study group in order to collect information. If West Yorkshire police were under the impression they had a number of Trans staff then it might well be worth asking about gender reassignment. But was it worth asking this question just on the off chance they might get some interesting data?
Perhaps the question was included because legally it is unacceptable to discriminate against a person who ‘intends to undergo, is undergoing or has undergone’ gender reassignment [Sex Discrimination (Gender Reassignment) Regulations 1999]. But again this would only make sense if there were follow up questions to identify if people did feel discriminated against – or needed further help. From the press coverage it isn’t clear in what context the question was presented.
A further key issue in survey design is you need to know your participants before you start. That might mean making a survey in large font for visually impaired participants, or delivering the survey online to increase anonymous response rates on a particularly sensitive issue.
It also means thinking about how participants may respond to certain issues. It’s pretty obvious that, given the macho culture that exists within the police force, any question on gender reassignment may well invite joke reactions.
You can get round this in the way you phrase your question, or move instead for an open ended option where participants can disclose more personal issues if they wish.
Clearly it’s not reasonable to avoid asking certain people certain things for fear they’ll mess about with your survey. That’s why piloting is important as it can highlight for you any questions that are unclear or invite joke responses.
Piloting also helps you identify if people even understand the question. In this survey the police spokeswoman stated participants may not have understood what the gender reassignment question was about. That might explain the high volume of answers agreeing participants had had a sex change.
Before you think that can’t be right, here’s an example I found in one of my surveys. I found in one study a lot of people were answering they were homosexual, but through piloting I found they weren’t actually gay. They just weren’t sure what heterosexual meant. So they opted for a term they had heard. This didn’t really represent their sexuality.
It is possible in the current study participants didn’t know what gender reassignment meant (if that’s how the question was phrased) so answered they were undergoing this process out of confusion.
It’s very unlikely West Yorks police force does have 300 employees who are undergoing or have undergone gender reassignment. It is possible many of the employees didn’t understand the question so over reported on this. Or – and this is most likely – a group of very sceptical participants either showed their dislike or disrespect for a survey question.
What can you do when this happens? Well, hopefully in a well designed and well piloted survey this kind of problem won’t happen. But that’s not always guaranteed. Even the best survey designers can find themselves stuck with answers that don’t seem quite right.
Fair play to West Yorks police for admitting they spotted a problem with the survey. Many people wouldn’t have done this.
Shame the media still consider this to be a story though. Part of me does smile when I think of the more conservative members of the police force being faced with a question on gender reassignment. But part of me is sad to think we still consider this an issue worth sniggering about.
I agree they made the right decision in deleting the ‘problem question’ from the analysis. However, this still raises an issue of whether the responses to the wider survey were reliable. If 300 people reported they were undergoing/had undergone gender reassignment then is it okay to just drop that question from the analysis – or might you conclude if that answer was unreliable the rest of the responses also weren’t trustworthy. So you might have to drop all 300 people from the analysis altogether.
Gender issues are undoubtedly worth reporting on, and certainly have a role in explaining workplace issues. But there’s no point in asking about this if you’re not going to get any reliable responses.
If I were doing a survey like this I wouldn’t ask a closed ended question about gender reassignment. I would ask an open ended question that allowed people to report on Trans issues (or reveal any other sensitive topic they felt was relevant to their work). It might not capture everything, but you’d probably get a better response rate.
Less likely to give the public the opportunity for Carry-On style jokes though.Tweet