Yesterday I wrote about the case of UK morning television show GMTV who’d got into hot water after hosting a biased survey about breastfeeding that had upset mums a lot. It resulted in the programme being publicly criticised for their actions.
I mentioned how I’d been approached by GMTV about a planned series on sex, which they wanted to base on a sex survey. I agreed in principle to take part, with the proviso I had some input to the survey. They sent me their draft questions and I gave feedback. I’ve not heard anything back from them (not even an acknowledgement they received my message), but looking at their sex survey (which is now live on their site) it seems they have listened to the feedback I provided.
Their original list of questions wasn’t too bad, although as with any survey they required reflection and change. For example they planned to ask ‘do you have a partner?’, but this is often misleading and folk who are either dating or married assume it doesn’t apply to them. So a better term (which I recommended) is ‘are you in a relationship?’ They also wanted to find out ‘is your sex life good’ which I suggested they change as ‘good’ is a difficult concept to measure (my standards for ‘good’ may be much more stringent than yours). Instead I recommended ‘how satisfied are you with your relationship?’ (and added asking about partner satisfaction was equally important). I recommended they drop questions that are difficult to reliably measure through a survey (like ‘how often do you think about sex?’) along with questions relating to ‘meeting a stranger online for sex’ (which seemed at odds with the rest of the questions and judgemental).
Based on their suggestions my rewrite looked something like this – aimed to be very basic and to get some general data that could be compared with existing studies and used as a talking point for their programme.
- Demographics: age, gender, location (I usually don’t start a survey like this as demographics are quite threatening, but given the rest of the questions were sensitive this wasn’t a bad place to begin).
- Are you in a relationship? If yes, how long have you been together?
- Are you happy with your sex life?
- Is sex important to your relationship?
- What things help you enjoy sex? (I suggested this might remain open ended or give multiple choices such as communication, willingness to explore new things, time to spend together, trusting each other)
- What are the main barriers to enjoying sex? (worries about body image, confidence issues, lack of partner support, relationship problems, health problems, fertility worries, tiredness, lack of privacy/kids in
the way, sleeping in separate beds)
- Do you have any questions about sex you would like us to answer? (open ended)
Looking at their live survey it seems many of these pointers have been taken on board. These include:
6. Are you happy with your sex life? Yes/No
7. Do you think your partner is happy with your sex life? Yes/No
8. Is sex important to your relationship? Yes/No
9. What would help you enjoy sex more?
Good communication
Willingness to explore new things
Spending time together
Trusting each other
I’m happy with our sex life
11. What prevents you having more sex?
Kids
Tiredness
Poor body image
Health problems
Partner’s sexual difficulties
Relationship problems
Sleeping in separate beds
I’m happy with how often we have sex
12. What things enhance sex for you?
Time together
Good communication
Confidence
Help around the house / childcare
13. Do you still find your partner attractive? Yes/No
If I were doing this study myself I’d have more space to talk about positive things that enhance sex and I’d combine questions 9 and 12. I’d allow more scope for participants to share what does/doesn’t work for them and I’d focus on relationships and intimacy as much as sex. However, the inclusion of factors that get in the way of desire which are very practical for most women (lack of time/privacy due to kids being around, body image etc) is important and will allow for a good discussion for the programme.
However, there are still major problems with the survey and the first one you spot on the welcome page which says “Valentine’s Day is upon us, but are you fed up with your sex life? Is your partner inattentive and unadventurous? Or was the last sexual compulsion you experienced many, many moons ago? Maybe you’re lucky enough to enjoy a passionate relationship with your partner. Whatever you’re getting up to between the sheets, share your sex stories and thoughts with us in our sex survey”.
Yes, it’s in media speak, but there are problems here around the aim of the study – is it about Valentine’s day or sex? It’s leading from the outset with the first statement asking whether people are fed up, or if their partner is inattentive or unadventurous, or sex is something that’s not happened for years. It’s hideously sex negative and likely to invite people who’re either having problems with their sex lives or want to prove they’re super sexual. It’s very bad practice to begin a sex survey with statements like this and no clear aims, but it’s another example of how little attention people give to even basic things like the way you begin your survey and invite participation. Sadly, this opening to the survey suggests it is likely to be as problematic and biased as the previously flawed one on breastfeeding.
At the close of the survey there are suddenly several questions about porn and paying for sex, which don’t really fit with the rest of the questions asked:
14. Do you use pornography? Yes/No
15. Does your partner use pornography? Yes/No
16. Would you ever pay for sex? Yes/No
This question is optional.
These are flawed questions because it’s unclear what does ‘use’ pornography mean? ‘Use’ as in something to turn you on during sex, or before sex with a partner? ‘Use’ to give you ideas for sex? ‘Use’ for masturbation on your own?
Not only do these questions not measure porn ‘use’ (because of a lack of explanation in the questions) but they also don’t tell us useful information like whether women are happy or not about porn or their partner looking at it. It’s almost as though this was tagged on at the end because porn might come up as part of the planned series on sex so there had to be some data on it. Because there wasn’t any thought about this (or using existing survey questions on porn, which again there are loads of) these are just wasted questions.
The buying sex one is also not clear in terms of why it’s being asked (on a reputable survey you’d always ask why you want someone to tell you something). Making the question optional is good, but it’s still not going to tell us anything more than whether people think they might consider buying sex, not whether they have and what their attitudes to paying for sex are. Because this question is placed above asking people for their name and a daytime contact number it makes the porn/paying for sex questions even more threatening – and less likely to be answered.
It’s a shame the introduction to the questionnaire and closing questions are so flawed as the questions in between aren’t too bad. You can easily see, however, that by not taking care throughout the research process, that you can unduly influence and potentially scupper an otherwise good piece of work.
As mentioned in yesterday’s blog, for many journalists (and for most people who’re novice survey designers) it’s common to think if you throw down a few questions on the back of an envelope your survey is good to go. I think from looking over this current survey that there’s an additional problem I often encounter when teaching survey design. Which is folk grasp they need feedback, but assume if they get a little bit then that’s the green light to run with a piece of work.
In fact, to make a good survey (and this applies whatever it is for) you need to find out whether other surveys already exist on the topic you’re interested in. Use or adapt them for your own study. Get feedback on them from colleagues. Amend the questionnaire accordingly. Pilot on a representative subsample. Amend as needed. Then proceed with your work.
This can be more time consuming than bunging down a few questions, but if you liaise with an expert it shouldn’t take long. And it means what you do produce is reliable and useful and far less likely to risk criticism or non completion.
I’m glad to see GMTV appear to have taken feedback on board, although sad to see that by not fully understanding how surveys work that errors in their existing sex survey means it’s not very reliable. Of course they will be able to use it to inform discussions, but the outcomes they get may not be all that representative and thus conversations may be skewed and not deliver what viewers really need.
After writing yesterday several journalists emailed me and asked what my tips were for them to create good surveys for use in features or programmes. I’d recommend following the steps for good survey design outlined above and also:
- get experts to help (they can tap into questionnaires quicker than you can)
- don’t be afraid to use existing measures
- don’t base programmes/features on a survey you’ve designed without going through steps outlined above (if you don’t have time commission someone to do this for you)
- may be better to use the findings from another survey as a talking point
- don’t make the mistake of seeing a survey as quick and easy
- if you get help and feedback then have a dialogue about it (don’t assume feedback equals a green light to proceed unless specifically stated)
People like myself should be willing to work more closely with journalists to improve practice around surveys – which might include training on how surveys work or survey design/analysis, or assistance with writing and disseminating surveys. I see it as a duty of care to make existing measures more accessible and to help journalists put together good quality surveys.
However, there is a problem around offering such a service. Who pays for it? Is it something journalists will expect for free? Is it part of wider public engagement duties we ought to be providing? If so do our institutions cover that cost or the media outlet we’re working with? What do we do if our advice isn’t heeded? How much time should we give to such ventures? And how do we measure ’success’?
For example I managed to get some questions changed from the original GMTV list. Or so it seems. I’ve had no further contact with them to confirm this (I can only go by what they originally sent, what I suggested, and seeing my suggested changed questions now appear on their website). So you could judge that a success. However, responding took time and you have to ask yourself is it worth providing such feedback if you’ve no guarantee the overall quality of a survey would be good (or represent the standards you’d expect in your usual practice). Is it worth doing if you get no feedback whatsoever? Or with no guarantees you’d be able to appear on television (or in print) to talk about the survey?
I’d really be interested to hear from journalists or health/social scientists/practitioners about this topic so please email me your thoughts.
Journalists – I’d love to know if you are interested in getting training on using/understanding surveys, or feedback on creating individual surveys well as the need arises. What kind of support (if any) are you bothered about?
Researchers/practitioners – I’m keen to hear what your experiences in dealing with the media around research have been. Do you think we should do more to encourage collaborative working or training? If not, why not? If so, how might this relationship be encouraged to avoid exploitation and overcome poor practice?
And to everyone reading, is it right to expose bad research in the way I’ve done in yesterday or today’s blog? Or best to keep discussions about research private?
I’ll feed back any comments in a future blog.
Related posts
TV company make some changes to a sex survey, but not enough to make it useful and unbiased
Yesterday I wrote about the case of UK morning television show GMTV who’d got into hot water after hosting a biased survey about breastfeeding that had upset mums a lot. It resulted in the programme being publicly criticised for their actions.
I mentioned how I’d been approached by GMTV about a planned series on sex, which they wanted to base on a sex survey. I agreed in principle to take part, with the proviso I had some input to the survey. They sent me their draft questions and I gave feedback. I’ve not heard anything back from them (not even an acknowledgement they received my message), but looking at their sex survey (which is now live on their site) it seems they have listened to the feedback I provided.
Their original list of questions wasn’t too bad, although as with any survey they required reflection and change. For example they planned to ask ‘do you have a partner?’, but this is often misleading and folk who are either dating or married assume it doesn’t apply to them. So a better term (which I recommended) is ‘are you in a relationship?’ They also wanted to find out ‘is your sex life good’ which I suggested they change as ‘good’ is a difficult concept to measure (my standards for ‘good’ may be much more stringent than yours). Instead I recommended ‘how satisfied are you with your relationship?’ (and added asking about partner satisfaction was equally important). I recommended they drop questions that are difficult to reliably measure through a survey (like ‘how often do you think about sex?’) along with questions relating to ‘meeting a stranger online for sex’ (which seemed at odds with the rest of the questions and judgemental).
Based on their suggestions my rewrite looked something like this – aimed to be very basic and to get some general data that could be compared with existing studies and used as a talking point for their programme.
- Demographics: age, gender, location (I usually don’t start a survey like this as demographics are quite threatening, but given the rest of the questions were sensitive this wasn’t a bad place to begin).
- Are you in a relationship? If yes, how long have you been together?
- Are you happy with your sex life?
- Is sex important to your relationship?
- What things help you enjoy sex? (I suggested this might remain open ended or give multiple choices such as communication, willingness to explore new things, time to spend together, trusting each other)
- What are the main barriers to enjoying sex? (worries about body image, confidence issues, lack of partner support, relationship problems, health problems, fertility worries, tiredness, lack of privacy/kids in
the way, sleeping in separate beds)
- Do you have any questions about sex you would like us to answer? (open ended)
Looking at their live survey it seems many of these pointers have been taken on board. These include:
6. Are you happy with your sex life? Yes/No
7. Do you think your partner is happy with your sex life? Yes/No
8. Is sex important to your relationship? Yes/No
9. What would help you enjoy sex more?
Good communication
Willingness to explore new things
Spending time together
Trusting each other
I’m happy with our sex life
11. What prevents you having more sex?
Kids
Tiredness
Poor body image
Health problems
Partner’s sexual difficulties
Relationship problems
Sleeping in separate beds
I’m happy with how often we have sex
12. What things enhance sex for you?
Time together
Good communication
Confidence
Help around the house / childcare
13. Do you still find your partner attractive? Yes/No
If I were doing this study myself I’d have more space to talk about positive things that enhance sex and I’d combine questions 9 and 12. I’d allow more scope for participants to share what does/doesn’t work for them and I’d focus on relationships and intimacy as much as sex. However, the inclusion of factors that get in the way of desire which are very practical for most women (lack of time/privacy due to kids being around, body image etc) is important and will allow for a good discussion for the programme.
However, there are still major problems with the survey and the first one you spot on the welcome page which says “Valentine’s Day is upon us, but are you fed up with your sex life? Is your partner inattentive and unadventurous? Or was the last sexual compulsion you experienced many, many moons ago? Maybe you’re lucky enough to enjoy a passionate relationship with your partner. Whatever you’re getting up to between the sheets, share your sex stories and thoughts with us in our sex survey”.
Yes, it’s in media speak, but there are problems here around the aim of the study – is it about Valentine’s day or sex? It’s leading from the outset with the first statement asking whether people are fed up, or if their partner is inattentive or unadventurous, or sex is something that’s not happened for years. It’s hideously sex negative and likely to invite people who’re either having problems with their sex lives or want to prove they’re super sexual. It’s very bad practice to begin a sex survey with statements like this and no clear aims, but it’s another example of how little attention people give to even basic things like the way you begin your survey and invite participation. Sadly, this opening to the survey suggests it is likely to be as problematic and biased as the previously flawed one on breastfeeding.
At the close of the survey there are suddenly several questions about porn and paying for sex, which don’t really fit with the rest of the questions asked:
14. Do you use pornography? Yes/No
15. Does your partner use pornography? Yes/No
16. Would you ever pay for sex? Yes/No
This question is optional.
These are flawed questions because it’s unclear what does ‘use’ pornography mean? ‘Use’ as in something to turn you on during sex, or before sex with a partner? ‘Use’ to give you ideas for sex? ‘Use’ for masturbation on your own?
Not only do these questions not measure porn ‘use’ (because of a lack of explanation in the questions) but they also don’t tell us useful information like whether women are happy or not about porn or their partner looking at it. It’s almost as though this was tagged on at the end because porn might come up as part of the planned series on sex so there had to be some data on it. Because there wasn’t any thought about this (or using existing survey questions on porn, which again there are loads of) these are just wasted questions.
The buying sex one is also not clear in terms of why it’s being asked (on a reputable survey you’d always ask why you want someone to tell you something). Making the question optional is good, but it’s still not going to tell us anything more than whether people think they might consider buying sex, not whether they have and what their attitudes to paying for sex are. Because this question is placed above asking people for their name and a daytime contact number it makes the porn/paying for sex questions even more threatening – and less likely to be answered.
It’s a shame the introduction to the questionnaire and closing questions are so flawed as the questions in between aren’t too bad. You can easily see, however, that by not taking care throughout the research process, that you can unduly influence and potentially scupper an otherwise good piece of work.
As mentioned in yesterday’s blog, for many journalists (and for most people who’re novice survey designers) it’s common to think if you throw down a few questions on the back of an envelope your survey is good to go. I think from looking over this current survey that there’s an additional problem I often encounter when teaching survey design. Which is folk grasp they need feedback, but assume if they get a little bit then that’s the green light to run with a piece of work.
In fact, to make a good survey (and this applies whatever it is for) you need to find out whether other surveys already exist on the topic you’re interested in. Use or adapt them for your own study. Get feedback on them from colleagues. Amend the questionnaire accordingly. Pilot on a representative subsample. Amend as needed. Then proceed with your work.
This can be more time consuming than bunging down a few questions, but if you liaise with an expert it shouldn’t take long. And it means what you do produce is reliable and useful and far less likely to risk criticism or non completion.
I’m glad to see GMTV appear to have taken feedback on board, although sad to see that by not fully understanding how surveys work that errors in their existing sex survey means it’s not very reliable. Of course they will be able to use it to inform discussions, but the outcomes they get may not be all that representative and thus conversations may be skewed and not deliver what viewers really need.
After writing yesterday several journalists emailed me and asked what my tips were for them to create good surveys for use in features or programmes. I’d recommend following the steps for good survey design outlined above and also:
- get experts to help (they can tap into questionnaires quicker than you can)
- don’t be afraid to use existing measures
- don’t base programmes/features on a survey you’ve designed without going through steps outlined above (if you don’t have time commission someone to do this for you)
- may be better to use the findings from another survey as a talking point
- don’t make the mistake of seeing a survey as quick and easy
- if you get help and feedback then have a dialogue about it (don’t assume feedback equals a green light to proceed unless specifically stated)
People like myself should be willing to work more closely with journalists to improve practice around surveys – which might include training on how surveys work or survey design/analysis, or assistance with writing and disseminating surveys. I see it as a duty of care to make existing measures more accessible and to help journalists put together good quality surveys.
However, there is a problem around offering such a service. Who pays for it? Is it something journalists will expect for free? Is it part of wider public engagement duties we ought to be providing? If so do our institutions cover that cost or the media outlet we’re working with? What do we do if our advice isn’t heeded? How much time should we give to such ventures? And how do we measure ’success’?
For example I managed to get some questions changed from the original GMTV list. Or so it seems. I’ve had no further contact with them to confirm this (I can only go by what they originally sent, what I suggested, and seeing my suggested changed questions now appear on their website). So you could judge that a success. However, responding took time and you have to ask yourself is it worth providing such feedback if you’ve no guarantee the overall quality of a survey would be good (or represent the standards you’d expect in your usual practice). Is it worth doing if you get no feedback whatsoever? Or with no guarantees you’d be able to appear on television (or in print) to talk about the survey?
I’d really be interested to hear from journalists or health/social scientists/practitioners about this topic so please email me your thoughts.
Journalists – I’d love to know if you are interested in getting training on using/understanding surveys, or feedback on creating individual surveys well as the need arises. What kind of support (if any) are you bothered about?
Researchers/practitioners – I’m keen to hear what your experiences in dealing with the media around research have been. Do you think we should do more to encourage collaborative working or training? If not, why not? If so, how might this relationship be encouraged to avoid exploitation and overcome poor practice?
And to everyone reading, is it right to expose bad research in the way I’ve done in yesterday or today’s blog? Or best to keep discussions about research private?
I’ll feed back any comments in a future blog.
TV company make some changes to a sex survey, but not enough to make it useful and unbiasedRelated posts