Recently a UK morning television show hosted an online poll about breastfeeding. Not a problem you might think. Breastfeeding is an important and topical issue and one that often creates anxiety for mothers. So finding out what women’s experiences are of breastfeeding is important.
However, if you are going to do this you also need to ask questions in a fair and unambiguous way. That’s just basic good survey design. If you ask questions in a leading, biased or confused way you’ll either find people won’t respond to you, or those who do won’t be representative so any resulting data is meaningless.
Unfortunately it seems GMTV hadn’t gone to the ‘survey 101’ class and posted a poll that included questions like:
“Do you think breasts should not be displayed in public for any reason?”
“Do you think women should use discretion when breastfeeding?”
“Do you think it’s a woman’s right to breastfeed in public?”
It also apparently asked whether it was “wrong” to breastfeed a child over 12 months.
These questions are a problem because they’re leading and negative. If you wanted to find out women’s views and experiences of breastfeeding (good and bad) you could ask them about where they feed, or whether they’ve encountered support or barriers when feeding at home or in public. You might ask them who gave them advice about breastfeeding and whether that was helpful (and if not why not), and focus on key issues like family/partner support.
You’d want to be sure anything you asked delivered a clear answer. For example for the question ‘is it wrong to breastfeed a child aged over 12 months’ someone might answer yes but meaning they felt it was wrong to feed an older child in public, but were not averse to the child being fed at home. Or they might answer yes to the question not believing it’s ‘wrong’ to feed a child over 12 months in public, just that they would feel uncomfortable doing so. Or their understanding of ‘wrong’ related to exclusively breastfeeding a 12 month old (although they might be fine with occasional feeds to a little one). With vaguely worded yet judgemental questions you can never be sure what someone’s responding to.
That’s not to say you can’t ask provocative questions, but you would do it in a way that was clear that was your aim. So you could say ‘some people believe the time to stop breast feeding is when baby is 12 months old?’ and ask people to agree or disagree with that statement. It is suggesting there’s a cut off time for feeding, but not that feeding or not over a particular age band is right or wrong.
Indeed good quality surveys are characterised by the fact that they can ask sensitive questions but in a way that does not suggest blame or moralising. Good social research is about finding out about peoples views, not imposing views on them or making them feel their beliefs are faulty.
Usually it’s just geeky social science types like me who get in a flap when dubious surveys hit the media, but on this occasion members of the website Mumsnet also noticed what was going on. As you can see from the linked thread members of the site were upset about the wording of the questions which they felt devalued breastfeeding and mothers who chose to do it. The discussion also alleged the reason for the dubiously worded survey was down to the show being sponsored by Nestle who manufacture baby milk.
Interestingly this story then hit the headlines (albeit more with a focus on the perceived anti breastfeeding angle rather than the duff survey design). However this is an important step forward in campaigning around shoddy surveys used within media as if the public notice and speak up against biased surveys we may be able to challenge more of their widespread use.
It’s clearly difficult to show a link between the dodgy GMTV survey and the show’s advertisers (Nestle claim they had no influence on the survey). So leaving aside the possibility that the advertisers played some role, why else might this situation have arisen?
Within media it’s very common to use surveys. These can be used as promotional tools to plug a magazine or television show, or to create content or a peg to hang a story on.
While there is huge amounts of data that exists that could inform stories (more on this shortly) most journalists assume to make their story/programme ‘unique’ they have to create a brand new survey to inform later content. Which might be an article using data/quotes gleaned from a survey, or a discussion based radio or television show based around your survey results.
My hunch is this was simply the case of a journalist writing down a few questions and getting their producer to approve them. Possibly sharing them with colleagues in the office but certainly not reflecting on them, sharing ideas about the question tone, or piloting in any way.
Generally journalists aren’t aware of the amount of work that needs to go into designing, delivering and analysing a survey, so a back of the envelope sketch of a few questions is enough to run with.
It’s a classic example of poor journalistic practice as not only are the surveys created usually slipshod and meaningless, they’re also unethical since they may well upset participants (as in this case) or mislead the public on key health/social issues.
Moreover it’s bad practice because it shows no research has been done. For pretty much any topic on health or social issues there will be a body of research investigating that. Studies that can be found if you search using Scholar or PubMed. In the case of breastfeeding not only are there countless surveys on attitudes towards breastfeeding, experiences of breastfeeding, feeding duration, partner support, impact on relationships, there are even journals specifically dedicated to lactation. So it’s not exactly an area where you couldn’t quickly find numerous existing and well designed surveys that could inform a programme. As an example here’s what comes up when you put in the very basic searches attitudes towards breastfeeding and breastfeeding experiences.
Sadly many journalists appear unaware these resources exist, or unsure how to search and find information. When I speak to journalists about such resources they often tell me they couldn’t use any pre-existing surveys because they have to design a ‘new’ one in order for it to be fresh for their programming. There is no sense that the very purpose of existing published research is to inform and underpin education and practice.
This is worrying in terms of practice, but also depressing since evidence is published in order to inform public understanding and practice. It is remiss not to make use of such information that could still provide a talking point/background for a story or programme, but would ensure what was discussed was accurate and based on the experiences of a wide range of people.
My guess is that GMTV hosted a sloppy survey for reasons set out above. However there is really no excuse for this practice to continue. They have had a very public warning that such activities are unacceptable.
More than that, I was contacted by GMTV last week who are planning a forthcoming series of programmes. They asked if I might be involved and told me they were basing the whole event around a survey. I explained I’d be delighted to be involved but I’d need to see the survey first. I was told the survey was based on a few questions they’d written down but they did agree to send me them and in turn I gave detailed feedback on how to reword the questions and add more meaningful ones.
To date I’ve heard nothing back so I can either assume they’re busy firefighting the breastfeeding debacle, never got my emails, or my approach to trying to ensure their survey was correct was not welcome and they’ve found another expert (perhaps one who doesn’t ask difficult questions about surveys). Experience tells me it’ll be the latter, but I’m happy to be proved wrong.
If I do hear anything I will let you know. Given this barrage of complaint against the programme and an offer from at least one experienced social scientist to help improve their survey design one might assume GMTV would be keen to get it right in future.
Let’s see what happens. And in the meantime let’s continue to name, shame and campaign each time we spot a shoddy survey.
Related posts
Journalists who run bad and biased surveys beware! Your audience is starting to speak out against shoddy practice
Recently a UK morning television show hosted an online poll about breastfeeding. Not a problem you might think. Breastfeeding is an important and topical issue and one that often creates anxiety for mothers. So finding out what women’s experiences are of breastfeeding is important.
However, if you are going to do this you also need to ask questions in a fair and unambiguous way. That’s just basic good survey design. If you ask questions in a leading, biased or confused way you’ll either find people won’t respond to you, or those who do won’t be representative so any resulting data is meaningless.
Unfortunately it seems GMTV hadn’t gone to the ‘survey 101’ class and posted a poll that included questions like:
“Do you think breasts should not be displayed in public for any reason?”
“Do you think women should use discretion when breastfeeding?”
“Do you think it’s a woman’s right to breastfeed in public?”
It also apparently asked whether it was “wrong” to breastfeed a child over 12 months.
These questions are a problem because they’re leading and negative. If you wanted to find out women’s views and experiences of breastfeeding (good and bad) you could ask them about where they feed, or whether they’ve encountered support or barriers when feeding at home or in public. You might ask them who gave them advice about breastfeeding and whether that was helpful (and if not why not), and focus on key issues like family/partner support.
You’d want to be sure anything you asked delivered a clear answer. For example for the question ‘is it wrong to breastfeed a child aged over 12 months’ someone might answer yes but meaning they felt it was wrong to feed an older child in public, but were not averse to the child being fed at home. Or they might answer yes to the question not believing it’s ‘wrong’ to feed a child over 12 months in public, just that they would feel uncomfortable doing so. Or their understanding of ‘wrong’ related to exclusively breastfeeding a 12 month old (although they might be fine with occasional feeds to a little one). With vaguely worded yet judgemental questions you can never be sure what someone’s responding to.
That’s not to say you can’t ask provocative questions, but you would do it in a way that was clear that was your aim. So you could say ‘some people believe the time to stop breast feeding is when baby is 12 months old?’ and ask people to agree or disagree with that statement. It is suggesting there’s a cut off time for feeding, but not that feeding or not over a particular age band is right or wrong.
Indeed good quality surveys are characterised by the fact that they can ask sensitive questions but in a way that does not suggest blame or moralising. Good social research is about finding out about peoples views, not imposing views on them or making them feel their beliefs are faulty.
Usually it’s just geeky social science types like me who get in a flap when dubious surveys hit the media, but on this occasion members of the website Mumsnet also noticed what was going on. As you can see from the linked thread members of the site were upset about the wording of the questions which they felt devalued breastfeeding and mothers who chose to do it. The discussion also alleged the reason for the dubiously worded survey was down to the show being sponsored by Nestle who manufacture baby milk.
Interestingly this story then hit the headlines (albeit more with a focus on the perceived anti breastfeeding angle rather than the duff survey design). However this is an important step forward in campaigning around shoddy surveys used within media as if the public notice and speak up against biased surveys we may be able to challenge more of their widespread use.
It’s clearly difficult to show a link between the dodgy GMTV survey and the show’s advertisers (Nestle claim they had no influence on the survey). So leaving aside the possibility that the advertisers played some role, why else might this situation have arisen?
Within media it’s very common to use surveys. These can be used as promotional tools to plug a magazine or television show, or to create content or a peg to hang a story on.
While there is huge amounts of data that exists that could inform stories (more on this shortly) most journalists assume to make their story/programme ‘unique’ they have to create a brand new survey to inform later content. Which might be an article using data/quotes gleaned from a survey, or a discussion based radio or television show based around your survey results.
My hunch is this was simply the case of a journalist writing down a few questions and getting their producer to approve them. Possibly sharing them with colleagues in the office but certainly not reflecting on them, sharing ideas about the question tone, or piloting in any way.
Generally journalists aren’t aware of the amount of work that needs to go into designing, delivering and analysing a survey, so a back of the envelope sketch of a few questions is enough to run with.
It’s a classic example of poor journalistic practice as not only are the surveys created usually slipshod and meaningless, they’re also unethical since they may well upset participants (as in this case) or mislead the public on key health/social issues.
Moreover it’s bad practice because it shows no research has been done. For pretty much any topic on health or social issues there will be a body of research investigating that. Studies that can be found if you search using Scholar or PubMed. In the case of breastfeeding not only are there countless surveys on attitudes towards breastfeeding, experiences of breastfeeding, feeding duration, partner support, impact on relationships, there are even journals specifically dedicated to lactation. So it’s not exactly an area where you couldn’t quickly find numerous existing and well designed surveys that could inform a programme. As an example here’s what comes up when you put in the very basic searches attitudes towards breastfeeding and breastfeeding experiences.
Sadly many journalists appear unaware these resources exist, or unsure how to search and find information. When I speak to journalists about such resources they often tell me they couldn’t use any pre-existing surveys because they have to design a ‘new’ one in order for it to be fresh for their programming. There is no sense that the very purpose of existing published research is to inform and underpin education and practice.
This is worrying in terms of practice, but also depressing since evidence is published in order to inform public understanding and practice. It is remiss not to make use of such information that could still provide a talking point/background for a story or programme, but would ensure what was discussed was accurate and based on the experiences of a wide range of people.
My guess is that GMTV hosted a sloppy survey for reasons set out above. However there is really no excuse for this practice to continue. They have had a very public warning that such activities are unacceptable.
More than that, I was contacted by GMTV last week who are planning a forthcoming series of programmes. They asked if I might be involved and told me they were basing the whole event around a survey. I explained I’d be delighted to be involved but I’d need to see the survey first. I was told the survey was based on a few questions they’d written down but they did agree to send me them and in turn I gave detailed feedback on how to reword the questions and add more meaningful ones.
To date I’ve heard nothing back so I can either assume they’re busy firefighting the breastfeeding debacle, never got my emails, or my approach to trying to ensure their survey was correct was not welcome and they’ve found another expert (perhaps one who doesn’t ask difficult questions about surveys). Experience tells me it’ll be the latter, but I’m happy to be proved wrong.
If I do hear anything I will let you know. Given this barrage of complaint against the programme and an offer from at least one experienced social scientist to help improve their survey design one might assume GMTV would be keen to get it right in future.
Let’s see what happens. And in the meantime let’s continue to name, shame and campaign each time we spot a shoddy survey.
Journalists who run bad and biased surveys beware! Your audience is starting to speak out against shoddy practiceRelated posts