Skip to content

What happens when journalists are misled about a study (or when they don’t understand research)

February 6th, 2007

Dr Petra

Recently I had a call from a journalist asking for comment on a forthcoming book. It’s a book based on a sex survey of several thousand people. The author, I was told, was a very respected psychotherapist and the work had been completed in conjunction with a television company. They wanted me to give additional quotes around the topic to explain a feature they were writing on the back of this new book.

I said I’d try and help but first I wanted to know the context in which this book had been put together. Had it been completed as a piece of research? Yes, I was told. Did the author carry out the research from within a university or therapy setting, or was it done for a television series. They journalist wasn’t sure but said as the therapist was very well respected they thought it would all be above board. They’d read the book and it definitely was based on research, in fact it was the biggest study of it’s kind ever. It did have a link with a television channel but they didn’t think it had much influence on the research. The book was based on 25000 interviews completed over a two year period which the journalist said were all completed in person by the psychotherapist and some of the surveys took up to 5 hours to finish.

I questioned whether the book was based on surveys or interviews. Whilst the terms can be used interchangeably a survey is a structured range of questions usually with closed ended answers or ratings where people give their views. Most surveys are designed to be as short as possible so I couldn’t understand why they might be taking 5 hours to complete. Interviews can be based around using a structured survey or highly structured questions or could centre on more broad topics of discussion. It might be possible for an interview to take up to five hours although you wouldn’t subject a research participant to an interview of that length, as it’s just too demanding and unethical. Whilst I could see it was a possibility for someone to collect 25000 surveys over the course of two years (at a push), I couldn’t see it was possible for one person to interview 25000 people in depth (and analyse that data) in a two-year period. So I wondered what had been going on in this research. I asked was the research completed online or in person? In person I was told.

The journalist wasn’t sure about how the research for the book had been carried out and didn’t know the difference between survey or interview (which is fair enough, it’s something many social scientists struggle to understand). They said they’d not got the book to hand so they couldn’t tell me any more about it. Anyway, they said, it didn’t matter because they weren’t writing about the book, they were basing their story on the book and they wanted me to give more information to explain their story.

I explained I was happy to help but before I could do that I really needed to know more about the research that had gone into this book since if it wasn’t particularly good I didn’t want to be seen to be endorsing it.

Although I knew I probably wouldn’t get an answer to the question I also asked whether the author of this research had any ethical approval. The journalist had never heard of this. And again that’s fair enough, after all lots of people working outside research won’t know what ethical approval is – in fact many people working in research don’t know much about it either. In a nutshell if you’re going to invite members of the public to be interviewed by you and you are employed as a therapist within an NHS (National Health Service) setting then you have a duty of care which means you need NHS ethics committee approval before you can approach people for research. If you are an independent therapist seeing NHS patients the same applies. If you are doing research within a university setting you’d need ethical approval from your university ethics committee. If you are an independent therapist seeing private patients or someone doing research outside of a university you aren’t under such strict requirements to apply for ethics although good research practice suggests you should.

It can seem very bureaucratic but it is designed to safeguard the public and ensure good standards of research. Of course anybody can decide they want to run a sex survey or interview study and they can invite people to participate. However there are issues about ethical coverage within such research – how do you manage if someone becomes distressed or if they disclose abuse? How do you protect their survey/interview data and store it? What safeguards are there over your skills and abilities as a researcher? Who is responsible if there are any complaints made about you or your work? How can people be sure you’ll represent them fairly and also know how to interpret and report what they tell you? Putting your work through an ethics committee is a means of answering these questions and proving your competencies – and for an independent body to assess that you aren’t going to cause intentional or unintentional harm in your work.

Setting up a survey online is a means some people use to currently get around this issue, although that is changing. Whether or not you run a survey online if you invite people to participate using your role as an academic or therapist you are still bound by the same duties of care and ethical standards whether you’re seeing someone in person or collecting their responses online. Just claiming you’ve designed something that is anonymous isn’t enough to get around not having ethical approval. I didn’t go into all this in-depth with the journalist since they clearly weren’t interested, but I had a number of worries about how this book (and accompanying research) was being promoted to the media where journalists didn’t have the skills to ask critical questions about the method, ethics and robustness of the study.

I could tell I was now irritating the journalist since they obviously wanted some quick quotes for their story and not the third degree on the book their story was based on. They restated the person who had written the book was a ‘well respected psychotherapist’ so would have obviously done everything legally. They repeated to me the book was ‘very important’ since it was based on 25000 interviews. They added the book was being published by a ‘very respectable publishing house’ and had been reviewed in The Observer.

And from a journalist’s perspective these are all things that would make a story stack up and make you trust an expert – they’ve written a book that sounds like it’s included a lot of interviews, comes from a good publisher and has been reviewed in a broadsheet paper.

Unfortunately that doesn’t make the book or the research it was based on good or bad. For all I know the psychotherapist who wrote the book is highly qualified, had searched the scientific literature before putting their research together and had investigated existing studies that would ensure they completed a quality interview/survey. They may well have got ethical approval and saved, managed and stored data from their thousands of interviews in a safe and appropriate way. They may well have analysed all the answers to their research with qualitative analysis (if they used interviews) or with statistical analysis (if they used a survey). Their book may have been painstakingly written with feedback from peers who shaped it into an accurate portrait of sexual experiences in the UK.

But it may equally be that the person concerned applied none of the checks and balances above. That they just threw together some questions and asked the public very intimate questions on sex without any care for their well-being. They may have not analysed their data in any way and they may simply have written a basic account of their findings that adds little or nothing to our knowledge in this area.

Unfortunately a commercial publisher may not know the difference and a review in The Observer isn’t exactly a glowing endorsement of good research – it just means they liked your book. And it’s possible to make a book sound good even if the underlying work it’s based on isn’t all that special.

Having done a bit more digging on the book it seems that a large number of responses were collected from the public via an online survey, not an interview. A subset of participants may have been interviewed in person by the therapist but they would only be those who were prepared to leave their name, contact details and email on the survey website. The ‘survey’ itself wasn’t really a survey but was really a range of open ended questions. Which means the data will need to be analysed qualitatively, rather than quantitatively – although the website suggests that statistical analysis will be done. There is no mention of ethics, protection of participants’ well-being or data protection although the lead researcher (the therapist) is described as working within a clinical setting.

Participants are not told exactly why the research is being conducted or what it will be used for. The questions are useful prompts but aren’t in depth and obviously the study is skewed towards those who a. have access to the Internet and b. are referred to the site in question – so they’re not a representative sample. There are no controls on the site to prevent people submitting several responses – so it’s unclear whether the number of respondents are several thousand uniques or some people posting several times. Finally, although this may not seem important to anyone outside academia, the therapist concerned does not appear to have published anything other than this book. You would expect someone doing research of this magnitude to also share it with the wider scientific community in a peer-reviewed journal – why hasn’t that happened in this case?

Your average journalist sent a press release and free book from a publishing house is not going to go back and check the original survey or ask questions about the method – or probably understand that this particular survey isn’t all that well designed. Which is a shame because the book is going to get a lot of publicity in a few weeks and is already inspiring countless magazine features and television programmes. At the moment I don’t want to say more about the book as it’s not been published, but once it has been published I’ll give a more in-depth analysis of it.

I’ve no problem if someone says ‘look I just happened to be interested in this topic, as an individual I set up a website and I asked some people to tell me about their sex lives’. But I do have a problem if they present to the press that they’ve run one of the biggest academic studies of its kind but don’t appear to have been all that careful with their research.

Whilst I understand that a journalist may not ask as many questions about the research as I have here it wouldn’t hurt to employ such standards to decide what to base features on – particularly if they are something you’re claiming will educate or inform the public (as was the case with this piece). I don’t think it’s unreasonable to expect a journalist who is basing a feature on a book based on research to do a little digging and see if the research is kosher.

This represented a fairly typical crossed-purposes situation between the journalist and myself. As a sex researcher I obviously wanted to know about the research this book was based on and without answers to my questions I couldn’t be sure it was okay – and I didn’t want to comment on it until I knew more. From the journalist’s perspective they wanted some quick quotes to back up their story which was based on a book and they didn’t care whether the book was based on ethical research – in fact they felt it was obviously a ‘good book’ on account of the publisher, book review, author credentials and large survey/interview the text was based on.

I did offer to comment for the feature but since it was for a magazine that presents itself as sharing cutting edge social science I didn’t feel comfortable giving feedback without knowing more about the book. I suggested the journalist might check my concerns with a few questions to the book’s author. ‘I think we’ll leave it there’ they replied and hung up on me.

So the feature will get written about a book that may or may not be based on any reliable research. The journalist won’t probably be checking whether the book is okay and no doubt they will get someone else to provide supporting quotes for their story. The public will read the feature and it may well interest them or make them feel their sex lives should happen in a particular way, and they’ll be given the impression because the book is used in a feature for a magazine that it’s based on reliable research.

Comments are closed.