There are a number of dimensions I view research across to help me understand the value of a particular piece. I claim no scientific underpinning for the way I do this, all I can say is that it works for me and over the long term has put me a place where most of my views are supported by a body of reliable evidence.
When I see a piece of research quoted in the press the following thought processes kick into play:
Does this sound right?
By this I mean does it gel with my own understanding of the area under study? Does it agree with, or conflict with my own personal experiences, and those experiences I have first hand knowledge of? Applied alone this would obviously be a very bad test. What it does is sets up the level of scepticism with which I will read the actual research. Why does this help me? Well, if it’s a piece that resonates with my experiences then I’m going to want to make sure the claims being made are properly supported, that I can rely on them. If it doesn’t resonate then I’m going to want to see where I think its got it wrong. I understand those two approaches are different but each ensures I look at the information with some rigour.
Where is the original paper?
If I can’t read the original then it’s not research it’s a newspaper article. I’ve read enough newspaper reports of particularly education-based research to know that the interpretation is not always, say we say, rigorous. There are a few exceptions to this rule, but it is only a few.
Who has produced this?
Yes, I know, who produced the research should be irrelevant. But really, it’s not. If I were to produce a paper claiming the Higgs Boson didn’t actually exist then that would (and should) be viewed in an entirely different light than if CERN produced the same paper.
What does the method tell us?
What’s the sample size? Where was the research conducted? Who were the subjects? I see a lot of research carried out on US college students which then leads to claims being made about what will happen in school classrooms in Plymouth. The context is very different, the people are very different, the culture is very different. The research may well be rigorous and the outcomes make sense in context, but they aren’t necessarily transferable. I also try to keep an eye on the sample sizes. Small sample sizes aren’t always an indicator of ‘dodgy” research but they do usually speak to the transferability of the research out of its original context.
Is this replicable?
This is a tricky one. The keystone of research in the physical sciences is replicability. In the social sciences this is not always entirely possible. I like to consider the question – could this be replicated in a context I know nd what would be the likely outcome? Can I imagine outcomes contrary to those in the actual study? Obviously the issues of bias come into play here – this depends on me being honest with myself.
Is there a viable causal mechanism?
This is the important one. A lot of social science research tells what is happening, but cannot always explain why. Given that it’s a relevant piece of research from a reliable source can I see a rational explanation of the outcomes? To consider a topical example, can I see a reason for mid-teen girls displaying more symptoms of depression that mid-teen boys? Having one of each in the house the answer to that has to be yes. There are a wider range of pressures on girls than boys. It is logical to assume that the wider the range of pressures then the more likely the impact. An alternative hypothesis could be that greater to exposure to more diverse pressures might ‘inoculate’ one against the effect, but that doesn’t seem as likely to me. Of course, this could also just be my own biases creeping in again. But in this case I don’t think so. Every parent, every headteacher, every mental health professional I speak to tells me they see the same effects.
No one of the above though processes can ensure the odd renegade paper can’t seep through the cracks, but when they are all applied it means being taken in by such a thing is much less likely.