It’s simple really, isn’t it? I have evidence about something so what I’m saying is right.
Well the reality is that it depends.
The truth is that we are often more easily seduced by evidence that matches up with either our existing world-view or with what we can view of the world.
An example.
Most people can understand the basics of Newtonian mechanics. Hold a stone and ask what will happen when you drop it. They’ll invariably get that right. Ask them what happens if you drop it from a greater height then they’ll usually know that the object will be faster when it hits the ground. It’s quite easy. These are events they can see and even do themselves. They have experience of the event.
Now start the discussion around what happens to objects travelling at or close to the speed of light. You get a very different result. It’s possible that even Newton would have got it wrong.
So the value of evidence to the observer depends on the position of the observer.
What the above (contrived) example shows is that evidence has parameters within which it is true. Good researchers will be very clear about the limitations of their evidence. Significant parts of any research paper will be devoted to a discussion about these limitations. This is important for any other researchers trying to replicate or extend the work. This discussion very rarely carries over into any public forum where the research is discussed. Evidence is cited in absolute terms without the necessary caveats. This is not to say that there is anything intentionally bad about this. However, when someone is quoting evidence third or fourth or twenty-third hand then it is potentially a problem.
In education the parameters of the research are as important as in any other field. And I think we have a problem here.
I’m the sort of person that when someone quotes evidence at me I go and look it up. What I invariably find with research quoted to support educational preferences is that the research subjects are not school children. More often than not they are undergrads at the university where the researcher resides. Sometimes they are other academics at the same university. This is usually clearly stated not just in the body of the paper but often in the abstract, so the researchers are being very upfront about it.
This is where we have to take care. We know that the brain grows considerably from birth to adulthood. We also know that structural changes occur through this period. We also know that hormone levels that affect memory formation change over this period. Now, I cannot say for certain that this means research carried out on adult learning are not transferable to children, but the evidence suggests that we should be very wary of assuming that they are in the their entirety.
This is a similar approach to that taken in medicine. Very few pharmacological products are licensed (at least initially) for use with children. This is due to ethical considerations about testing such products on them. So assumptions cannot be made either about dosage levels or, because the body of the developing child is different to that of an adult, the actual effect of the drug. Even very common drugs such as aspirin have different side effects in children to those in adults.
We also have to look at cultural differences. If the research happens to be on self-selecting groups of, say, West Coast US undergrads, who have experienced a West Coast US upbringing and a West Coast US education (and diet), are those research results automatically transferrable to, say, Dhaka or Shanghai?
Evidence is important, but it is not absolute, especially within the social sciences.
So, the next time someone quotes the science at you, ask for the parameters.
Distant Ramblings on the Horizon 
Absolutely, Mike, and the answer you get depends on the question you start with, which depends in turn on the view you are trying to explore/get support for.
Often it seems that we test for ‘most efficient’ (i.e. synthetic phonics) and forget that quickest/cheapest/fastest does not necessarily mean best. Perhaps it does, and phonics is a very efficient method backed up by evidence, but it is not the only method for every child to learn to read.
It does worry me that we might pin our hopes too much on a single ‘correct’ approach, when all my experience suggests that a blend according to the individual teacher and the group of children is perhaps better.
I agree that care needs to be taken when using single source evidence. As you say, they often explore only a single route and are not even intended to be used comparatively. Even large scale meta-analyses have to be read in context with an understanding of the limitations of both the original source research and the dangers inherent when bringing them together. Too often such studies are taken as established scientific fact rather than for what they are. Adding together the results of 50 studies carried out on university undergrads still doesn’t tell you a great deal the impact of any effect on 5yr old children. It just gives you a starting point for further investigation.
And there are so many variables. Who’s to know what impact a teaching technique that works well at age 7 will have on a child 10 years later? Or who’s to know what impact their social background has, or the region where they live, or the family group they live in? Or who’s to know how a technique that works for one teacher will work for another? There are so many variables and context is so vital that to me longitudinal studies seem the most sensible approach to gathering evidence and those take time (obviously) and are v. expensive..
[…] All the evidence says | Distant Ramblings on the Horizon […]
Reblogged this on The Echo Chamber.