This content is locked. Please login or become a member.
Why Context Matters
Evidence is only evidence in one particular setting. So even if you had cast-iron evidence that Tom killed Sarah and Tom was the husband, this doesn’t mean that in every setting in which a woman dies, it’s always the husband that did it. It’s only true in that particular case. But a proof is universal. When Archimedes proved that the area of a circle is pi times the radius squared, he proved that not only for circles in ancient Greece in the third century BC, but for circles all around the world today. But often what we do with evidence is we take evidence gathered in one particular setting and claim that applies universally.
A Case Study in Overextrapolating
The best-selling management book in the first half of the twentieth century was looking at the practice of scientific management, which is you break down an individual task into small micro-tasks, you measure each task, and you pay somebody for output. So if you are doing the process of shoveling pig iron, you break this down into how much to lift at one particular time, how to hold a shovel, how long to rest for, and then you count the amount of output, the amount of pig iron lifted, and you pay the worker for the output at the end of the day. Now that was extremely successful in pig iron handling. In some cases, it actually quadrupled people’s output. It didn’t just double it. But the problem is that people over-extrapolated from this. They thought it was a universal proof, and they thought that the way to increase productivity in any setting was to scientifically analyze this. And this was something that was applied to the US education system under the auspices of the No Child Left Behind Act.
So what this did is it broke down education into scripted curriculum. It told teachers exactly what to say, in which lesson, how to teach people how to spell, exactly what words to teach the kids to spell. And it also had standardized test scores to try to break down the effectiveness of a teacher in terms of their students’ performance on these tests, similar to how you would measure the output of a pig iron worker. But this is crazy because teaching and pig iron handling are very different contexts. So there might be one best way to lift pig iron, but for teaching, there’s lots of different ways. Different teachers have different styles. Different pupils might be responsive to different learning methodologies. And similarly, you cannot measure the success of a teacher with one single output like you can with the amount of iron lifted. It may well be that what you seek from a teacher is not just teaching to the test, but critical thinking, a love of learning, and a respect for authority. And this is why No Child Left Behind was criticized so much by both Republicans and Democrats that it was eventually abandoned.
But the problem here was nothing to do with the original studies on scientific management being flimsy, mistaken correlation for causation, and the like. Those studies were correct, but they were over extrapolated beyond the initial context. People thought this was a universal proof. When it was not, it was a context-specific result.
Checking for Universality
So how do we know whether we have evidence or proof? So first, we will look at whether the study was undertaken in the same context that we actually care about. Now we won’t always have a perfect study in exactly the same setting, but we can ask ourselves, well, how similar are the settings? So if the study was on pig iron handling and we are looking at another setting where there’s a single output and one best way, such as manufacturing, that we might think is close enough that we can apply the findings of the benefits of scientific management. But sometimes, it might be that the setting that we care about, such as teaching, is sufficiently different. But then how do we define what is sufficiently different? Well, this is where a lot of common sense and judgment comes into play. Discerning misinformation is not just about statistical pyrotechnics, but common sense and discernment.