24.9.12

Your Scientific Reasoning Is More Flawed Than You Think: Scientific American

Your Scientific Reasoning Is More Flawed Than You Think: Scientific American


New concepts don’t replace incorrect ones: they just learn to live together
By Jason Castro

In one sense, science educators have it easy. The things they describe are so intrinsically odd and interesting — invisible fields, molecular machines, principles explaining the unity of life and origins of the cosmos — that much of the pedagogical attention-getting is built right in.  Where they have it tough, though, is in having to combat an especially resilient form of higher ed’s nemesis: the aptly named (if irredeemably clichéd) ‘preconceived idea.’ Worse than simple ignorance, naïve ideas about science lead people to make bad decisions with confidence. And in a world where many high-stakes issues fundamentally boil down to science, this is clearly a problem.

Naturally, the solution to the problem lies in good schooling — emptying minds of their youthful hunches and intuitions about how the world works, and repopulating them with sound scientific principles that have been repeatedly tested and verified. Wipe out the old operating system, and install the new. According to a recent paper by Andrew Shtulman and Joshua Valcarcel, however, we may not be able to replace old ideas with new ones so cleanly. Although science as a field discards theories that are wrong or lacking, Shtulman and Valcarcel’s work suggests that individuals —even scientifically literate ones — tend to hang on to their early, unschooled, and often wrong theories about the natural world. Even long after we learn that these intuitions have no scientific support, they can still subtly persist and influence our thought process. Like old habits, old concepts seem to die hard.

Testing for the persistence of old concepts can’t be done directly. Instead, one has to set up a situation in which old concepts, if present, measurably interfere with mental performance. To do this, Shtulman and Valcarcel designed a task that tested how quickly and accurately subjects verified short scientific statements (for example: “air is composed of matter.”). In a clever twist, the authors interleaved two kinds of statements — “consistent” ones that had the same truth-value under a naive theory and a proper scientific theory, and “inconsistent” ones. For example, the statement “air is composed of matter”  is inconsistent: it’s false under a naive theory (air just seems like empty space, right?), but is scientifically true. By contrast, the statement “people turn food into energy” is consistent: anyone who’s ever eaten a meal knows it’s true, and science affirms this by filling in the details about digestion, respiration and metabolism.

Shtulman and Valcarcel tested 150 college students on a battery of 200 such statements that included an equal and random mix of consistent and inconsistent statements from several domains, including astronomy, evolution, physiology, genetics, waves, and others. The scientists measured participants’ response speed and accuracy, and looked for systematic differences in how consistent vs. inconsistent statements were evaluated.

If scientific concepts, once learned, are fully internalized and don’t conflict with our earlier naive concepts, one would expect consistent and inconsistent statements to be processed similarly. On the other hand, if naive concepts are never fully supplanted, and are quietly threaded into our thought process, it should take take longer to evaluate inconsistent statements. In other words, it should take a bit of extra mental work (and time) to go against the grain of a naive theory we once held.

This is exactly what Shtulman and Valcarcel found. While there was some variability between the different domains tested, inconsistent statements took almost a half second longer to verify, on average. Granted, there’s a significant wrinkle in interpreting this result. Specifically, it may simply be the case that scientific concepts that conflict with naive intuition are simply learned more tenuously than concepts that are consistent with our intuition. Under this view, differences in response times aren’t necessarily evidence of ongoing inner conflict between old and new concepts in our brains — it’s just a matter of some concepts being more accessible than others, depending on how well they were learned.

There are, though, a few lines of evidence arguing against this interpretation. First, the authors found that participants who had best mastered scientific concepts (determined by their overall accuracy) were especially slow to verify inconsistent statements. A learning-based explanation of performance would have predicted the opposite — that mastery and speed should go hand in hand. More convincingly, a different study has shown that even those who’ve achieved an extremely high level of competence in a specific scientific field are still prone to make classifications based on naive, early concepts from childhood. In a speeded classification task analogous to Shtulman and Valcarcel’s, university biology professors were found to take longer to classify plants as living relative to moving nonliving things, a bias toward equating motion with life that is evident in young children.

Taken together, these findings suggest that we may be innately predisposed to have certain theories about the natural world that are resilient to being drastically replaced or restructured. These naive theories provide hunches and rules of thumb that likely helped us survive long before we needed to contemplate the atom, cells, or relativity. While some theories of learning consider the unschooled mind to be a ‘bundle of misconceptions’ in need of replacement, perhaps replacement is an unattainable goal. Or, if it is attainable, we may need to rethink how science is taught.

Are you a scientist who specializes in neuroscience, cognitive science, or psychology? And have you read a recent peer-reviewed paper that you would like to write about? Please send suggestions to Mind Matters editor Gareth Cook, a Pulitzer prize-winning journalist at the Boston Globe. He can be reached at garethideas AT gmail.com or Twitter @garethideas.



52 Comments

Add Comment
Show All | Jump To: 1-10 | 11-20 | 21-30 | 31-40 | 41-50 | 51-60 | Next
View

  1. 1. RSchmidt 12:38 PM 8/21/12
    Yet we have many that not only continue with the Old OS but install in their children a bronze age OS. I have been talking about the ALL curriculum for years. Instead of the 3Rs, reading, righting and 'rithmatic you have Arithmetic, Literacy and Logic. The other subjects, excluding physical education, are there to apply the knowledge gained by these core subject. With the addition of logic, less emphasis is placed on teaching "facts" and more is placed on reasoning and the ability to evaluate data sources. The best thing you can do for your child's future is to teach them the benefits of delayed gratification, I believe the second best is to teach them critical thought. One of the worst thing to teach them is that faith is a valid replacement for knowledge, and that faith is more important that evidence. Reply | Report Abuse | Link to this


  2. 2. RogerPink 01:10 PM 8/21/12
    I'm more concerned with unsupported superstitions that scientists, in spite of all evidence to the contrary, believe. For instance "The simplest answer is most likely the correct one" is almost never correct and requires herculean feats of rationalization to defend, yet this misunderstanding of Occam's Razor (not even close to what his intent was when he wrote it) pervades science and our society as a whole. These are the real dangers to science, not that we have to think an extra second in order figure out the correct answer to distinguish between colloquial and scientific definitions. Reply | Report Abuse | Link to this


  3. 3. Derick in TO in reply to RSchmidt 01:23 PM 8/21/12
    If you teach your children critical thinking, and teach them to apply it to everything, you don't need to teach them that faith is not a valid replacement for knowledge - it's self-evident to anyone who has ever thought critically about their own beliefs. Of course, critical thinking among the general population is anathema to religious and political leaders. It's so much harder to control a populace that questions what it sees and hears and challenges established ideas. Hence the lack of logic and critical thinking in school curricula, and the inclusion of (un)intelligent design and "teaching the controversy". Reply | Report Abuse | Link to this


  4. 4. WRQ9 01:37 PM 8/21/12
    You ( the SI editing staff ) continue to refer to some people as "you". When I read that, it refers to me, and anybody else reading it. It gets annoying at times. I am fully aware that even the best scientific reasoning can be flawed, and the flaws are therefore generally of greater scope than imagined. I can anticipate weaknesses in any of my notions and make allowances regarding language and so forth. Like a good engineer, I imagine what total breakdown could look like in every phase of an operation. Still in all, I do not assume imperfection in others as an exorcise in respect. I assume adults have performed to their peak capacity and any failure would be directly related to the complexity of the task and/or their ability to perform it. Assumed perfection is an unavoidable flaw in reasoning, but without it no reasoning is possible. Without evidence to the contrary, many arguments, however ridiculous in hindsight are daily tried in trepidation. Many humans will retry pet ideas well beyond the point of reason fearing the alternative, this is a form of psychosis, with the exception of legal circles. An open mind respects all attempts at reasoning. Religious people are granted by the constitution, the right to assume in a benign fashion certain unprovable concepts. Science does not allow this argument, but many scientists share dual loyalties. In effect much scientific knowledge became possible because of biblical teachings. Multiple interpretations of phenomena are not always contentious. Any good scientist must be comfortable with the circumstance of many possibilities in order to progress without the stigma of prejudice. It is good exorcise to imagine the possibility of these " unprovable concepts " if just to " cleanse the palette " for new endeavors. Reply | Report Abuse | Link to this


  5. 5. lucaspa in reply to RogerPink 01:52 PM 8/21/12
    I applaud you for knowing that "The simplest explanation is most likely the correct one" is not Ockham's Razor nor correct. It actually derives from Newton. But it is not a danger to science, because scientists don't use it in theory evaluation. It comes up in apologetics, usually atheist apologetics. What Ockham meant was not to add hypotheses to descriptions of phenomenon. His example, from his time, was "objects move because of an impetus". Ockham argued that movement was change in space over time, so all you needed to say was "objects move". That "because of an impetus" was a hypothesis that was not necessary to explain the phenomena. To Derrick: not all religious leaders or politicians deride reason. To say that they all do is to violate reason and critical thinking. Nor is faith necessarily an absence of evidence or reason. It can be, but often is not. Faith is belief in the absence of PROOF. We all believe things we cannot scientifically demonstrate. That does not mean we lack evidence or reason. For instance, the idea that democracy is a good form of government is not scientific nor "proven" by scientific evidence. Yet it does have evidence and reasons behind it. "It is important to recognize that not all "facts" are susceptible to scientific investigation, simply because some observations and experiences are entirely personal. I cannot prove that someone loves his or her child. The emotions that any individual claims to have are not susceptible to scientific documentation, because they cannot be independently verified by other observers. In other words, science seeks to explain only objective knowledge, knowledge that can be acquired independently by different investigators if they follow a prescribed course of observation or experiment. Many human experiences and concerns are not objective, and so do not fall within the realms of science." Douglas Futuyma, Science on Trial, the Case for Evolution, 1995, p 167. My children have faith that I love them. That faith is based upon evidence. But it's not science and it's not "proof". Millions of people have evidence for the existence of deity. It's faith, but not without evidence or reason. Reply | Report Abuse | Link to this


  6. 6. lawman108 02:14 PM 8/21/12
    I wonder what the researchers would have seen had they done brain scans as people answered the questions. Just speculating, maybe the pre-conceived notions that people grow up with correspond to different parts of the brain. So an answer that is consistent doesn't need to be checked, but one that is inconsistent has somehow be "tagged" so that the answer gets shunted off to a different part of the brain for consideration. Reply | Report Abuse | Link to this


  7. 7. Will_in_BC 03:34 PM 8/21/12
    I am not sure I would read too much into the longer time it took people with a good science background to respond to the inconsistent statements. A simple explanation would be that they are trained to look for subtleties and not accept the obvious. For example when I looked at the statement about air and matter I stopped to think of the various definitions of matter and which one might best apply. I think it's a good think that trained minds stop and ponder. Reply | Report Abuse | Link to this


  8. 8. FloriodaJ782 in reply to lucaspa 04:36 PM 8/21/12
    "For instance, the idea that democracy is a good form of government is not scientific nor "proven" by scientific evidence. Yet it does have evidence and reasons behind it." Democracy is not, necessarily a good form of government, nor does it have evidence behind it. Historically, democracies have been short-lived. We have only been a democracy since we legislated universal suffrage. It's true that we are betting our lives on it, but I don't think most of us are comfortable making that bet; we just don't seem to have much choice. As to "reasons behind ..." your belief that democracy is good, that's demonstrably wrong. We already see evidence of the idea that 50% can get what they want by making the other 50% pay for it, is alive and growing. Reply | Report Abuse | Link to this


  9. 9. tharriss in reply to lucaspa 05:10 PM 8/21/12
    "My children have faith that I love them. That faith is based upon evidence. But it's not science and it's not "proof". Millions of people have evidence for the existence of deity. It's faith, but not without evidence or reason." Whatever this evidence based faith tells your children and whatever this "evidence" is for existence of a deity, it is not a process well recommended for making good decisions or finding the actual truth. Children have faith Santa will leave presents under the tree for them, and evidence backs them up on it. The fact is that faith easily misleads people, even if they use some facts to rationalize their faith. If people want to take a statue that cries blood as "evidence" their faith is valid, and don't bother to look at the rusty plumbing in the ceiling above the statue, it says a lot about the value of their faith in reaching a true conclusion, but very little about the actual existence of a deity. A bit of science would go a long way into clearing up such delusions, but hey, by your stated standards, better to declare areas that are unprovable by science and just romanticize faith (backed up by "evidence") as holding the key.... what sillyness. "We all believe things we cannot scientifically demonstrate. That does not mean we lack evidence or reason. " Perhaps you are right that the one thing doesn't necessarily directly mean the other, but it does demonstrate a willingness to believe things that actually may not be true. Just because we all do it, doesn't mean it is a good way to reach a good conclusion. The beauty of the scientific method is that the process gets around our natural inclination to just believe all sorts of silly things, whether the evidence in front of our eyes seems to back it up or not, and instead makes us go through a strict process that strips away our biases and brings us ever closer to knowing what is actually happening around us. It isn't a perfect process, and usually takes time and many iterations to get past our built in preconceptions and misconceptions and human failings, but it is the best tool ever devised for eventually getting at the truth of things.... and I imagine before very much longer, will even have a scientifically verifiable way of scanning your brain and actually proving you love your kids.... although I agree most kids won't be much interested in such proof of something that seems obvious to them.... but it isn't the obvious nature of it (that you describe as faith) that makes it true... plenty of obvious seeming things actually aren't true. Reply | Report Abuse | Link to this


  10. 10. jgrosay 05:14 PM 8/21/12
    Yeah, it seems that Abrahan Maslow was the first in pointing that any new knowledge is put on the basis of the previously existent info on the field the subject has, be it a conjecture, a prejudice, or anything lacking an experimental basis. This would be one of the reasons learning things for an examination by reviewing old multiple choice questions on the subject is very dangerous: your mind records at the same level, and associated with the question, all the possible answers offered, and when in an actual situation, it will present to your conscience at similar levels of certainty the false and the right answers, in real life decision making this can lead to dangerous mistakes hard to notice until you're confronted with its consequences. Reply | Report Abuse | Link to this

Posted by at