What can we learn from polls?
Part of my job here at Bryan is teaching courses every fall semester, and part of that teaching responsibility is creating tests for my students. Over the years, I've discovered (often the hard way) that composing a good test question can be difficult. I've had students ask me during exams about what some of the words in my questions mean (I guess I have a complicated vocabulary), and occasionally I've gotten correct answers to poorly-worded questions that were not what I was looking for. Even more puzzling is the common habit of students to answer one question one way and a related question a different way, even though the answers they give are contradictory. It's bizarre.
I say all that not to denigrate Bryan students, but to highlight the difficulty of assessing knowledge through written questions, especially multiple choice questions. As far as Bryan students go, after hearing some real horror stories at the BSG conference from teaching colleagues at other colleges, I think we're quite blessed here with some truly exceptional students. But in general, people struggle with answering questions well. At least, that's been my experience.
The latest Reports of the NCSE has a paper that I think illustrates this problem quite well. It's a report by Bishop et al. on a complex survey of Americans' beliefs about evolution and creation. Not surprisingly, people seem confused.
For example, 39% of respondents marked the following statement as "True:"
That's something I've learned over the years of writing tests. I find that the more information I cram into a question, the more opportunity for confusion. Shorter questions with just one idea are easier for people to answer.
Other questions just seemed to confuse people. Consider this one:
Other questions are more blunt and to the point. Here are some interesting ones:
A couple other clear ones:
One final question of interest to me: Only 35% of respondents agreed that "The theory of evolution is not supported by any confirmed facts." Only 52% recognized that statement as false, but 13% were unsure. I wonder how much of that disagreement or uncertainty arises simply from confusion about science and how facts relate to scientific explanations.
It's an interesting paper well worth examining.
Bishop et al. 2010. Americans' scientific knowledge and beliefs about human evolution in the year of Darwin. Reports of the NCSE 30(3):16-18.
Feedback? Email me at toddcharleswood [at] gmail [dot] com.
I say all that not to denigrate Bryan students, but to highlight the difficulty of assessing knowledge through written questions, especially multiple choice questions. As far as Bryan students go, after hearing some real horror stories at the BSG conference from teaching colleagues at other colleges, I think we're quite blessed here with some truly exceptional students. But in general, people struggle with answering questions well. At least, that's been my experience.
The latest Reports of the NCSE has a paper that I think illustrates this problem quite well. It's a report by Bishop et al. on a complex survey of Americans' beliefs about evolution and creation. Not surprisingly, people seem confused.
For example, 39% of respondents marked the following statement as "True:"
God created the universe, the earth, the sun, moon, stars, plants, animals, and the first two people within the past 10 000 years.But only 18% marked this question as "True:"
The earth is less than 10 000 years old.Aren't those really kind of the same questions? But twice as many believe the "God created..." version than the simple statement about the age of the earth? That suggests that the first question is just too long. People are probably skimming it. When asked point blank whether they believe the earth is young, we get a better handle on who believes what.
That's something I've learned over the years of writing tests. I find that the more information I cram into a question, the more opportunity for confusion. Shorter questions with just one idea are easier for people to answer.
Other questions just seemed to confuse people. Consider this one:
God allows organisms to survive by way of natural selection in a post-Flood world.I would guess that the specialized creationist jargon "post-Flood world" is probably going to be unrecognizable by the average American. Not surprisingly, 19% responded "Not sure" to that question.
Other questions are more blunt and to the point. Here are some interesting ones:
Dinosaurs lived at the same time as people.That's a pretty standard creationist belief, and 40% of respondents said it was true (48% said false). A more troubling question:
Human fossils have been found mixed in with dinosaur fossils showing that humans existed at the same time that dinosaurs existed.Only 41% reject that as false, 43% think it's true, and 16% are "not sure." Again, though, it's unclear what this means precisely. Are Americans just confused about the nature of the fossil record, which would be suggested by the 16% who were unsure? Or have they been bamboozled by old and erroneous creationist claims about Paluxy?
A couple other clear ones:
God created the fundamental laws of physics and chemistry in just the right way, so that life, particularly human life, would be possible.and
The complexity of life cannot have arisen by chance or random events.To be honest, the first one seemed a bit complicated to me, but a whopping 69% said it was true and only 9% said they were unsure. The second question was marked as true by 59%. I'd guess they second one was answered intuitively, but the first seems to be specifically targeting belief in one argument popularized by the Intelligent Design movement.
One final question of interest to me: Only 35% of respondents agreed that "The theory of evolution is not supported by any confirmed facts." Only 52% recognized that statement as false, but 13% were unsure. I wonder how much of that disagreement or uncertainty arises simply from confusion about science and how facts relate to scientific explanations.
It's an interesting paper well worth examining.
Bishop et al. 2010. Americans' scientific knowledge and beliefs about human evolution in the year of Darwin. Reports of the NCSE 30(3):16-18.
Feedback? Email me at toddcharleswood [at] gmail [dot] com.