Experimental design in Munchkinland
I've been struggling with Facebook once again. It really is a breathtaking combination of arrogance and ignorance. And even though I feel like I'm tilting at windmills, I thought I might say something about experimental design.
Let's say, just for the sake of argument (wink, wink), that there's a terrible plague ravaging the land of Oz. Off in Munchkinland, the mayor of the Munchkin City catches the plague but starts taking a spoonful of oregano every day, and amazingly, he recovers from the plague.
So did the spoonfuls of oregano cure or help the Munchkin mayor? As things stand, we can't tell, and I hope you can see why that is, because it's one of the most basic rules of reasoning: Correlation is not causation. Just because two things happen together doesn't mean one causes the other, especially when you have only one example. It could very well be that the Munchkin mayor's strong, healthy body fought off the plague all by itself without help from the oregano. We just don't know.
Now let's say the rest of the Munchkins in Munchkinland hear about their mayor's oregano dosage and start to take oregano themselves. Soon enough, word starts to spread: Maria Munchkin was near death with the plague, but as soon as she started taking oregano, she felt better. The same thing happened to Mel Munchkin, Michael Munchkin, and the entire Lollipop Guild!! Now can we say that oregano makes a difference?
This may be surprising, but no, we still can't conclude that oregano is a good treatment for the plague. Why? Because we don't know how many people took oregano and died anyway. Let's say 100 Munchkins got the plague and took oregano and 12 of them recovered. Does that sound like an effective treatment?
Well, yeah, you say, for those 12 Munchkins it was very effective! OK, but what if the survival rate for the plague was already 12% even without the oregano? What then? The only thing we could say was that oregano was no better or worse than no treatment at all.
But what if the survival rate was 80% without the oregano, and 92 out of 100 Munchkins who took oregano survived? What then? The survival is better than no oregano at all. Doesn't that prove that oregano works?
OK, don't get mad at me, but no, that still doesn't do it. Why? Because minds and bodies are weird things. An optimistic outlook really can help a person recover, and nothing inspires optimism like thinking that you're getting life-saving medicine. That's called the placebo effect. So if I get the plague, and I'm convinced that taking oregano will cure that plague, guess what? Chances are, I'm likely to recover, even if the oregano does nothing to my body, because the oregano does something to my confidence, which then helps my body.
But here's the trick: The placebo effect doesn't make oregano an effective treatment, any more than any other placebo. If I got the plague, and I was absolutely convinced that cod liver oil or peanut butter or baked potatoes would make me better, I will attribute any improvement in my condition to those placebos. This is how medical charlatans of the past centuries could keep attracting customers: All they needed was a few "testimonials" (if they didn't just outright lie to people), and they could keep peddling their snake oil. It's also how folk medicine persists. Think about all the bizarre folk treatments for warts (like washing your hands in a silver bowl by the light of a full moon). Why do you think such outlandish ideas persist? All it takes is one or two people to swear, "It worked for me!"
But the placebo effect doesn't tell us if oregano is really effective at treating the terrible plague of Oz. What we need to do is somehow eliminate the placebo effect, and that's why we need controlled, blind trials. We need one group of Munchkins to get oregano, and we need another group of Munchkins to not get oregano. But we need both groups to think that they are getting oregano so that we can see if oregano really has an effect or if it's just the placebo effect. How can we do this? We'd make up some pills, some of which contained a good dose of oregano and some of which do not. Then we'd give them random labels and send them out to doctors treating Munchkins with the plague. That way, we the researchers would know which doctors really have oregano and which don't, but the doctors and patients don't know what they're actually getting. They all think they're getting some medicine that might help (or it might not). The group that doesn't get the oregano is the control group, and it's a blind study because the doctors and patients don't know what medicine they're actually getting (they're blind to the medicine).
Then we watch what happens. If the ones taking the oregano recover at a greater rate than the ones who don't, then we can say that oregano really is effective, because we can be reasonably sure it's not just a placebo effect, because no one actually knew which medicine they were getting. So the placebo effect is even (and controlled) across the entire experiment.
Now you might think, this is all nuts! Even if oregano might save Munchkin lives, we should just give it to them. We can't experiment with people's lives! And there's some truth to that. That's why, during the experiment, we get the patient's consent to be a part of the study, and we inform them very clearly that there's only a 50/50 chance of getting the experimental drug. That way, they don't have to be a part of the experiment if they just want to get the best treatment possible.
We also need to keep all the other treatment variables the same. We don't just give them a fake pill and send them home to die. We keep up the rest of the treatments we would have otherwise, even if we gave them oregano. We don't act like monsters.
Plus, in cases of really dangerous diseases for which medicine already exists, the placebo control used in experiments will often be the best known treatment, so no one gets deprived of actual medicine that will help them. In those cases, the best outcome we can hope for is that oregano is at least as good as or better than the best medicine that we have.
I'll also add that doctors will halt experiments like this if it turned out that oregano really is super-effective at treating the terrible plague of Oz. If you're six months into a twelve month study, and 100% of the oregano-treated Munchkins recover while 100% of the Munchkins without oregano died, then you stop the study and start giving everyone oregano. Again, we don't act like monsters in our experimenting.
But the "give it to them anyway" argument really doesn't hold water after we practice these ethics of caregiving and medical research. Because at the heart of this is the real question that has a real answer: Is oregano actually good medicine for the terrible plague of Oz? If it is, then we definitely need to use it. But if it isn't good medicine, then telling people that it will help them is lying. It's not much different from quacks peddling whiskey cure-alls in the 1800s. We don't lie to people and tell them they're getting medicine, when we don't really know if it will be effective.
So when a Munchkin doctor writes an article in Munchkin Newsweek swearing that oregano is good medicine and that it helped people at his hospital, there's nothing wrong with asking whether the effectiveness of the treatment was established with a controlled, blind study. Because that's how good medical science works. Just noticing that oregano might help Munchkins with the plague is a great first start in medical research, but it's not enough, especially when other Munchkin doctors say they did controlled, blind studies at their hospitals, and they found oregano had no effect.
So be smart, and stop saying that this is all just a political game. Because science is not a political game, even when some of the scientists can't figure this out. If controlled, blind studies tell us that a medicine doesn't have any effect in treating a disease, it doesn't matter how many studies uncover the placebo effect. It's just not good medicine.
Have you read my book? You should check that out too!
