There is an interesting article in this month’s Atlantic Monthly regarding the “flexible” nature of medical statistics and the way that researchers (often unknowingly) massage statistical analyses to support favored hypotheses. The article is essentially a layman’s overview of the work of John P. A. Ioannidis, whose paper “Why Most Published Research Findings Are False” is the most downloaded article in the history of PLoS Medicine. From the article:

“The studies were biased,” he says. “Sometimes they were overtly biased. Sometimes it was difficult to see the bias, but it was there.” Researchers headed into their studies wanting certain results—and, lo and behold, they were getting them. We think of the scientific process as being objective, rigorous, and even ruthless in separating out what is true from what we merely wish to be true, but in fact it’s easy to manipulate results, even unintentionally or unconsciously. “At every step in the process, there is room to distort results, a way to make a stronger claim or to select what is going to be concluded,” says Ioannidis. “There is an intellectual conflict of interest that pressures researchers to find whatever it is that is most likely to get them funded.”

I don’t find his conclusions particularly surprising, but I think it’s interesting that they’ve found their way into the mainstream media. He makes an excellent point about why public disclosure of these problems is the right choice, despite the fact that public confidence in medical science may be shaken. The ability to admit that we’re wrong is what separates us from the snake oil salesmen.

“If we don’t tell the public about these problems, then we’re no better than nonscientists who falsely claim they can heal,” he says. “If the drugs don’t work and we’re not sure how to treat something, why should we claim differently? Some fear that there may be less funding because we stop claiming we can prove we have miraculous treatments. But if we can’t really provide those miracles, how long will we be able to fool the public anyway? The scientific enterprise is probably the most fantastic achievement in human history, but that doesn’t mean we have a right to overstate what we’re accomplishing.”