Ever since I was a teenager, I’ve intermittently read Consumer Reports, relying on it for guidance in all manner of purchase decisions. CR has been known for rigorous testing of all manner of consumer products and the rating of various services, arriving at its rankings through a systematic testing method that, while not necessarily bulletproof, has been far more organized and consistent than most other ranking systems. True, I haven’t always agreed with CR’s rankings of products and services about which I know a lot, but at the very least CR has often made me think about how much of my assessments are based on objective measures and how much on subjective measures.

Until now.

I just saw something yesterday on the CR website that has made me wonder just how scientific CR’s testing methods are, as CR has apparently decided to promote alternative medicine modalities by “assessing” them in an utterly scientifically ignorant manner. Maybe I just haven’t been following CR regularly for a while, but if there’s an article that demonstrates exactly why consumer product testing organizations should not be testing medical treatments; they are ill-equipped to do so and lack the expertise and knowledge. The first red flag was the title, namely Hands-on, mind-body therapies beat supplements. The second red flag was the introduction to the article:

A new survey of subscribers to Consumer Reports found that prescription drugs generally performed better than alternative therapies for 12 common health problems. But hands-on treatments such as chiropractic care and deep-tissue massage, as well as mind-body therapies such as yoga and meditation, held their own, especially for certain conditions. Far fewer said that dietary supplements helped a lot.

Prescription drugs helped the most for nine of the conditions we asked about: allergies, anxiety, colds and flu, depression, digestive problems, headache and migraine, insomnia, irritable bowel syndrome, and osteoarthritis.

But chiropractic care performed better than drugs for back pain, and deep-tissue massage beat drugs for neck pain. Massage was as also as good as drugs for fibromyalgia. Those hands-on therapies also scored near the top for osteoarthritis as well as for headaches and migraines.

Whatever rigorous testing methodologies CR might bring to various products, its editors clearly have zero clue when it comes to science- and evidence-based medicine if they think that a survey is the appropriate way to determine which treatments work or how well treatments work relative to each other. There was a perfect example of what that is so just the other day with the study that appeared in New England Journal of Medicine that was touted as evidence that the placebo effect is powerful but in reality what the study showed is that, while placebos can make patients think they feel better, they don’t actually do anything to change the underlying pathology of the disease for the better. That’s why science and randomized clinical trials are necessary to determine what therapies work, which ones do not, and which ones work better than others. Surveys are a notoriously unreliable and deceptive (as in self-deceptive) way of trying to assess the relative merits of various therapies, representing as they do, mainly an aggregation of testimonials. Yet that’s what CR is using to try to rank alternative medicine therapies.

Worse, CR concludes its introduction:

For details, see our full report on alternative therapies, including advice on how to find a good chiropractor, massage therapist, yoga instructor, or other alternative-medicine practitioner.

I would submit to you that any reputable testing organization should not–I repeat, should not–be providing advice on how to find alternative medicine practitioners. On the other hand, note the bait and switch. Massage therapy is not necessarily “alternative.” At least it’s not alternative until it’s infused with woo like talk of “life energies” and such. Ditto yoga instructors, given that yoga, stripped of its woo, is basically stretching exercises. As for chiropractors, my standards for what would constitute a “good” chiropractor would be a bit different than most; I’d choose chiropractors who function primarily as physical therapists, eschewing any suggestion that they can cure any disease or treat anything other than musculoskeletal complaints. Any chiropractor who still believes in those mystical, magical “subluxations” that only chiropractors can find or who promote the idea of “innate intelligence” would not be a “good” chiropractor in my book; he’d be a quack.

My scientific orientation aside, it is nonetheless rather interesting to peruse CR’s report on “alternative” treatments entitled Alternative treatments More than 45,000 readers tell up what helped. The first thing I noticed was a rather obvious logical fallacy in the form of argumentum ad populum (i.e., appeal to popularity):

Done anything alternative lately? If so, you have a lot of company. When we surveyed 45,601 Consumer Reports subscribers online, we found that three out of four were using some form of alternative therapy for their general health. More than 38 million adults make in excess of 300 million visits to acupuncturists, chiropractors, massage therapists, and other complementary and alternative practitioners each year in the United States.

One wonders how long it will be before quacks start quoting the figure of three quarters of CR readers using alternative medicine or subtly misrepresenting the figure as three-quarters of Americans. After all, “complementary and alternative medicine” (CAM) and “integrative medicine” (IM) practitioners try very, very hard to paint those of us who are skeptical of their claims as hopeless Luddites trying to resist the inevitable CAM wave washing over medicine. Argumentum ad populum is one of the strongest logical fallacies that CAM proponents use. In fact, CR virtually admits that its survey is utterly useless for telling its readers what does and doesn’t work (although no doubt that’s not how the editors see it) when it states:

A total of 30,332 survey respondents gave us their perceptions of the helpfulness of treatments for their most bothersome conditions over the past two years. The respondents were Consumer Reports subscribers, and our findings might not be representative of the general population. Respondents based their opinions on personal experience, so the results can’t be compared with scientific clinical trials. And our results do not take into account the power of the placebo effect, the tendency of people to find even simulated or sham interventions helpful.

None of these caveats stops CR, though, which bravely dives right into the pool of pseudoscience promoted by appeals to popularity.

Even though this survey is pretty much useless (as CR admits while carrying it out anyway) for providing guidance about what therapies to use, I do have to admit–grudgingly–that this article does provide some rather interesting information, chief of which is that its results show that those nasty, evil, reductionistic prescription drugs in general outperformed any alternative therapy, even in a subjective, self-reported, Internet survey of CR’s subscribers like this. For instance, for allergy, prescription medications and over-the-counter medications were listed by CR readers by far as the top two therapies that “helped a lot.” It’s also interesting to note that only 2% of its readers used chiropractic to treat allergies, although 41% said that it “helped a lot.” Surprisingly, even for conditions for which “mind-body” therapies might be considered effective, prescription medications ruled the roost, including depression and anxiety. Other favorite conditions for which alt-med is used and for which alt-med claims success even though its success is virtually all placebo response yielded to the power of big pharma: Irritable bowel syndrome, insomnia, headache, and colds and flu.

In fact, for only two conditions did any “alternative” modality beat or even come close to conventional medicine even in this biased, self-reported survey: back pain/neck pain (which I lumped together because they’re both the spine), osteoarthritis, and fibromyalgia. And what was effective? For neck/back pain chiropractic for neck/back pain, chiropractic and massage were reported to be as effective as or more effective than prescription medication, which is not surprising because conventional medicine prescribes physical therapy for these conditions anyway and for spine problems chiropractic is generally physical therapy with woo liberally sprinkled on top. For fibromyalgia deep tissue massage was in a dead heat with prescription medication, which is probably more an indication of the lack of good therapies for fibromyalgia right now than it is an indication that alternative therapies work. Ditto osteoarthritis, where the effectiveness of massage probably indicates the same thing.

There are three other results of this survey that are worth mentioning, for instance this passage:

For most conditions we asked about, the No. 1 reason respondents gave for choosing an alternative treatment was simply that they were “a proponent” of it.

“Some people use these therapies because it’s just the way they were raised,” says Richard Nahin, Ph.D., M.P.H., senior adviser for scientific coordination and outreach at the National Institutes of Health’s National Center for Complementary and Alternative Medicine.

Some say they have gone through a transformational process, such as a major illness that has caused them to look at their life in a different way, Nahin says. Others believe dietary supplements are safer than prescription medication because they’re natural, even though that’s not necessarily the case, he says.

In other words, CAM is belief-based medicine, not science-based medicine, and here we appear to have a senior advisor from NCCAM admitting just that.

The second thing is that this article once again tries to make the claim that conventional, scientific medicine is “embracing” CAM by including a section discussing doctors and CAM. Of course, the interesting thing about this section is that it actually portrays physicians who practice CAM as being relatively uncommon, while representing one as being a “brave maverick doctor” who practices acupuncture. Even though I’ve seen doctors who practice acupuncture before (I’m talking to you, Brian Berman), I still can’t figure out the mental contortions and cognitive dissonance that must be necessary to make that happen. Still, in this article, we have a family practitioner named Dr. Rick Hobbs sticking needles into people to realign their qi. He’s even a kindly-appearing, bespectacled old guy wearing a bow tie! Can it get any more Norman Rockwell than that? Not without extreme difficulty or resurrecting Norman Rockwell himself to paint something.

Finally, CR’s Hands-on and mind-body therapies: A user’s guide is depressing to behold. First, one notes the classic bait and switch, where therapies that could be considered part of science-based medicine, such as massage, yoga (which is just gentle stretching exercise), and meditation (which is more or less relaxation) are represented as “alternative” and then lumped in with quackery (acupuncture) as though the quackery were equivalent. Chiropractic itself would be a form of physical therapy if it were stripped of its vitalistic woo. In other words, not only has CR apparently decided that pointless Internet surveys where the respondents are all self-selected are a valid way to assess the efficacy of therapies, even while in essence admitting that they are not, but it’s apparently bought into the pseudoscientific world view at the heart of CAM. Whether it did so through belief or cynically in order to pander to a large, woo-loving segment of its audience, I don’t know and don’t care. What I do know is that CR has in this article utterly failed its readers and betrayed its history of rigorous product testing.

I hope it was worth it for CR to shoot its own credibility in the foot with such abandon.

<!--<!--