One of my goals in writing for this blog is to educate the general public about how to evaluate a scientific study, specifically medical studies. New studies are being reported in the press all the time, and the analysis provided by your average journalist leaves much to be desired. Generally, they fail to put the study into context, often get the bottom line incorrect, and then some headline writer puts a sensationalistic bow on top.

In addition to mediocre science journalism we also face dedicated ideological groups who go out of their way to spin, distort, and mutilate the scientific literature all in one direction. The anti-vaccine community is a shining example of this – they can dismiss any study whose conclusions they do not like, while promoting any horrible worthless study as long as it casts suspicion on vaccines.

Yesterday on Age of Autism (the propaganda blog for Generation Rescue) Mark Blaxill gave us another example of this, presenting a terrible pilot study as if we could draw any conclusions from it. The study is yet another publication apparently squeezed out of the same data set that Laura Hewitson has been milking for several years now - a study involving macaque infants and vaccinations. In this study Hewitson claims a significant difference in brain maturation between vaccinated and unvaccinated macaque infants, by MRI and PET analysis. Blaxill presents the study without noting any of its crippling limitations, and the commenters predictably gush.

The first (and really only) thing you need to know about this study is that it involves 9 vaccinated monkeys and 2 controls. That’s right – just 2 controls. The fact that Hewitson bothers to do statistical analysis on such a small set of subjects is laughable. Let’s keep in mind that most pilot studies turn out to be wrong – they are called pilot studies because they are intended to point the way to further research, not as a basis for any conclusions. Serious researches recognize that pilot studies are shots in the dark – and that counts even for good pilot studies, which this is not.

If the outcome were something hard and dramatic – like survival vs death, then 2 subjects would be a reasonable pilot study. But in this case Hewitson is doing a somewhat tricky measurement of brain volume changes over time and binding of opioid ligands in the amygdala. It is also worth noting that there were originally 4 controls, but one was eliminated due to improper protocol. We never learn what happened to the third monkey, we are just told there is data on two controls. This kind of missing data, especially when the overall numbers are so pathetically low, is very concerning.

She is also making multiple analyses (another red flag by itself), which means she can compare multiple variables looking for any difference. Then she invokes the sharpshooter fallacy and declares any change she does find to be clinically meaningful. So while there is no difference in brain volume or amygdala volume between exposed and unexposed monkeys, she finds differences in the change over time. We don’t know if still other variables were looked at and not reported – this is another weakness of pilot studies and why follow up studies replicating the specific effects reported are necessary before any conclusions can be drawn.

As further evidence of looking for any difference then declaring that the outcome of interest, we can look back to Hewitson’s 2008 reporting of her monkey data, in which she wrote:

“Compared with unexposed animals, exposed animals showed attenuation of amygdala growth and differences in the amygdala binding of [11C]diprenorphine.”

But in the current study she finds increased amygdala growth in exposed monkeys:

Not surprisingly, given the different maturational trajectories in exposed vs. unexposed animals, (unexposed decreasing and exposed increasing) there was a statistically significant interaction between exposure and time on total amygdala volume (Wald ?2=10.93; P=0.001). However, there were no significant main effects on total amygdala volume of either exposure (Wald ?2=0.75; P=0.39) or time (Wald ?2=1.14; P=0.29).

So which is it? Reading the results of the current study, especially in light of previous publications, gives the overall impression of a random scatter of data with incredible cherry picking in order to make the argument that there are any meaningful results at all.

Taken by itself, this is a worthless study. The numbers of subjects is too small to do any meaningful analysis. The results are all over the place, and not even consistent with prior publications by the same authors. The analysis is also far-fetched. Hewitson argues that both thimerosal-containing vaccine and MMR (which does not contain thimerosal) contribute to the alleged brain changes she is reporting. While the word “autism” does not appear in her report, Blaxill is concluding in his reporting that these brain changes are the same as those found in autism (an absurd conclusion given how non-specific these changes are, even if real, which cannot be concluded from this study). The anti-vaccine agenda is now clear – they get to have their cake and eat it too. They can now argue that an interaction between thimerosal and MMR cause or contribute to autism, through completely independent mechanisms, apparently.

To put this study further into context, this research is being conducted by the Thoughtful House Center for Children – Andrew Wakefield’s home after he was essentially kicked out of the UK and subsequently struck off. Wakefield’s name, however, does not appear anywhere on the current study, although he was listed as final author on previous publications from the same research. Apparently his name has become too toxic for the Thoughtful House.

The current study also appears in a obscure journal, Acta Neurobiologiae Experimentalis – which dedicated an entire issue to publishing dubious research on autism. The same issue includes two articles by the father and son Geier team – other vaccine and autism researchers who are off in their own world and whose research cannot be replicated.

Conclusion

This current study, as well as the entire macaque research program by Hewitson, is a good example of terrible research. The subject numbers are far too small for any meaningful statistics, and the outcomes being followed are numerous and tricky with a random scatter of results not even consistent between different publications of the same research.

What we have is far worse than ideological reporting and spinning of the scientific research – apparently we have the ideological conduction of research in the first place. This is similar to the research program of Benveniste on homeopathy.

In general it is a good rule to be suspicious of research that seems to be unique to one researcher or research team and is out of step with the broader research community. Unfortunately, such research contaminates the literature and is easily exploited to confuse the media and the public who often do not distinguish crank research from legitimate science.

___________________

Crossposted on NeuroLogicaBlog

Others reporting on this study:

Respectful Insolence – Orac also points out that Hewitson failed to disclose her COI – that she has a child with autism who is part of the Autism Omnibus suit.

Ibrb – Author, Sullivan, also points out that amygdala size should increase in macaques, so it is especially odd that the non-exposed monkeys’ amygdalas shrank. That makes no sense, and is likely due to the quirkiness of having only two controls. So the authors conclusions are entirely based upon a weird result in their tiny control group – i.e. this is completely bunk science.