Last week at TAM8 some SBM colleagues (David Gorski, Kimball Atwood, Harriet Hall, Rachel Dunlop) and I gave two workshops on how to find reliable health information on the web. As part of my research for this talk I came across a recent and interesting study that I would like to expand upon further – Quality and Content of Internet-Based Information for Ten Common Orthopaedic Sports Medicine Diagnoses.

The fact that the article focuses on orthopedic diagnoses is probably not relevant to the point of the article itself, which was to assess the accuracy of health information on the web. They looked at 10 orthopedic diagnoses and searched on them using Google and Yahoo, and then chose the top results. They ultimately evaluated 154 different sites with multiple reviewers for quality of content and also for their HON rating.

For background, the HON rating comes from an independent organization, the Heath on the Net Foundation, that rates health care sites on a number of criteria. These include assessment of how authoritative the sources are, the level of transparency, and if opinions expressed are justified with evidence and references. While generally reasonable, the HON assessment does not necessarily involve a thorough assessment of the quality of the science on a given website, and many sites with what I would consider dubious information have earned the HON seal of approval.

Among other things, this new study evaluated how scientifically accurate health information on the websites they reviewed was, and also compared them to their HON ratings. They divided the websites evaluated into various types – non-profit, academic, commercial, and individually run. What they found was that the quality of information was significantly better on non-profit and academic sites than on commercial and individually run sites. This is not surprising – commercial sites are likely to be compromised by a desire to advertise or sell product. But “commercial” also refers to sites that monetize content – not necessarily selling products, but simply providing content as their product in order to sell advertising. This includes sites such as WebMD.

It is also not surprising that individual sites also scored relatively low on average. An individually run site is only as good as the individual running it, so there is bound to be a great deal of variability. Also, individuals are more likely to make mistakes or have missing information than groups.

Non-profit and academic sites are more likely to have editorial policies that emphasize quality and integrity of content. But also they are more likely to have some vetting process for information. At SBM (a non-profit site) for example, we carefully guard our editorial integrity and also provide some layer of editorial oversight.

But the study also provides reason to be cautious, even about the best sites. They rated quality of information on a 100 point scale and found a range of 45-61%. So even the best sites had a mediocre score. This is likely due to the fact that health information is complex and rapidly changing. Nothing short of a thorough editorial and peer-review process is likely to generate both reliable and thorough up-to-date information. This study is therefore reason for all providers of health information on the net to raise their game. There is definitely room for improvement.

The study also found that having the HON seal of approval did significantly correlate with higher quality and integrity scores. So the code does mean something, even if it is still not a guarantee of science-based content.

Conclusion

Further study of health information on the net is warranted as the results of this one study argue for caution. For providers it suggests we need to improve our filters and editorial process to improve the quality of our content. For consumers the results suggest that non-profit and academic health information sites are most reliable, while commercial and individual sites should be viewed with caution.

But further I would suggest to consumers of health information on the net that no single site or article should ever be relied upon for information. The best way to get a thorough and accurate treatment of a health topic is to look at multiple sites. Try to determine what the consensus of opinion and information is, and be very wary of outliers. This is generally good advice for any research, not just health information.