The GiveWell Blog has some suggestions for “Suggestions for the Social Sciences”. Here is the big one:

Our single biggest concern when examining research is publication bias, broadly construed. We wonder both (a) how many studies are done, but never published because people don’t find the results interesting or in line with what they had hoped; (b) for a given paper, how many different interpretations of the data were assembled before picking the ones that make it into the final version.

The best antidote we can think of is pre-registration of studies along the lines ofClinicalTrials.gov, a service of the U.S. National Institutes of Health. On that site, medical researchers announce their questions, hypotheses, and plans for collecting and analyzing data, and these are published before the data is collected and analyzed. If the results come out differently from what the researchers hope for, there’s then no way to hide this from a motivated investigator.

As the example of the NIH illustrates this is not just a social science problem, it is rife in any science which utilizes statistics. Statistical methods have become metrics to attain by any means necessary, when in reality they should be guidelines to get a ...