False Positives

False Positives

via Climate Audit by Steve McIntyre on 6/28/12

Some readers may have noticed a Dutch scandal in the academic psychology industry. See here (h/t Pielke Jr).
The previously undisclosed whistleblower is said to be Uri SImonsohn, co-author of the article: "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant." The authors set out the following sensible solution to the problem of false positive publications:

Table 2. Simple Solution to the Problem of False-Positive Publications
Requirements for authors
1. Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article.
2. Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data-collection justification.
3. Authors must list all variables collected in a study.
4. Authors must report all experimental conditions, including failed manipulations.
5. If observations are eliminated, authors must also report what the statistical results are if those observations are included.
6. If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate.
Guidelines for reviewers
1. Reviewers should ensure that authors follow the requirements.
2. Reviewers should be more tolerant of imperfections in results.
3. Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions.
4. If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication.

If these rules were applied by real_climate_scientists, most of the criticisms at Climate Audit would be eliminated.
However, there are no signs that real_climate_scientists have any intention of adopting these rules, as evidenced by Gavin Schmidt's bilious outrage at the idea that Briffa should have reported the Yamal-Urals regional chronology considered and discarded in favor of the known HS of the small Yamal chronology.
The language of false positives was also used by the Texas sharpshooters, Wahl and Ammann, in connection with the failed verification statistics from MBH98.