Eight Warning Signs of Junk Science

Armed and Dangerous » Blog Archive » Seven Eight Warning Signs of Junk Science

Here is a non-exclusive list of seven eight symptoms to watch out for:

Science by press release. It’s never, ever a good sign when ‘scientists’ announce dramatic results before publishing in a peer-reviewed journal. When this happens, we generally find out later that they were either self-deluded or functioning as political animals rather than scientists. This generalizes a bit; one should also be suspicious of, for example, science first broadcast by congressional testimony or talk-show circuit.

Rhetoric that mixes science with the tropes of eschatological panic. When the argument for theory X slides from “theory X is supported by evidence” to “a terrible catastrophe looms over us if theory X is true, therefore we cannot risk disbelieving it”, you can be pretty sure that X is junk science. Consciously or unconsciously, advocates who say these sorts of things are trying to panic the herd into stampeding rather than focusing on the quality of the evidence for theory X.

Rhetoric that mixes science with the tropes of moral panic. When the argument for theory X slides from “theory X is supported by evidence” to “only bad/sinful/uncaring people disbelieve theory X”, you can be even more sure that theory X is junk science. Consciously or unconsciously, advocates who say these sorts of things are trying to induce a state of preference falsification in which people are peer-pressured to publicly affirm a belief in theory X in spite of private doubts.

Consignment of failed predictions to the memory hole. It’s a sign of sound science when advocates for theory X publicly acknowledge failed predictions and explain why they think they can now make better ones. Conversely, it’s a sign of junk science when they try to bury failed predictions and deny they ever made them.

Over-reliance on computer models replete with bugger factors that aren’t causally justified. No, this is not unique to climatology; you see it a lot in epidemiology and economics, just to name two fields that start with ‘e’. The key point here is that simply fitting historical data is not causal justification; there are lots of ways to dishonestly make that happen, or honestly fool yourself about it. If you don’t have a generative account of why your formulas and coupling constants look the way they do (a generative account which itself makes falsifiable predictions), you’re not doing science – you’re doing numerology.

If a ‘scientific’ theory seems tailor-made for the needs of politicians or advocacy organizations, it probably has been. Real scientific results have a cross-grained tendency not to fit transient political categories. Accordingly, if you think theory X stinks of political construction, you’re probably right. This is one of the simplest but most difficult lessons in junk-science spotting! The most difficult case is recognizing that this is happening even when you agree with the cause.

Past purveyers of junk science do not change their spots. One of the earliest indicators in many outbreaks of junk science is enthusiastic endorsements by people and advocacy organizations associated with past outbreaks. This one is particularly useful in spotting environmental junk science, because unreliable environmental-advocacy organizations tend to have long public pedigrees including frequent episodes of apocalyptic yelling. It is pardonable to be taken in by this the first time, but foolish by the fourth and fifth.

Refusal to make primary data sets available for inspection. When people doing sound science are challenged to produce the observational and experimental data their theories are supposed to be based on, they do it. (There are a couple of principled exceptions here; particle physicists can’t save the unreduced data from particle collisions, there are too many terabytes per second of it.) It is a strong sign of junk science when a ‘scientist’ claims to have retained raw data sets but refuses to release them to critics.

It would be way, way too easy to list the ways these symptoms have manifested with respect to the AGW panic. It’s a more useful exercise for the reader to think back and try to recognize them in previous junk-science flaps. Go and learn. And don’t get fooled again.