Tuesday, November 04, 2008

Peer review: Conspirators for status quo?

This piece from The Scientist caught my attention.
People who advocate alternative scientific theories, including Creationism, free energy, and any number of forms of alternative medicine, claim their ideas would find their way to publication in scientific journals and win converts if only they weren't being screened out by the peer review process.  Peer review, as they see it, is a conspiracy to protect the status quo against the threat of new truths.

During a break at an NIH review panel a few years ago, I was scanning the list of grant applications that were not being scored because they were considered uncompetitive (usually about 50% of all applications). One caught my eye because it was a resubmission from a famous scientist that I knew. I wondered why this accomplished scientist would have his grant summarily rejected twice.

First, I read the reviews of the proposal from its initial submission, which revolved around the technical feasibility of an approach he was implementing. All three reviewers mentioned the same central issue. I then read the response of the applicant, which began: "It is so typical of the status quo, that when their sacred cow is gored, they circle the wagons in defense…." Ouch! I immediately guessed why the grant was rejected the second time.

Although such an emotional response from a well-regarded scientist was surprising, it also made me uncomfortable. It reminded me of my own similar response a decade earlier to what I took as slights by a review panel. In my revised proposal, I was circumspect enough to try to cover my opinions, but I did not take their criticisms seriously, much to my detriment (I was rejected again). Recently, I dug up these old reviews and was chagrined to find nothing insulting from reviewers in them. Time has given me the emotional distance that I had sorely needed.

Serving on multiple review panels has also given me a better perspective. Rather than being self-serving ogres who are part of an elaborate conspiracy to thwart the ambitions of their fellow scientists and maintain the status quo, the reviewers I know are usually motivated by a desire to serve the community and to help fix a system that we all see as inherently flawed. Although some of these reviewers do inadvertently contribute to problems with peer review, it is usually by being too nice rather than too critical. Instead of telling an applicant that their proposal is hopeless, they are far more likely to suggest ways to make it better.

No comments: