Lewandowsky: study “Useless” unless authors demonstrate “data integrity”

Lewandowsky: study “Useless” unless authors demonstrate “data integrity”

via Climate Audit by Steve McIntyre on 9/12/12

Lewandowsky has stated that an online survey by an opponent was "useless" "without the authors demonstrating the integrity of their data" and that their study "should not have been published without the authors demonstrating the integrity of their data—I doubt that they could". Words that apply even more forcibly to his own study. I guess it all depends on whose ox is being gored.
On Sep 27, 2010, about a month after the Lewandowsky survey was posted at several anti-skeptic blogs, John Cook posted the following at the private SkS forum entitled "Excerpt from Steve [Lewandowsky]'s email". At the time, Hans von Storch and Dennis Bray had just carried out an online survey. Lewandowsky complained that there was "no way to check or verify the integrity of the data" and therefore the data was probably "useless". Lewandowsky said that the study should not have been published "without the authors demonstrating the integrity of their data":

I'm sure Steve won't mind if I excerpt from the email he sent me about von Storch's paper:
Got the paper some time ago and I have corresponded with Bray. He seems like a nice guy but the survey data are apparently compromised by the login and password information having been circulated on a denialist mailing list. I have a copy of that email (via Deltoid).
Bray also violated all internet survey methodological standards by not recording dates, times, and IP numbers of respondents (I know this from him personally). He thus has no way to check or verify the integrity of his data. In other words, the data are possibly (probably?) useless, although the published paper seems to include more data than the previous unpublished report which was entirely compromised as just stated.
Now, all that said, the results are not particularly distressing from our perspective, and he correctly identifies that there is a large segment of the scientific community who think that the IPCC understated the problem.
Overall, though, this study should not have been published without the authors demonstrating the integrity of their data—I doubt that they could.

Lewandowsky himself then commented at the SkS forum on Sep 27 entitled "Background info on survey methodology", favorably citing a critique at Deltoid of methodology in a similar earlier survey, results of which Lewandowsky believed to have been included:

Hi all, see here for background on the survey problems: http://scienceblogs.com/deltoid/2005/05/bray.php
I believe that at least a subset of the data reported in the published paper relies on the data in the unpublished survey critiqued by Deltoid.
By the way, a Swedish journalist, Jens Ergon, has done a better controlled survey of Swedish cllimate scientists and their publications and found the usual 97% agreement on AGW. This was reported on Swedish TV and I have corresponded with him.

Graham Wayne then commented that the "problems of data integrity" put him off writing about the topic:

Well, the problems with the data integrity do throw something of a spanner in the works. If I were to write this up, I don't see how I could be less than candid about the problems, which would in turn sufficiently dilute the value of the research. Climate contrarians will jump on the flaws and hammer them to the exclusion of all else (surprised WUWT didn't do this already), so I'm rather put off writing about it.
Anyone else got a view on the practicality of focusing on this?

The (2005) Deltoid post that Lewandowsky had cited is here, which discussed the following survey result – a result that appears to contradict the "97%" of all climate scientists that is commonly cited:

Lambert commented as follows:

Is global warming skepticism amongst climate scientists as widespread as this survey indicates? To answer this we need to look at how the sampling was conducted:
The 2003 survey was conducted as an on-line survey. The existence of the survey was posted in the Bulletin of the American Meteorological Society, the Climlist server, and was sent to institutional lists in Germany Denmark and the U.K. As an effort to prevent general access to the survey, the survey was password protected. The password was contained in the informative message distributed according to the above.
However, the information about the survey was reposted (mail list membership required to read link) to the climatesceptics mail list by Timo Hameranta on Sep 20 2003: (my emphasis)

the survey (below) is directed to those involved in the natural sciences related to climate change and not, for example, those involved in policy analysis or economic issues.
I suggest that you participate by completing the questionnaire (instructions to participate below).
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
L�hett�j�: CLIMLIST Climatology Distribution List
[mailto:CLIMLIST@...] Puolesta CLIMLIST
L�hetetty: 19. syyskuuta 2003 18:12
Vastaanottaja: CLIMLIST@…
Aihe: Survey of Climate Scientists
CLIMLIST Mailing Number 03-09-24
Origin: "Dr. Dennis Bray"
Due to the nature of the distribution of electronic surveys some recipients of this message might have received the same previously. If that is the case, my apologies. Please do not submit the survey twice, although, once would be much appreciated. The survey is directed to those involved in the natural sciences related to climate change and not, for example, those involved in policy analysis or economic issues. Your discretion in choosing to participate in the survey on this basis would be greatly appreciated, as would your cooperation in making the survey a success. If you know of colleagues not contained on climlist but involved in the climate sciences, it would be appreciated if you could bring this survey to their attention with the suggestion that they too might like to participate by completing the questionnaire. Simply forwarding this message is likely the most convenient method.
… If you do choose to participate, the survey can be reached by opening your web browser and going to the following link:
http://w3g.gkss.de/G/Mitarbeiter/bray.html/
When the page opens click the link to "survey of climate scientists" Here you will be asked for a username and password.
For username enter "respondent" (without quotation marks)
For password enter "ccsurvey" (again without the quotation marks).
The survey is password protected as an effort to limit the respondents to those involved in the climate sciences. There is also the option to print the survey from a PDF file and submit though regular postal services. Electronic submissions do not transmit your email address and consequently anonymity is ensured. Thank you in advance for your cooperation.

Since the survey was anonymous, there is no way to ensure that only climate scientists participated and no way to prevent people from submitting the survey multiple times. Furthermore, the survey was distributed on the climatesceptics list which has over 200 members, almost all of them strongly skeptical about global warming. Since the total number of participants was just 557, this could serious skew the results. I don't believe that the results of this survey are representative of the views of climate scientists.

Lambert then cited a response from Bray at realclimate in which Bray stated:

It would be possible however, should one wish, to submit a duplicate return in the 2003 survey. 2003 surveys were checked for identical response patterns and none existed. Of course this does not mean that a single person could not fill out two surveys using different responses.

Lambert criticized this as follows:

The only thing Bray did to check for multiple responses by the same person was to see if there were any responses that were exact duplicates

As I interpret Lewandowsky's methodology, Lewandowsky's check was whether consecutive responses from the same IP address were identical (but this is only an interpretation of poorly described methodology.)
Lambert summarized his criticism as follows:

While no study is perfect, this study is so imperfect as to be useless. Since it was posted on the climatesceptics list, the sample is not representative. As a social scientist Bray should know this.

It is obvious that all the criticisms leveled by Lewandowsky and Lambert against the von Storch and Bray survey apply even more strongly to his own survey. Applying Lewandowsky's own standards, his own survey is "useless" unless he can demonstrate "data integrity", which he can't.
Instead of demonstrating data integrity, in Lewandowsky's response today (his first non-juvenile response), Lewandowsky argues, in effect, that data integrity doesn't matter – a familiar enough reaction in past paleoclimate disputes.
Lewandowsky, although he was quick to fabricate a jibe at me about a supposed pastrami sandwich, did not respond to or cite my post on this matter, though he did cite Tom Curtis' more limited critique. Nor did Lewandowsky examine the crosscut of data that I had used. Instead, Lewandowsky carried out a different crosscut of the data, arguing that he got similar results with this different crosscut. I placed source code for my analysis online so any critics could readily interpret my conclusions and understand points at issue. Unfortunately Lewandowsky has not dne so, making the process of replication and analysis less efficient than it ought to be. I'll try to take a look at Lewandowsky's most recent post over the next few days.
however, one point is obvious: Lewandowsky's defence is framed in terms of "outliers" as opposed to "data integrity". It seems evident that, using Lewandowsky's own words (about the Bray-con Storch study):

this study should not have been published without the authors demonstrating the integrity of their data—I doubt that they could