Geoff Chambers posted some of the comments from the invite only members section at Skeptical Science (that the admins there apparently left public for a period of time). One post from John Cook is highly relevant to this discussion (emph. mine):
2010–10-8 ” …a while ago, I added a bias field to the user database and a bit of code so as comments came in, I could specify whether the user was skeptics or warmest/proAGW/mainstream (still haven’t found a satisfactory term for our side). I only assign bias if its obvious from the comment. I haven’t done anything with that data yet, I’m not even sure why I’m doing it other than my obsessive compulsion to collect data. The other day, Steve Lewandowsky (cognitive scientist) asked if I had any numbers on the ratio of skeptics to warmists so I dove into the database and counted up around 100 assigned skeptics and around 400 assigned warmists.”
If I did my math correctly 100 out of 500 total = 20% … exactly the same number as the alleged “content analysis” done on “1067 comments” from allegedly “unique visitors” at Skeptical Science (SKS).
It is simply far too great a coincidence that an analysis of 1067 comments, provided exactly the same ratio as Lewandowsky co-author John Cook said his internal database showed was the ratio of skeptic vs non-skeptic participants.
Which also almost exactly matched the 18% Lewandowsky reports another survey in the US found:
“… survey of the U. S. public in June 2010 pegged the proportion of “skeptics” in the population at 18% (Leiserowitz, Maibach, Roser-Renouf, & Smith, 2011) found.
That all is bad enough on its own – however, there is another gigantic hole in their claims.
Their “content analysis” of 1067 comments, and their resultant claimed ‘skeptic share’ at Skeptical Science has effectively no real life relationship with the number and share of skeptic participants at SKS. This is especially true considering SKS’s highly antagonistic attitude and actions towards skeptics who dare venture to their little club – from regular non-skeptic participants and moderators alike.
Comparing comment counts offers no legitimate insight into the skeptic vs non-skeptic participant ratio, which is the metric supposedly being measured.
A simple review of one post at Skeptical Science is illustrative of the almost complete lack of association between skeptic comments and skeptic participant shares.
Lets choose the recent cross post from Shaping Tomorrows World, of John Cook’s excuse-making, err … ‘explanation’ of the Recursive paper posted here.
This post has generated, to date, a total of 106 responses. Lets break down the responses:
- Barry Woods – skeptic – posts 9,13,14,15,22
- Geoff Chambers – skeptic – posts 24,27,28,38,40,60,78 (heavily moderated and strong mod warning), 89 (moderated again), 101 (moderated – final warning)
- Brad Keyes – skeptic – posts 30, 31 (deleted), 32, 34, 49, 51, 52, 55, 57, 58, 63, 66, 69, 70, 73, 92, 93, 96, 98 (moderated and banned – Brad’s foray is a typical skeptic experience at SKS)
There are a couple other posts that could be definied as skeptical of the SKS position in this thread, but cannot be defined as AGW skeptic’s – which is the subject of this study.
While we’re reviewing lets look at a few of the typical comments from non-skeptics to see how skeptics are treated at SKS:
- Post 47 definitely NOT a skeptic – but a perfect demonstration of the attitude towards skeptics at the site (you’ll note this clear ad hominem attack was ignored by mods):
Although a recent occurrence came close, no thread was better deserving of the following metaphorical advice: “don’t wrestle with a pig. You’ll both be covered in mud and the pig loves it.” I urge all that are able to think rationally to not waste their time.
- Post 68 – from a site moderator:
I’m not going to engage you, because this is silly. John Cook’s paper says it all, and I have no intention of spending my time arguing about “sides” that have been entirely fabricated by a small community of “victims” who have identified themselves as separate and special.
- Post 74 – another site moderator – John Hartz:
@John Cook: Do you now have enough raw material from this comment thread for another paper in your series?
- Post 102 – post by site moderator – “John Hartz” another site moderator tells him not to “feed the troll” referring to geoff chambers.
So – what’s the count:
- Skeptic comment count: N=35 comments (out of 106 total) … 33.02% of all comments
- Non-Skeptic comment count: 71 comments (of 106 total) … 65.98% of all comments
- Skeptic individual participants count: N=3 (out of 42 total participants) … 7.14% of all participants are SKEPTIC’s
- Non-Skeptic individual participants count: N=38 (out of 42 total participants) … 90.48% of all participants are NON-SKEPTIC’s
An interesting aside – regarding participation by moderators and/or SKS staff:
- SKS moderators/staff commenting individually count: N=40 comments (out of 106 total) … 37.74% of all comments were made by SKS staff/moderators
- SKS moderators/staff individual participant count: N=10 (of 38 non-skeptic posters) … SKS moderators/staff comprised 26.32 of all participants
The data – at least in this post from SKS – is clear … contrary to the authors conclusions, while 33% of the comments here were from skeptics, just 7% of the participants were the same.
This shows the SKS formula provided in the Lewandowsky Supplemental Information – which uses total skeptic comments vs non-sceptic comments as support for and proof of the alleged diversity of the SKS, provides little or no meaningful information on the share of skeptic’s who participate at SKS.
Additionally, the paper’s “comment analysis” is, to be charitable, highly suspect, considering this supposed standalone comment analysis finds EXACTLY THE SAME conclusion as author John Cook found in his wholly separate and unrelated forum post – using his own internal participant “bias” information
Worse, the Lewandosky authors extend this seriously flawed “comment analysis” conclusion as definitively representative of all 7 other non-skeptic sites that offered the survey.
A more detailed review would likely find somewhat different results, however the basic premise here – that a “comments” review does not provide meaningful insight into the share of “participants” who are skeptics, is not likely to significantly change.
[Authors Note: please consider this a draft – quick online – version. Am in process of more detailed fact and error checking – thanks]