ClimateGate: Oh, the Animals!

Heh!

ClimateGate: How They All Squirmed...

By James Delingpole at the London Telegraph:

Among the many great amusements of the Climategate scandal are the myriad imaginative excuses being offered by the implicated scientists and their friends in the MSM as to why this isn’t a significant story. Here are some of the best:

Just go read it.  You know you want to...

ClimateGate: More Source Code Analysis...

Bloggers and scientists all over the world are digging into the IDL and Fortran programs that are part of the ClimateGate trove.  Here's a nice summary of the current findings from American Thinker.  A taste:

One can only imagine the angst suffered daily by the co-conspirators, who knew full well that the "Documents" sub-folder of the CRU FOI2009 file contained more than enough probative program source code to unmask CRU's phantom methodology. 

In fact, there are hundreds of IDL and FORTRAN source files buried in dozens of subordinate sub-folders. And many do properly analyze and chart maximum latewood density (MXD), the growth parameter commonly utilized by CRU scientists as a temperature proxy, from raw or legitimately normalized data. Ah, but many do so much more. 

Skimming through the often spaghetti-like code, the number of programs which subject the data to a mixed-bag of transformative and filtering routines is simply staggering. Granted, many of these "alterations" run from benign smoothing algorithms (e.g., omitting rogue outliers) to moderate infilling mechanisms (e.g., estimating missing station data from that of those closely surrounding). But many others fall into the precarious range between highly questionable (removing MXD data which demonstrate poor correlations with local temperature) to downright fraudulent (replacing MXD data entirely with measured data to reverse a disorderly trend-line).

In fact, workarounds for the post-1960 "divergence problem," as described by both RealClimate and Climate Audit, can be found throughout the source code. So much so that perhaps the most ubiquitous programmer's comment (REM) I ran across warns that the particular module "Uses 'corrected' MXD - but shouldn't usually plot past 1960 because these will be artificially adjusted to look closer to the real temperatures."

What exactly is meant by "corrected” MXD," you ask? Outstanding question -- and the answer appears amorphous from program to program. Indeed, while some employ one or two of the aforementioned "corrections," others throw everything but the kitchen sink at the raw data prior to output.

The more I discover or read about these “scientists”, the madder I get at them...

A Different Twist on ClimateGate...

Some commentators – predominantly political commentators – who are skeptical of AGW are not so much skeptical of the science as they are skeptical of the motivation for the science.  In particular, they believe that the environmental movement is largely controlled by socialists (or those even further to the left) who see the environmental issues as an excellent source of FUD (fear, uncertainty, and doubt) that can be used as levers to get the electorate in democracies to choose to give government control over many areas of their lives.  These commentators think that the pro-AGW scientists are “useful idiots”, perfect for exploitation by the socialist activists in control of the environmental movement. 

This all reeks of conspiricism to me, and it's in my general nature to be skeptical of such claims.  Stupidity and avarice, it seems to me, are perfectly sufficient to explain everything I've seen so far in ClimateGate.

But here's an observation that I must admit lends a little weight to the contentions of these commentators.  I'm going to start watching this a little more credulously than I have been...

Video Primer on the AGW “Settled Science”

Thanksgiving in Jamul...

Today is the first day Debbie and I have had together in about two weeks – first I was off on a combined personal and business trip, then she was up in Murietta, cooking for a fancy-schmancy dog training seminar (trainers and trainees).  She got back home late last night.

Our Thanksgiving Day feast this year will be with our friends and neighbors, Jim and Michelle Barnick (who live just up the hill from us).  They're having a few dozen friends and relatives over.  Our contribution is the pumpkin pie, which Debbie and I just finished making. 

We had some help from the dogs.  I made the pie crusts (a simple crust made of ginger snap crumbs and butter).  I accidentally brushed a few crumbs off onto the floor, where Race (our border collie yearling) found them.  You could see him immediately take on a wondering look, as if to say “Where the hell has this been my whole life?” 

Then a little while later I had the job of mixing the filling.  After pouring the filling into the pies (and a couple of extra custard cups), I had a nice, big mixing bowl coated with the filling mixture.  Basically that's eggs, evaporated milk, and pumpkin, with some spices and sugar.  If you're a dog, I don't think it gets any better than that.  Suffice it to say that the mixing bowl was shortly extremely clean, at a molecular level.  I could have just let it dry and thrown it back in the cabinet, but for form's sake I actually did wash it off...

Now the pies are in the oven, the dishes are all washed, and we're just relaxing for a bit.  In a few hours we'll wander up the hill to start our feast.  A few hours after that, we'll be painfully full of excellent food and drink, and we'll be smiling from the good company of our friends...

Happy Thanksgiving to you all, on this beautiful day in the chaparral of Lawson Valley, just east of Jamul...

ClimateGate Editorial...

By Lorrie Goldstein in the Toronto Sun.  The lede:

If you're wondering how the robot-like march of the world's politicians towards Copenhagen can possibly continue in the face of the scientific scandal dubbed "climategate," it's because Big Government, Big Business and Big Green don't give a s*** about "the science."

They never have. 

Go read the whole thing.

Can citizens win against the warmenists?  I think that maybe, just maybe, the answer is “yes!”

The Proclamation...

We have much to be thankful for, even as there is much to be worried about.  I've never heard it said better than this:

By the President of the United States of America.


A Proclamation.


The year that is drawing towards its close, has been filled with the blessings of fruitful fields and healthful skies. To these bounties, which are so constantly enjoyed that we are prone to forget the source from which they come, others have been added, which are of so extraordinary a nature, that they cannot fail to penetrate and soften even the heart which is habitually insensible to the ever watchful providence of Almighty God. In the midst of a civil war of unequaled magnitude and severity, which has sometimes seemed to foreign States to invite and to provoke their aggression, peace has been preserved with all nations, order has been maintained, the laws have been respected and obeyed, and harmony has prevailed everywhere except in the theatre of military conflict; while that theatre has been greatly contracted by the advancing armies and navies of the Union. Needful diversions of wealth and of strength from the fields of peaceful industry to the national defence, have not arrested the plough, the shuttle or the ship; the axe has enlarged the borders of our settlements, and the mines, as well of iron and coal as of the precious metals, have yielded even more abundantly than heretofore. Population has steadily increased, notwithstanding the waste that has been made in the camp, the siege and the battle-field; and the country, rejoicing in the consiousness of augmented strength and vigor, is permitted to expect continuance of years with large increase of freedom. No human counsel hath devised nor hath any mortal hand worked out these great things. They are the gracious gifts of the Most High God, who, while dealing with us in anger for our sins, hath nevertheless remembered mercy. It has seemed to me fit and proper that they should be solemnly, reverently and gratefully acknowledged as with one heart and one voice by the whole American People. I do therefore invite my fellow citizens in every part of the United States, and also those who are at sea and those who are sojourning in foreign lands, to set apart and observe the last Thursday of November next, as a day of Thanksgiving and Praise to our beneficent Father who dwelleth in the Heavens. And I recommend to them that while offering up the ascriptions justly due to Him for such singular deliverances and blessings, they do also, with humble penitence for our national perverseness and disobedience, commend to His tender care all those who have become widows, orphans, mourners or sufferers in the lamentable civil strife in which we are unavoidably engaged, and fervently implore the interposition of the Almighty Hand to heal the wounds of the nation and to restore it as soon as may be consistent with the Divine purposes to the full enjoyment of peace, harmony, tranquillity and Union.


In testimony whereof, I have hereunto set my hand and caused the Seal of the United States to be affixed.


Done at the City of Washington, this Third day of October, in the year of our Lord one thousand eight hundred and sixty-three, and of the Independence of the Unites States the Eighty-eighth.


By the President: Abraham Lincoln

It Begins...

Skeptical scientists are starting to tear into the raw data recently revealed by hackers who posted thousands of documents from East Anglia's CRU (see earlier posts).  Some New Zealanders have found that the raw data from their country was heavily – and arbitrarily – manipulated to show a hockey stick where none exists in the raw data.

Expect much, much more of this sort of disclosure...

And Yet It Cools...

Go read Jules Crittenden...

ClimateGate Satire...

IowaHawk is well-known for his satirical pieces lampooning just about everything deserving of lampooning.  Some of them are roll-around-on-the-floor funny, especially if they happen to be on a topic that's near and dear to your heart.

Well, now he's speared ClimateGate.  And I'm just now recovering.  If you enjoy satire, and you have a working knowledge of the AGW debate and ClimateGate, then put down your morning beverage, swallow, and go read.

One little sample:

Many of those behaviors are on display in the security film, as we see a sexless group of drone graduate students processing a raw datum with saliva, sawdust and Fortran code. After each iteration the time series is presented to the Alpha Grantwriter to see if fits inside his graph. Several graduate drones die of exhaustion, but the data eventually fit the template.

Next the Alpha Grantwriter flies to an international climate research conference with the completed PowerPoint template, where he will share his guano with other Alpha Grantwriters over cocktails in the hotel lounge. This is a process metascientists refer to as "peer review."

Lordy, that man can throw those words around...

Cruel Farce...

You've probably seen the recent news stories about the Belgian man who spent 23 years supposedly in a coma, but actually fully conscious – but unable to communicate.  A Poe-esque horror story...

Well, not so fast.  There are two separate lines of evidence that the poor man is actually conscious.  One of these lines of evidence is based on brain scanning, which the researcher involved cautions is “experimental” and “speculative”.  The other line of evidence is based on “facilitated communication” – the notion that a skilled facilitator can assist the patient in communicating.

Facilitated communication (FC for short) has a long and heart-breaking record.  Hundreds of parents of autistic children were bilked out of thousands of dollars by supposed FC experts claiming they could help their children communicate.  Experts have thoroughly demolished every claim of FC, yet it persists (though thankfully at a much reduced level).  Basically the FC experts just made it all up, collecting a nice fee in the process.

So when I heard that the Belgian man was communicating “with the assistantance of an FC expert”, all my red flags went up.  My favorite debunker of FC was James Randi, the magician and well-known skeptic, famous for his debunkings of junk science in general.  So this morning I went poking around at his web site, and voila! – Mr. Randi has been writing about this incident already, calling it a cruel farce and a vehicle for the FC expert to get a lucrative book deal.

Read the whole thing, and watch this little video of the FC expert in action.  Then make up your own mind about whether this poor Belgian man is being exploited or assisted...

Hide the Decline...

Well, that didn't take long!

CRU Document Hack Thoughts...

In reading various reactions to the documents hacked from CRU, I see three general trends:

• Those who, like me, feel betrayed by the corrupt science on exhibit – the manipulation of data, the suppression of honest debate, the withholding of information.  These people tend to view the CRU revelations as casting doubt on the “science” production of the CRU participants – which include most of the leading lights of pro-AGW science.
  
  
• Those who dismiss everything exposed in the CRU documents as merely the human side of science.  These folks seem to believe that all science is conducted in this fashion, and that the CRU documents simply expose this in an unusually candid fashion for the public to see.  Sheesh, I certainly hope these people are wrong!
  
  
• Those who see the CRU revelations as a blow to pro-AGW science, but think there's enough untainted science remaining to sustain their pro-AGW beliefs.  To me, this sounds like grasping at straws – the work tainted by the CRU documents is foundational work that remains, so far as I know, the only direct evidence of long-term global warming that is correlated to human activity.  I deliberately exclude climate models, as they are not observational science at all, but rather an attempt to project hypotheses into the future.  

What little coverage I've seen in the U.S. lamestream media has been superficial and oriented toward the scandalous aspects.  Nearly all the interesting analysis comes from blogs – hundreds of them.  In the U.K., the situation is different – there's more complete coverage from the media, including commentary by prominent scientists.  Compared to the U.S. coverage, it's quite impressive.

Steve McIntyre of Climate Audit is clearly enemy number one in the CRU documents.  These must be very interesting times for him.  His site has been overwhelmed with traffic, and he's set up a mirror site to handle the volume.  Some of the most interesting – and sober – analysis of the CRU documents can be found up there.  If this is a topic that interests you, I recommend monitoring his site.

It's going to be interesting to watch how this all shakes out...

An Email and My Response...

I received an email this morning that raises an interesting question.  The email (edited to preserve the anonymity of the writer):

I have enjoyed reading your posts on climategate, and am quite sympathetic to your position.  I am very skeptical about the science underlying the various global warming claims.  I am also sympathetic with your claim that publicly funded researchers should release their software codes to the public.

Nonetheless, the counter of these scientists is that other scientists can produce their own code and see what results they obtain.  I was wondering whether you knew if such attempts at reproduction have occurred, and if not, why not.  Is it that it is very expensive to undertake these studies and the ordinary funding sources do not want to fund such efforts?  Or do you think something else is goind on.  I would love to know the answer.  Thanks.

In any event, keep up the good work.
M.

Here's my response (slightly edited from the original to remove personal information):

Hi, M.!

Not only do they claim that other scientists could reproduce their results, they claim that two groups (one in Russia, and one in Australia) have already done so.  But I'm very skeptical of these results, as the two groups are hardly independent of the CRU crowd -- they're "insiders" in every sense.

It's an interesting question you raise, though, and one I've pondered a bit as I read through those CRU emails.  Reproducing their results has several possible meanings.  In the purest scientific sense, "reproduction" would mean that the same results were obtained with independently acquired and analyzed data.  In this sense, the CRU's results have NOT been reproduced, and they do not claim that they have been.  I believe reproduction in that sense would be quite challenging, as the CRU's claim is that virtually all available proxy temperature data was assessed in their efforts.  One step down on the reproduction scale would be for independent groups to take the same data but independently analyze it.  I can't be certain, as I don't have access to the actual studies, but from the emails it sounds like this approach is also NOT the kind of reproduction that the two groups did.  So one more step down on the reproduction scale would be for independent groups to take the same data and use the same general analytical approach.  I believe that this is what Mann et al are referring to when they claim their results have been reproduced, and could be reproduced by any qualified scientist.  Essentially that means someone checked their math, not exactly what I would call a validation of the approach.

What Mann et al have consistently refused to provide are detailed descriptions of exactly how (and why) they selected and adjusted their data sets.  Other researchers (like Steve McIntyre) with access to the raw data have been unable to reproduce the CRU's results with anything that resembles unbiased data selection or adjustment.  In fact, in some cases they come to very different results that would lead one to very different conclusions.  Hence the debate on Climate Audit, which has become increasingly informed over the past couple of years as McIntyre and his cohorts get more and more educated.

Speaking as a concerned citizen whose pocketbook is highly at risk here, I believe that what is urgently needed is for a truly independent and skeptical group of scientists to attempt to reproduce the CRU's results in the first, purest sense I described above.  We should not be considering sweeping and expensive mitigation efforts until we have done our very best to confirm the accuracy of the underlying science.

Enceladus Flyby...

Fantastic photos from yesterday's flyby of Enceladus, by Cassini.  Awesome!

CRU Roundup...

The blogosphere is chock full of stuff on the CRU data hack, now being dubbed “ClimateGate”.  Some that caught my fancy are here, here, here, and here.  The third link is to ClimateAudit's mirror site – Steve McIntyre has been so overwhelmed with traffic the past couple of days that he's had to engage a bigger server!

The lamestream media has fallen on its face with this one, as they have with so many other scandals that required even the tiniest bit of research and imagination.  Also, as with some of the past scandals (such as “RatherGate”), they seem to believe they have a vested interest – this time, in a pro-AGW stance.  Pajamas-clad amateurs have showed up the “pros” once again...

Still More on the CRU Data Dump...

Here's one more code file, this one calibrate_nhrecon.pro.  This is another IDL code file, and it appears to be calibrating tree ring data against direct temperature measurement data:

;
; Calibrates, usually via regression, various NH and quasi-NH records
; against NH or quasi-NH seasonal or annual temperatures.
;
; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline that affects tree-ring density records)
;
perst=1881.  ;note 1
peren=1960.
thalf=10.     ; filter to use to give extra hi & lo pass info
;
; Select season of the temperature data against which to calibrate
;
if n_elements(doseas) eq 0 then doseas=0      ; 0=annual, 1=Apr-Sep, 2=Oct-Mar
seasname=['annual','aprsep','octmar']
;
; Select record to calibrate
;
if n_elements(dorec) eq 0 then dorec=0
recname=['esper','jones','mann','3tree','briffa','iwarm','mj2000','crowley03','rutherford04']
print,recname[dorec]
doland=0      ; for Mann et al. only, 0=use NH mean, 1=use land>20N mean
;
if recname[dorec] eq 'mann' then begin
  if doland eq 0 then openw,1,'recon_mannNH.out'+seasname[doseas] $
                 else openw,1,'recon_mannLN20.out'+seasname[doseas]
endif else begin
  openw,1,'recon_'+recname[dorec]+'.out'+seasname[doseas]
endelse
;
multi_plot,nrow=2
if !d.name eq 'X' then window,ysize=800
;
; Compute the >20N land instrumental temperature timseries and filter
;
datst=1860
daten=1980
print,'Reading temperatures'
ncid=ncdf_open('/cru/u2/f055/data/obs/grid/surface/crutem2_18512001.mon.nc')
tsmon=crunc_rddata(ncid,tst=[datst,0],tend=[daten,11],grid=gtemp)
ncdf_close,ncid
nmon=12
ntemp=gtemp.nt
nyrtemp=ntemp/nmon
yrtemp=reform(gtemp.year,nmon,nyrtemp)
yrtemp=reform(yrtemp(0,*))
;
; Compute the northern hemisphere >20N land series
;
; First extract >20N rows
kl=where(gtemp.y gt 20.)
ylat=gtemp.y(kl)
tsnorth=tsmon(*,kl,*)
; Compute latitude-weighted mean
nhmon=globalmean(tsnorth,ylat)
; Compute seasonal/annual mean
nhmon=reform(nhmon,nmon,nyrtemp)
case doseas of  ;note 2
  0: lvy=mkseason(nhmon,0,11,datathresh=6)   ; could try 9,8 (Oct-Sep annual)!
  1: lvy=mkseason(nhmon,3,8,datathresh=3)
  2: lvy=mkseason(nhmon,9,2,datathresh=3)
endcase
;
; Filter it
;
filter_cru,thalf,tsin=lvy,tslow=lvylow,tshigh=lvyhi,/nan
;
; Read in record and filter
;
case recname[dorec] of
  'esper': begin
    openr,2,'/cru/u2/f055/data/paleo/esper2002/esper.txt'
    readf,2,nyr
    headst=''
    readf,2,headst
    rawdat=fltarr(7,nyr)
    readf,2,rawdat
    close,2
    x=reform(rawdat(0,*))
    densall=reform(rawdat(1,*))
  end
  'jones': begin
    openr,2,'../tree5/phil_nhrecon.dat'
    nyr=992
    rawdat=fltarr(4,nyr)
    readf,2,rawdat,format='(I5,F7.2,I3,F7.2)'
    close,2
    x=reform(rawdat(0,*))
    densall=reform(rawdat(3,*))
  end
  'mann': begin
    if doland eq 0 then begin
      openr,2,'../tree5/mann_nhrecon1000.dat'
      nyr=981
      rawdat=fltarr(2,nyr)
      readf,2,rawdat           ;,format='(I6,F11.7)'
      close,2
      x=reform(rawdat(0,*))
      densall=reform(rawdat(1,*))
    endif else begin
      openr,2,'../tree5/mannarea_all.dat'
      nyr=981
      rawdat=fltarr(11,nyr)
      headdat=' '
      readf,2,headdat
      readf,2,rawdat           ;,format='(I6,F11.7)'
      close,2
      x=reform(rawdat(0,*))
      densall=reform(rawdat(10,*))
    endelse
  end
  '3tree': begin
    openr,2,'../tree6/tornyamataim.ave'
    readf,2,nnn
    rawdat=fltarr(2,nnn)
    readf,2,rawdat
    close,2
    x=reform(rawdat(0,*))
    densall=reform(rawdat(1,*))
  end
  'briffa': begin
    restore,filename='/cru/u2/f055/tree6/bandtempNH_calmultipcr.idlsave'
    x=yrmxd
    densall=prednh
  end
  'iwarm': begin     ; use warm-season instrumental series as the predictor!
    x=yrtemp
    densall=mkseason(nhmon,3,8,datathresh=3)
  end
  'mj2000': begin
    openr,2,'/cru/u2/f055/data/paleo/ipccar4/data/mann03_orig.dat'
    readf,2,ncol
    readf,2,icol
    readf,2,nyr
    readf,2,nhead
    headst=strarr(nhead)
    rawdat=fltarr(ncol,nyr)
    readf,2,headst
    readf,2,rawdat
    close,2
    x=reform(rawdat(0,*))
    densall=reform(rawdat(icol,*))
  end
  'crowley03': begin
    openr,2,'/cru/u2/f055/data/paleo/ipccar4/data/crowley03_orig.dat'
    readf,2,ncol
    readf,2,icol
    readf,2,nyr
    readf,2,nhead
    headst=strarr(nhead)
    rawdat=fltarr(ncol,nyr)
    readf,2,headst
    readf,2,rawdat
    close,2
    x=reform(rawdat(0,*))
    densall=reform(rawdat(icol,*))
  end
  'rutherford04': begin
    openr,2,'/cru/u2/f055/data/paleo/ipccar4/data/rutherford04_orig.dat'
    readf,2,ncol
    readf,2,icol
    readf,2,nyr
    readf,2,nhead
    headst=strarr(nhead)
    rawdat=fltarr(ncol,nyr)
    readf,2,headst
    readf,2,rawdat
    close,2
    x=reform(rawdat(0,*))
    densall=reform(rawdat(icol,*))
  end
endcase
;
oopsx=x
oopsy=densall
;
kl=where((x ge datst) and (x le daten))
x=x(kl)
densall=densall(kl)
if total(abs(yrtemp-x)) ne 0 then message,'Incompatible years'
y1=densall
filter_cru,thalf,tsin=y1,tslow=ylow1,tshigh=yhi1,/nan
;
; Now correlate and regress them
;
printf,1,'Correlations and regression coefficients for '+seasname[doseas]
keeplist=where(finite(y1+lvy) and (x ge perst) and (x le peren),nkeep)
x=x(keeplist)
ts1=y1(keeplist)
ts2=lvy(keeplist)
r1=correlate(ts1,ts2)
dum=linfit(ts1,ts2,sigma=err_sigma)
c1=dum
printf,1,'Full timeseries:',r1,c1
plot,x,ts2
oplot,x,ts1*c1[1]+c1[0],thick=2
;
tsa=ts1(0:nkeep-2)
tsb=ts1(1:nkeep-1)
;mxd_ar1=correlate(tsa,tsb)
mxd_ar1=a_correlate(ts1,[-1])
tsa=ts2(0:nkeep-2)
tsb=ts2(1:nkeep-1)
;nht_ar1=correlate(tsa,tsb)
nht_ar1=a_correlate(ts2,[-1])
printf,1,'AR1 for MXD and NHEMI:',mxd_ar1,nht_ar1
;
tspred=c1(0)+c1(1)*ts1
tswant=ts2
plot,tspred,ts2,psym=1,$
  xtitle='Scaled Esper et al. anomaly  (!Uo!NC)',$
  ytitle='Northern Hemisphere temperature anomaly  (!Uo!NC)',$
  /xstyle,xrange=[-0.6,0.3],$
  /ystyle,yrange=[-0.6,0.3]
oplot,!x.crange,[0.,0.],linestyle=1
oplot,[0.,0.],!y.crange,linestyle=1
dum=linfit(tspred,ts2,sigma=se)
oplot,!x.crange,!x.crange*dum(1)+dum(0)
oplot,!x.crange,!x.crange*dum(1)+dum(0)+se(0)
oplot,!x.crange,!x.crange*dum(1)+dum(0)-se(0)
oplot,!x.crange,!x.crange*(dum(1)+se(1))+dum(0)
oplot,!x.crange,!x.crange*(dum(1)-se(1))+dum(0)
oplot,!x.crange,!x.crange*(dum(1)+se(1))+dum(0)+se(0)
oplot,!x.crange,!x.crange*(dum(1)-se(1))+dum(0)-se(0)
oplot,!x.crange,!x.crange*(dum(1)+se(1))+dum(0)-se(0)
oplot,!x.crange,!x.crange*(dum(1)-se(1))+dum(0)+se(0)
;
ts1=yhi1(keeplist)
ts2=lvyhi(keeplist)
r2=correlate(ts1,ts2)
dum=linfit(ts1,ts2)
c2=dum
printf,1,'High-pass      :',r2,c2
;
ts1=ylow1(keeplist)
ts2=lvylow(keeplist)
r3=correlate(ts1,ts2)
dum=linfit(ts1,ts2)
c3=dum
printf,1,'Low-pass       :',r3,c3
;
; Now compute the rms error between the reconstruction and the original
; timeseries
;
tserr=tswant-tspred
rmserr=sqrt( total( tserr^2 ) / float(n_elements(tserr)) )
printf,1,'RMS error between land>20Napr-sep temperature and Esper et al. reconstruction'
printf,1,rmserr
printf,1,'Uncertainties surrounding regression coefficients'
printf,1,err_sigma
;
printf,1,' '
printf,1,'Computations carried out over the period ',perst,peren
printf,1,' '
printf,1,'To separate low and high frequency components, a gaussian weighted'
printf,1,'filter was used with a half-width (years) of ',thalf
;
close,1
;
end

Where to start?  Here, I guess:

1.  Another arbitrary-looking choice of limits.  In this case, they've selected the period 1881 through 1960 to correlate the two data series.  The comment above, however, is very suggestive of why the 1960 cutoff was chosen.

2.  This piece of code looks like classic cherry-picking and fudging.  Here the software writer is choosing arbitrary months and data thresholds when sampling the annual mean of the instrumental data (which will be used to calibrate the tree-ring data.  The comment reinforces this notion.

Something that strikes me as I read through all this code: it looks amateurish, not at all like what I'd imagine code written by scientists expecting peer review would produce.  I'm no scientist, so it may well just reflect my naiveté about things scientific.  But I'm quite disappointed in it...

More on the CRU Data Dump...

The blogosphere is chock full of stuff this morning, mainly analyzing the emails and documents (more on that in another post).  So this morning I thought I'd take a gander at some of the software code that was included in the hacker's dump.  This is the contents of an IDL file named briffa_sep98_e.pro.  This code appears to be used to pre-process raw tree ring data.  I've highlighted areas of interest in bold red, and added a comment with a footnote number; otherwise the file's contents are unmodified:

;
; PLOTS 'ALL' REGION MXD timeseries from age banded and from hugershoff
; standardised datasets.
; Reads Harry's regional timeseries and outputs the 1600-1992 portion
; with missing values set appropriately.  Uses mxd, and just the
; "all band" timeseries
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********
;
yrloc=[1400,findgen(19)*5.+1904]  ;note 1
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
  2.6,2.6,2.6]*0.75         ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,'Oooops!'
;
loadct,39
def_1color,20,color='red'
plot,[0,1]
multi_plot,nrow=4,layout='large'
if !d.name eq 'X' then begin
  window, ysize=800
  !p.font=-1
endif else begin
  !p.font=0
  device,/helvetica,/bold,font_size=18
endelse
;
; Get regional tree lists and rbar
;
restore,filename='reglists.idlsave'
harryfn=['nwcan','wnam','cecan','nweur','sweur','nsib','csib','tib',$
  'esib','allsites']
;
rawdat=fltarr(4,2000)
for i = nreg-1 , nreg-1 do begin
  fn='mxd.'+harryfn(i)+'.pa.mean.dat'
  print,fn
  openr,1,fn
  readf,1,rawdat
  close,1
  ;
  densadj=reform(rawdat(2:3,*))
  ml=where(densadj eq -99.999,nmiss)
  densadj(ml)=!values.f_nan
  ;
  x=reform(rawdat(0,*))
  kl=where((x ge 1400) and (x le 1992)) ;note 3
  x=x(kl)
  densall=densadj(1,kl)     ; all bands
  densadj=densadj(0,kl)     ; 2-6 bands
  ;
  ; Now normalise w.r.t. 1881-1960
  ;
  mknormal,densadj,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
  mknormal,densall,x,refperiod=[1881,1960],refmean=refmean,refsd=refsd
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x) ;note 2
densall=densall+yearlyadj
  ;
  ; Now plot them
  ;
  filter_cru,20,tsin=densall,tslow=tslow,/nan
  cpl_barts,x,densall,title='Age-banded MXD from all sites',$
    xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
    zeroline=tslow,yrange=[-7,3]
  oplot,x,tslow,thick=3
  oplot,!x.crange,[0.,0.],linestyle=1
  ;
endfor
;
; Restore the Hugershoff NHD1 (see Nature paper 2)
;
xband=x
restore,filename='../tree5/densadj_MEAN.idlsave'
; gets: x,densadj,n,neff
;
; Extract the post 1600 part
;
kl=where(x ge 1400)
x=x(kl)
densadj=densadj(kl)
;
; APPLY ARTIFICIAL CORRECTION
;
yearlyadj=interpol(valadj,yrloc,x) ;note 2
densadj=densadj+yearlyadj
;
; Now plot it too
;
filter_cru,20,tsin=densadj,tslow=tshug,/nan
cpl_barts,x,densadj,title='Hugershoff-standardised MXD from all sites',$
  xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
  zeroline=tshug,yrange=[-7,3],bar_color=20
oplot,x,tshug,thick=3,color=20
oplot,!x.crange,[0.,0.],linestyle=1
;
; Now overplot their bidecadal components
;
plot,xband,tslow,$
  xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
  yrange=[-6,2],thick=3,title='Low-pass (20-yr) filtered comparison'
oplot,x,tshug,thick=3,color=20
oplot,!x.crange,[0.,0.],linestyle=1
;
; Now overplot their 50-yr components
;
filter_cru,50,tsin=densadj,tslow=tshug,/nan
filter_cru,50,tsin=densall,tslow=tslow,/nan
plot,xband,tslow,$
  xrange=[1399.5,1994.5],xtitle='Year',/xstyle,$
  yrange=[-6,2],thick=3,title='Low-pass (50-yr) filtered comparison'
oplot,x,tshug,thick=3,color=20
oplot,!x.crange,[0.,0.],linestyle=1
;
; Now compute the full, high and low pass correlations between the two
; series
;
perst=1400.
peren=1992.
;
openw,1,'corr_age2hug.out'
thalf=[10.,30.,50.,100.]
ntry=n_elements(thalf)
printf,1,'Correlations between timeseries'
printf,1,'Age-banded vs. Hugershoff-standardised'
printf,1,'     Region    Full   <10   >10   >30   >50  >100'
;
kla=where((xband ge perst) and (xband le peren))
klh=where((x ge perst) and (x le peren))
ts1=densadj(klh)
ts2=densall(kla)
;
r1=correlate(ts1,ts2)
rall=fltarr(ntry)
for i = 0 , ntry-1 do begin
  filter_cru,thalf(i),tsin=ts1,tslow=tslow1,tshigh=tshi1,/nan
  filter_cru,thalf(i),tsin=ts2,tslow=tslow2,tshigh=tshi2,/nan
  if i eq 0 then r2=correlate(tshi1,tshi2)
  rall(i)=correlate(tslow1,tslow2)
endfor
;
printf,1,'ALL SITES',r1,r2,rall,$
  format='(A11,2X,6F6.2)'
;
printf,1,' '
printf,1,'Correlations carried out over the period ',perst,peren
;
close,1
;
end

Well.  Where to start?  I guess I'll just dive right in...

1.  This code constructs a pair of 20 element arrays that are used later as inputs to an interpolation routine.  As the comments "fudge factor" and "very artificial correction for decline" would seem to indicate, these values appear to be completely arbitrary, and unsupported by any underlying theory about why the raw data should be manipulated this way.  I'll tell you what it looks like to this ancient, gray-bearded software who has constructed many software models in his career: this looks like someone manipulating the input to a model to get the desired results.  Publicly such a modeler would proclaim: “Lookee here!  I crunched the tree-ring data and got this result!” – when the reality is that the modeler “fudged” and “corrected” the data until the results matched his preconceived and desired result.  Smoking code gun number one.

2.  These two places are where the “very artificial correction” is being applied. 

3.  This code limits the data in the results to the years 1401 to 1991.  It would be interesting to know why those particular years were chosen.  I'm especially intrigued by the 1991 cutoff, as I've read elsewhere that the tree ring data from the late 20th century doesn't support the notion that tree ring width is a good proxy for temperature.  Could this be the reason for the early cutoff?

Another CRU Email...

The current climate models employed by the warmenist scientists use a temperature series that is based largely on tree ring data as a proxy for temperature.  Having grown up on a tree nursery, I was skeptical of the basic assumption that tree ring width in an individual tree will vary primarily because of ambient temperature.  I have personally observed much variance in the growth rate of trees based on other factors, like rainfall, nutrients in the ground, predation by animals, etc.  Certainly temperature has some effect on tree growth rate, but other factors also do.

So when I read this email (1256760240.txt) in the collection hacked from CRU, I was most interested.  Again, I've bolded and footnoted the sections I found interesting:

From: Phil Jones To: k.briffa@uea.ac.uk
Subject: FW: Yamal and paleoclimatology
Date: Wed Oct 28 16:04:00 2009

    Keith,
       There is a lot more there on CA now. I would be very wary about responding to this person now having seen  what McIntyre has put up. You and Tim talked about Yamal. Why have the bristlecones come in now.
   [1]http://www.climateaudit.org/?p=7588#comments
    This is what happens - they just keep moving the goalposts.
    Maybe get Tim to redo OB2006 without a few more series.
    Cheers
    Phil

     X-Authentication-Warning: ueamailgate02.uea.ac.uk: defang set sender to
     using -f
     Subject: FW: Yamal and paleoclimatology
     Date: Wed, 28 Oct 2009 15:39:48 -0000
     X-MS-Has-Attach:
     X-MS-TNEF-Correlator:
     Thread-Topic: Yamal and paleoclimatology
     Thread-Index: AcpDQ2sqWC+z2djuSqC1Ax4HdHoH1wUn1Ocw
     From: "Keiller, Donald"
     To:
     Cc:      X-ARU-HELO: CAMEXCH.ANGLIA.LOCAL
     X-ARU-sender-host: cambe01.ad.anglia.ac.uk (CAMEXCH.ANGLIA.LOCAL) [193.63.55.171]:25427
     X-ARU-Mailhub: yes
     X-ARU-Exchange: yes
     X-ARU-MailFilter: message scanned
     X-Spam-Status: no
     Reply-to: Don.Keiller@anglia.ac.uk
     X-Canit-CHI2: 0.00
     X-Bayes-Prob: 0.0001 (Score 0, tokens from: @@RPTN, f028)
     X-Spam-Score: 0.00 () [Hold at 5.00] SPF(none,0)
     X-CanItPRO-Stream: UEA:f028 (inherits from UEA:default,base:default)
     X-Canit-Stats-ID: 34330416 - 89bde843c4e5 (trained as not-spam)
     X-Antispam-Training-Forget:
     [2]https://canit.uea.ac.uk/b.php?i=34330416&m=89bde843c4e5&c=f
     X-Antispam-Training-Nonspam:
     [3]https://canit.uea.ac.uk/b.php?i=34330416&m=89bde843c4e5&c=n
     X-Antispam-Training-Spam: [4]https://canit.uea.ac.uk/b.php?i=34330416&m=89bde843c4e5&c=s
     X-Scanned-By: CanIt (www . roaringpenguin . com) on 127.0.0.1
     Dear Professor Briffa, I am pleased to hear that you appear to have recovered from your recent illness sufficiently to post a response to the controversy surrounding the use of the Yamal chronology;
     ([5]http://www.cru.uea.ac.uk/cru/people/briffa/yamal2009/cautious/cautious.htm)
     and the chronology itself;
     ([6]http://www.cru.uea.ac.uk/cru/people/briffa/yamal2009/)
     Unfortunately I find your explanations lacking in scientific rigour and I am more inclined to believe the analysis of McIntyre¹
     ([7]http://www.climateaudit.org/?p=7588)
     Can I have a straightforward answer to the following questions
     1) Are the reconstructions sensitive to the removal of either the Yamal data and Strip pine bristlecones, either when present singly or in combination?
     2) Why these series, when incorporated with white noise as a background, can still produce a Hockey-Stick shaped graph if they have, as you suggest, a low individual weighting?
     And once you have done this, please do me the courtesy of answering my initial email.
     Dr. D.R. Keiller
     -----Original Message-----
     From: Keiller, Donald
     Sent: 02 October 2009 10:34
     To: 'k.briffa@uea.ac.uk'
     Cc: 'p.jones@uea.ac.uk'
     Subject: Yamal and paleoclimatology
     Dear Professor Briffa, my apologies for contacting you directly, particularly since I hear that you are unwell.
     However the recent release of tree ring data by CRU has prompted much discussion and indeed disquiet about the methodology and conclusions of a number of key papers by you and co-workers.
     As an environmental plant physiologist, I have followed the long debate starting with Mann et al (1998) and through to Kaufman et al (2009).
     As time has progressed I have found myself more concerned with the whole scientific basis of dendroclimatology². In particular;
     1) The appropriateness of the statistical analyses employed
     2) The reliance on the same small datasets in these multiple studies
     3) The concept of "teleconnection" by which certain trees respond to the
     "Global Temperature Field", rather than local climate
     4) The assumption that tree ring width and density are related to temperature in a linear manner.
     Whilst I would not describe myself as an expert statistician, I do use inferential statistics routinely for both research and teaching and find difficulty in understanding the statistical rationale in these papers³.
     As a plant physiologist I can say without hesitation that points 3 and 4 do not agree with the accepted science.
     There is a saying that "extraordinary claims require extraordinary proof".
     Given the scientific, political and economic importance of these papers, further detailed explanation is urgently required.
     Yours sincerely,
     Dr. Don Keiller.

     --
     EMERGING EXCELLENCE: In the Research Assessment Exercise (RAE) 2008, more than 30% of
     our submissions were rated as 'Internationally Excellent' or 'World-leading'. Among the
     academic disciplines now rated 'World-leading' are Allied Health Professions & Studies;
     Art & Design; English Language & Literature; Geography & Environmental Studies; History;
     Music; Psychology; and Social Work & Social Policy & Administration. Visit
     [8]www.anglia.ac.uk/rae for more information.
     This e-mail and any attachments are intended for the above named
     recipient(s)only and may be privileged. If they have come to you in
     error you must take no action based on them, nor must you copy or show
     them to anyone please reply to this e-mail to highlight the error and
     then immediately delete the e-mail from your system.

     Any opinions expressed are solely those of the author and do not
     necessarily represent the views or opinions of Anglia Ruskin University.

     Although measures have been taken to ensure that this e-mail and
     attachments are free from any virus we advise that, in keeping with good
     computing practice, the recipient should ensure they are actually virus
     free.

     Please note that this message has been sent over public networks which may
     not be a 100% secure communications
     Email has been scanned for viruses by Altman Technologies' email management service -
     [9]www.altman.co.uk/emailsystems

   Prof. Phil Jones
   Climatic Research Unit        Telephone +44 (0) 1603 592090
   School of Environmental Sciences    Fax +44 (0) 1603 507784
   University of East Anglia
   Norwich                          Email    p.jones@uea.ac.uk
   NR4 7TJ
   UK
   ----------------------------------------------------------------------------

References

   1. http://www.climateaudit.org/?p=7588#comments
   2. https://canit.uea.ac.uk/b.php?i=34330416&m=89bde843c4e5&c=f
   3. https://canit.uea.ac.uk/b.php?i=34330416&m=89bde843c4e5&c=n
   4. https://canit.uea.ac.uk/b.php?i=34330416&m=89bde843c4e5&c=s
   5. http://www.cru.uea.ac.uk/cru/people/briffa/yamal2009/cautious/cautious.htm
   6. http://www.cru.uea.ac.uk/cru/people/briffa/yamal2009/
   7. http://www.climateaudit.org/?p=7588
   8. http://www.anglia.ac.uk/rae
   9. http://www.altman.co.uk/emailsystems

One thing that's clear from reading a hundred or so of these emails: the warmenist scientists are quite obsessed with, and defensive about, the Climate Audit web site by Steve McIntyre.  I've long linked to McIntyre's site as one of the best global warming skeptical sites I'd found.  I consider now that my choice has been completely validated!

I don't know anything about the Dr. Donald Keiller who wrote two of the emails in this thread, other than that I was able to confirm (via Google) that Dr. Keiller indeed exists, and works in the field implied in his emails.

My comments:

1. Dr. Keiller read of the controversy regarding the tree ring data being used in the temperature series on McIntyre's Climate Audit site.  He wrote to the authors for clarification.  They never responded, though they did post comments to the articles on Climate Audit.  Dr. Keiller finds McIntyre (a skeptic) more credible.

2. But more interestingly to me, Dr. Keiller questions the entire basis of dendroclimatology (using tree rings to infer anything about the climate).  The Wikipedia article I linked to explains the questions pretty clearly.  This is a big problem for the warmenists, as a major element of their models is the temperature series, which is in turn partly compised of dendroclimatology data.

3. Dr. Keiller is one of many scientists who have called BS on the warmenists creative use of statistics.  It's too bad that this hijacked email contained no more specifics.