Yuku free message boards

HaroldBalzaccio wrote:

---

bannedchef wrote:

---

"So what if the original data is missing. We have the internet now. Can't all those measuring stations around the world just send in replacement Excel files with the original raw data so the data extraction can be done again? If there was a careful methodology of data reduction the task should be simple and verifiable by 3rd parties."


I thought this too, but then realized it made too much sense and thus will never happen.

All of the original data was stored on paper and magnetic tape. The testing stations sent in everything they had at the time, and assuming that EAU would vault everything in the interest of science, they probably failed to keep bulky copies themselves. Even if they did, it's likely to be extremely difficult in the near term to find and provide the entirety of that original data to any third-party oversight.


There's no way in hell a competent third party would even consider converted data at this point. It'll have to be original paper/tape copies, or nothing.

I don't agree. All weather stations maintain data indefinitely, and older data going back to 150 years (of which the large data cache is claimed to be derived from) maintain data in a hand written format in actual log books, measured at several specific times every day when possible. They don't get rid of their data just because they sent off a copy to any particular researcher or University. Weather stations are among the most anal-retentive of all data handlers particularly since they strive to maintain accurate records of highs and lows for each and every day reaching back to the establishment of the station. Of course the data may not be there if the station was burned or destroyed, but even if a weather station is closed the records are deemed valuable enough to be archived at a local library or university. You can even go back to ship's logs in which daily temperatures and other weather information are logged daily at least once or twice a day at specific latitude and longitude. You could argue the accuracy of the data in terms of thermometer types from various stations, particularly in third world locations, but in general the records would not be intentionally discarded. The big problem is to re-do the legwork and find all the original data, transcribe it to electronic format, and to reprocess it. So the data isn't really lost but the odds of recreating the exact same set of "raw" data that was used may be difficult. Of course my point was that in today's world with rapid electronic transmission capability you wouldn't need to spend as much time as in the past to regather the data. And when I was talking about methodology I was talking about the English university group that processed the data. If they didn't document their methodology or procedure even they wouldn't be able to recreate the processed data as they did in the first place.

The misunderstanding here is the definition of the raw data. The real details of what is alleged in this story is contained in the CRU page on data availability, which is currently down precisely because of the problems caused by the illegal access. However, the information can still be found in, for example, a google cache of the page. The original CRU Data Availability, and I am using a snapshot of the page as it appeared on 21 Oct 2009 12:27:45 GMT, shortly before the hack.

We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data. The priorities we use when merging data from the same station from different sources are discussed in some of the literature cited below. Parts of series may have come from restricted sources, whilst the rest came from other sources. Furthermore, as stated in http://www.cru.uea.ac.uk/cru/data/landstations/ we have never kept track of changes to country names, as it is only the location and the station's data that are important. So, extracting data for a single country isn't always a simple task.

The claim that original data is now gone is simply incorrect. The original data is, as always, kept with the owners of that data -- the various national bodies who keep their own national weather and climate records. They still have it. The problem the CRU has is stating or replicating exactly how the raw data was folded into the homogenized data. This simply can't be repeated since the raw data, the data they received from various sources from around the world, is no longer in the possession of the CRU for re-use. That's not to say they can't re-acquire the data, given enough time and money, but even if they re-acquire the data it's probably impossible to process all the data and end up with the exact same results since you're talking about a 20 or 30 year process of folding raw data into the value-added data series which they hold sacred.


And what are the sources of the original data series? Presumably there's a more comprehensive list, but who knows?

What are the basic raw data used?

Over land regions of the world over 3000 monthly station temperature time series are used. Coverage is denser over the more populated parts of the world, particularly, the United States, southern Canada, Europe and Japan. Coverage is sparsest over the interior of the South American and African continents and over the Antarctic. The number of available stations was small during the 1850s, but increases to over 3000 stations during the 1951-90 period. For marine regions sea surface temperature (SST) measurements taken on board merchant and some naval vessels are used. As the majority come from the voluntary observing fleet, coverage is reduced away from the main shipping lanes and is minimal over the Southern Oceans. Maps/tables giving the density of coverage through time are given for land regions by Jones and Moberg (2003) and for the oceans by Rayner et al. (2003). Both these sources also extensively discuss the issue of consistency and homogeneity of the measurements through time and the steps that have made to ensure all non-climatic inhomogeneities have been removed. http://www.cru.uea.ac.uk/cru/data/temperature/