Scientific Data Disappears At Alarming Rate, 80% Lost In Two Decades 189
cold fjord writes "UPI reports, 'Eighty percent of scientific data are lost within two decades, disappearing into old email addresses and obsolete storage devices, a Canadian study (abstract, article paywalled) indicated. The finding comes from a study tracking the accessibility of scientific data over time, conducted at the University of British Columbia. Researchers attempted to collect original research data from a random set of 516 studies published between 1991 and 2011. While all data sets were available two years after publication, the odds of obtaining the underlying data dropped by 17 per cent per year after that, they reported. "Publicly funded science generates an extraordinary amount of data each year," UBC visiting scholar Tim Vines said. "Much of these data are unique to a time and place, and is thus irreplaceable, and many other data sets are expensive to regenerate.' — More at The Vancouver Sun and Smithsonian."
And in 20 years... (Score:5, Insightful)
And in 20 years, these results too shall be lost.
Re: (Score:2)
Unless it's published in a newspaper or magazine that is widespread. But printed matter seems to be on a decay.
Re:And in 20 years... (Score:5, Insightful)
That's not the point.
The actual published results - even if published in an obscure journal tend to stick around _much_ more.
Even old journals which go out of publication get their archives and the rights to distribute them bought - as there is some small amount of value there, in addition to the copies in the various reference libraries around the world.
The problem is that if you are wondering about that graph on page 14 of the paper that the whole paper rests on, you can't get the original data to recreate that graph.
This is a major problem because the only way to check that graph is now to redo the whole experiment.
Re:Why must you have their data? (Score:5, Interesting)
Re: (Score:2)
Reproduction of results isn't "add the numbers that they produced to see if they sum to the value they said it did". That isn't replication of science.
1. The data may not support their results. Without it, you can't verify that.
2. The data may be, let's say, 'adjusted' to give better results without admitting it. You may be able to show that by statistical checks, but you can't do that without the data.
Yes, you could completely re-do the experiment, but a) it may be historical data which can't be measured again (e.g. deceleration of a space probe from 1980 to 2000) and b) that may massively increase your costs.
Re: (Score:2)
"1. The data may not support their results. Without it, you can't verify that. 2. The data may be, let's say, 'adjusted' to give better results without admitting it. You may be able to show that by statistical checks, but you can't do that without the data."
And lots of people didn't seem to understand or care that this is why others caused an uproar when "original data" went missing from EAU and CRU right around the time of "climategate".
Without the original data, there is no way to reproduce the science to see if it was done responsibly. Without pointing fingers at anybody in particular, we know that in at least some cases, it is not.
Re: (Score:3)
Re: (Score:2)
"We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data."
Source: www.cru.uea.ac.uk [uea.ac.uk]
Re: (Score:3)
Re: (Score:2)
"That's why it was "impossible" for CRU to have withheld access to the raw data. Because they didn't collect it in the first place. Anyone who was actually interested in the data could always have gotten them from the same sources that CRU did."
I didn't claim that it was withheld. I merely stated that it was missing.
Further, initially others could NOT access that data, because National Meteorological Services in various countries refused to release the data to anyone else.
Granted, that situation has been largely fixed, but it WAS the situation when the "uproar" over the data was originally taking place. And without access to that data, there was simply no way to evaluate the quality of CRU's work.
According to the record, it is only becaus
Re: (Score:2)
For a while there was a perception that original data was "missing", but as you correctly point out, it was uncovered that most of the original data could (later) be obtained from the original sources. But it wasn't without a bit of a struggle with some of those sources.
Re: (Score:2)
Years ago, I explained in excruciating detail [dumbscientist.com] that this played absolutely no role in evaluating the quality of CRU's work because the majority of data in CRU's dataset "are derived from the same freely-available raw data sets used by NOAA and NASA." The Muir Russell review reproduced the necessary code in two days without any help from CRU.
And, of course, this isn't CRU's fault because “the authority for releasing unpublished raw data to third parties should stay with those who collected it.” Od
Re: (Score:2)
I didn't notice this comment before I wrote mine, otherwise I'd have been forced to correct this incorrect claim too. Again, the majority of data in CRU's dataset "are derived from the same freely-available raw data sets used by NOAA and NASA." Most of the data was already in the public domain, which is why the FOIA blizzard against CRU was so hysterically pointless [dumbscientist.com].
Re: (Score:2)
Repeat: access to the RAW DATA was NOT available. Only data that has already been "massaged" (to an unknown degree) was available before the "official" release, and that release was prompted by complaints about this very (and very valid) issue.
July 2011, and 5,113 weather stations, to be more precice [uea.ac.uk], in that particular release. Even then, some countries were holding out. (Most notably Poland.)
Whether the Muir-Russel review
Re: (Score:2)
"Most of the data was already in the public domain, which is why the FOIA blizzard against CRU was so hysterically pointless."
I agree with you that much of the data was already in the public domain. However, CRU could have avoided the FOIA requests if they'd simply handled things in a professional, reasonable manner, as opposed to one that was blatantly arrogant and dismissive.
They needlessly pissed a lot of people off. When you do that, you should not expect them to not piss you off in return.
Re: (Score:2)
1. Correctly, or incorrectly, there was a perception that data was missing or being withheld.
2. The importance of original data, which was perceived to be missing, was why people were raising a stink over it.
I'm not trying to say data was actually "missing", but it is true that some of it was not available. And CRU's documented attitude regarding requests about it contributed to an atmosphere of di
Re: (Score:2)
Previously, you could have used your ignorance as an excuse. Now you're just lying. And apparently neither you or Lonny Eachus have enough intellectual integrity to retract your latest steaming pile of civilization-paralyzing misinformation. This flood of misinformation isn't just staining "Jane Q. Public's" sock puppet legacy. It's also staining Lonny Eachus's real human legacy. Please stop.
Re: (Score:2, Funny)
Well, they're currently behind a paywall, so I don't see how most of us were even supposed to find them in the first place.
lulz (Score:3, Funny)
thats okay, the nsa has a backup
Concerning... (Score:5, Insightful)
Trying to ignore that a paper about the unavailability of scientific data is locked behind a paywall.
This is nothing new though, I do occasional conversion from ancient data formats, people need to pay better attention, imagine trying to read an 8" CP/M floppy today.
As libraries move to digital storage rather than the dead tree that's been fine for thousands of years they are inviting a catastrophe, possibly only one well aimed solar mass ejection from massive data loss.
Re:Concerning... (Score:5, Insightful)
Paper has its own issues. Talk to me about the durability of paper after you recover the books lost throughout time due to natural decay, burning (intentional or otherwise), floods, wars, and social forces (politics, religion, etc). Digital data can be easily copied and archived (when not behind a paywall, of course). It seems to me that redundancy is the best form of insurance against data loss. A solar mass is not going to wipe out every computer with a copy of important data on it, and all the relevant backups. And if it does, we're probably in a lot more trouble for reasons other than losing some scientific research.
Besides which, I sort of wonder if scientific data also follows the 80/20 rule. If so, how much are we really losing? I'm only half joking, of course, since it's difficult to ascertain the value of research immediately in some cases, but wouldn't it stand to reason that any important or groundbreaking research will naturally be widely disseminated, and thus protected against loss?
Re:Concerning... (Score:5, Insightful)
Digital data can be easily copied and archived
Can be. But mostly isn't.
Re: (Score:2)
Can be. But mostly isn't.
Kinda like paper.
Re:Concerning... (Score:5, Interesting)
Besides which, I sort of wonder if scientific data also follows the 80/20 rule. If so, how much are we really losing?
Probably not that much. I'm not claiming this is good, but I don't htink it's as bad as it appears.
If a paper is unimportant and more or less sinks without a trace (perhaps a handful of citations), then the data is probably of no importance since someone is unlikely to ever want it. Generally this is because papers tend to get more obscure over time and also get supereseded.
For important papers, the data just isn't enough: is a paper is important then it will establish some technique or result. In 20 years people will have generally already reanalysed the data and likely also independently verified the result if it is important enough. After 20 years I think the community will have moved on and the result will either be established or discredited.
I think the exception is for things that are "hard" to find such or non-repeatable such as finding fossils. Then again the Natural History Museum has boxes and boxes and boxes of the things in the back room. They still haven't gotten round to sorting all the fossils from the Beagle yet (this is not a joke or rhetoric: I know someone who worked there).
So my conclusion is that it's not really great that the data is being lost, but it's not as bad as it initially sounds.
Re:Concerning... (Score:5, Interesting)
I designed and built the equipment for scientific experiments that will never be repeated: cochlear implant stimulation of one ear, done in an MRI. This was safe because the older implant technology had a jack that stuck out of the subject's head, and which we could connect to electronics outside the MRI itself. But the old "Ineraid" implants have been replaced, clinically, with implants using embedded electronics and usually magnets. Those are hideously unsafe to to even bring in the same *room* as an MRI, much less actually scan the brain of a person wearing one.
So that experiment is unlikely to ever be repeated. Losing the data, and losing the extensive clinical records of those subjects, would be an immense loss to science. There is especially historical data from decades of testing on these subjects that show the long term effects of their implants, or of different types of redesigned external stimulators. That data is scientifically priceless. When I started that work, we used mag-tape for data, and scientific notebooks for recording measurements. I helped reformat and transfer that data to increasingly modern storage devices several times. We went through 3 different types of storage media in 10 years, and I remember having to write software to allow Exabyte drives to find the end of the tape and add data. (Exabytes had no End-Of-Tape marker.) Preserving that data.... was a lot of work.
Re: (Score:2, Interesting)
No it won't because it 20-30 years we will be able to do gene therapy to "grow" or "regrow" the stereocilia and hence cochlear implants will be considered as barbaric as medieval blood letting. Consequently the data will only be of obscure historical interest.
Re:Concerning... (Score:5, Insightful)
Wishful thinking.
Let's make a deal: *first*, the gene therapy works. *THEN* we assume we can afford to lose the data the grandparent talks about.
Re: (Score:2)
Re: (Score:2)
We went through 3 different types of storage media in 10 years, and I remember having to write software to allow Exabyte drives to find the end of the tape and add data. (Exabytes had no End-Of-Tape marker.) Preserving that data.... was a lot of work.
Do what everybody else does. Encrypt it using a strong password, then upload it to The Pirate Bay or the Semi-centralized Filesharing Platform Which Shall Not Be Named, and call it "insurance file xxxxx".
Re:Concerning... (Score:5, Insightful)
It needed some very special transcription technology but once in the clear and fed to modern 3D seismic software it revealed a lot more than the original reports gave.
Retaining old reports is nice, retaining old raw data even nicer.
Re: (Score:2)
Problem is that this data is funded and paid for while the research is active. After it's done the data is put in the closet so to speak. No one is being paid to go back every 5 years and re-read the data to convert to a new media format. Paper doesn't help because it can't store all this data anyway.
Ie, satellite collects lots of data on solar activity, it all gets stored onto 9 track tape drives, which then end up on a minicomputer for data analysis, image enhancement, etc. Project funded through gove
Re: (Score:2, Insightful)
The problem is not just an issue of digital storage, but also a problem of redundancy.
In the "old days", people understood and accepted the risk that a paper copy would be lost. In fact, it was a GIVEN that they would eventually be lost (or damaged or misplaced or stolen or checked out and simply never returned). So multiple copies were kept because centuries of experience dictated that some copies would be lost no matter how strong, carefully maintained and well preserved the originals were.
Nowadays, peopl
Re: (Score:3)
Re:Concerning... (Score:4, Interesting)
There was a documentry on the radio this week about the loss of letter writing as a form and how alarmed biographers were getting because it's getting very hard to trace someone's life, thoughts, actions etc without a paper trail as stuff like emails, digital photos etc generally get lost when someone dies.
Personally, I find the increasing rate of loss quite alarming - so much of our lives are digital and so little is properly curated with a view to future access. We know so much about the past from old documents, often hundreds if not thousands of years old but these days we're hard pushed to find something published ten years ago.
Re:Concerning... (Score:5, Informative)
I'd go to one of the British deposit libraries and ask to see their copy; deposit libraries have existed since the Statute of Anne in 1710. The British Library has 28,765 books and 1,480 journals in its catalogue from 1910...
Re:Concerning... (Score:5, Informative)
Re: (Score:2)
I have a collection of books on my shelf that date from the 1870s, all in top condition and never been stored in any special way.
Re: (Score:2)
Re: (Score:2)
If I didn't already know of two companies that could do that I'd look in the yellow pages. I get your point though and there are older or rarer formats than that which would require a bit of legwork or possibly even reverse engineering.
Re:Concerning... (Score:5, Insightful)
It's not that it's a new problem as such, it's that for the first time in history we have a simple way to solve it, yet we have stupid greedy rich people who sponsor and enact laws to stop us from solving the problem.
The way to solve the problem is through massive duplication of all the data, over and over again through time. We have the technical means to do this on an unprecedented scale.
Even 1000 years ago, people had to painstakingly copy books, by hand, one at a time. And after a handful of copies were produced, there still weren't enough to guarantee that most would survive the ages, wars, fires, censorship, etc. So we generally have tiny collections from the past.
But now it's digital data. Anyone could copy it. We could have millions of copies of some obscure scientific work, all perfect duplicates. If even 0.1% of these copies survive, that's still thousands of copies.
And what do we do? We let a bunch of 1 percenters, who themselves barely know how or care to read, sponsor draconian copyright laws to stop eeryone from copying all that stuff, just on the off chance that they might copy a bunch of songs or movies that are outmoded within two years. And the commercial scienrific pulishers are some of the worst.
It's pathetic.
Re: (Score:2)
We let a bunch of 1 percenters, who themselves barely know how or care to read, sponsor draconian copyright laws to stop eeryone from copying all that stuff, just on the off chance that they might copy a bunch of songs or movies that are outmoded within two years. And the commercial scienrific pulishers are some of the worst.
Commercial scientific publishers do indeed tend to be bottom-feeders, but if I'm understanding the article correct, they're not the root cause here - the issue is not that articles are
Re: (Score:2)
Re:Concerning... (Score:4, Insightful)
That's one take on things.
On the other hand we have already lost so much stuff over the centuries that perhaps what I just said is idiotic alarmism. After all we have rebuilt western civilisation after the fall of Rome(that just took the Dark Ages) and we didn't all die off after the Great Library of Alexandria burned down. The stuff that gets often replicated will propably not be lost. But let's hope it isn't a retweet of Miley Cyrus' knickers.
Re: (Score:2)
the dead tree that's been fine for thousands of years
Not so fine.... The Alexandria library fire was perhaps the most catastrophic loss of human knowledge ever. For example, it destroyed the details of a heliocentric theory, which was postulated by Greek astronomer Aristarchus of Samos, millennia before Copernicus brought it to mainstream.
Re: (Score:3)
Just because it was sitting around in a library is no guarantee anything would have happened with it.
Re: (Score:2)
Fair enough, it was just one piece of a massive collection of knowledge lost.
I was just questioning the claim that paper has "been fine for thousands of years".
While paper (if carefully stored and looked after) is more durable than any digital media invented so far, you can't ignore the many advantages of digital media: extremely easy and quick to copy with no loss of quality, possibility to do that remotely, far less bulky...
Re: (Score:2)
paper (if carefully stored and looked after) is more durable than any digital media invented so far
Except for punched cards and maybe other extremely bulky storage devices, now that I think about it.
Odd coincidence... (Score:3)
Some years ago I picked up a copy of "Dark Ages II -- When the Digital Data Die" by Bryan Bergeron (2002) but only now have gotten around to finishing reading (for some reason I never got past the first chapter at the time). When I bought it I had just had my own experience with the not-so-long life of digital data (some CDs I'd burned a few years earlier were already unreadable). The book's a bit dated (it says that there are many people out there with Zip drives connected to their PCs) as, obviously techn
Re: (Score:2)
Personally, I think we're just fine if everything is converted to bits and we remember that there is no guarantee that a set of bits might not be damaged or lost.
Just like books - there's no guarantee those won't fall victim to water damage, or fire, etc. You have to take care of it, and guard against the applicable failure modes. With digital data this is just as possible, but the techniques are as different as the failure modes.
Lifecycle management (Score:5, Interesting)
Will this change? Probably not.
Re: (Score:2, Troll)
Vines is calling on scientific journals to require authors to upload data onto public archives as a condition for publication.
If authors put their data into the public sphere, people might notice how much of it is fudged.
Re: (Score:2)
The answer to both problems is to publish everything in the Journal of Irreproducible Results
Re: (Score:2)
Precisely (Score:2, Insightful)
This is bang on. As a system administrator for a STEM department at a Canadian institution, my budget is 0 for data retention. Long term data retention is just not in the mindset of researchers.
Re: (Score:2)
One of the places that I've worked did various sorts of science / engineering type project work. Quarterly backups of filesystems were archived indefinitely. Even if the data was staying online, at the completion of every project an archive was made of the data on a minimum of two pieces of backup media along with various bits of metadata regarding the media and data. The archival copies were tested by restore and diffed before actually going into the archive. Of course they kept examples of the differen
Re: (Score:2)
Just increase the disk array size and copy the data as it grows to larger and larger storage systems. Data that's offline is useless.
Re: (Score:2)
I don't think you understand what "archive" means.
I beg to differ.... We archive "forever" (Score:2)
Hello,
Our mindset at my research institution is very different. We generate a certain amount of data per year (several terabytes), but the cost of storage decreases so fast we just copy old data onto new media and never delete ANYTHING.
In fact, we consider the cost of actually figuring out what data to delete to be higher than simply buying more storage.
I would not call it "well-indexed" however.
Our backup strategy is tailored to the nature
If... (Score:2)
On the bright side... (Score:5, Interesting)
Re: (Score:2)
There are many entities with vested interest to keep the data that supports their point of view, or their profit motive or their meal ticket alive. For example data collected meticulously by a underfunded biology professor about the allotropic speciation of the salamanders around the lake hole-in-the-mud would disappear in a jiffy. But flawed research supporting the efficacy of a patented clot busting drug would be perpetuated. Epidemiological studies showing the adve
So...? (Score:4, Insightful)
I'm a researcher and I don't have time or space to keep old data as I'm generating too much new data. We work hard to maximize the use of these data and analyses when we write and publish papers. If this was talking about the papers (or presentations), that were the product of the data, being lost at this rate it would be one thing, but the raw data isn't usually very useful to anyone without context or knowledge of subtle and poorly documented technicalities. This just seems like ammunition for the climate change deniers to bitch about. It's unreasonable to keep the old data indefinitely without a massive public repository that will be poorly indexed and organized.
Re: (Score:2)
Re: (Score:2)
Wouldn't documenting your experimental method be part of your job? There's really no reason why raw data should be this mysterious entity that nobody can possibly understand unless they were there when it was collected. IMHO, your results -- whatever they are (I only hope it doesn't have anything to do with a drug that physicians might be prescribing to patients) -- are highly qu
Re: (Score:2)
On several occasions I have tried to get data from researchers. Most of them guard their data jealously and will give any number of excuses for not distributing it, including:
- telling me that I don't have the knowledge or context to properly understand the data
- fear of me stealing their precious, precious secrets
- fear of me "misrepresenting" the data
- (unspoken) fear of me finding problems with their data or analysis
Unfortunately, most researchers live in a very closed, secretive world and fear exposure.
Re: (Score:2)
On several occasions I have tried to get data from researchers. Most of them guard their data jealously
I should note that this almost certainly violates the terms of publication for most journals, and possibly the terms of their research grants as well. I actually had one professor complain to me that it was "his" data and I had no right to it - conveniently ignoring the fact that he (like me) was being funded by taxpayers (albeit in different countries). My views on this subject aren't particularly radic
what the hell? (Score:2, Insightful)
I think it is ridiculous that Slashdot's keep posting articles that are behind paywalls. How the hell are we supposed to see them? Do you expect us to pay for subscriptions to services we'd only use once? you, OP, are out of your mind. articles such as this should be rejected as most users, if not all, can't even access the story. This site really has gone down hill in the last few years, over populated with clueless simpletons, frauds, so-called armchair IT experts and -obvious- subscription pushing trolls
Not the worst of it (Score:2)
Many things are based on this data... and when the data is gone it cannot be audited which makes it impossible to verify the finding of the data which is later simply referenced... but the data upon which it is based... *poof*
This practice also gives a free reign to fraudsters because if you don't catch them quickly they can claim the data was just in their other pair of trousers.
Re: (Score:2)
This practice also gives a free reign to fraudsters because if you don't catch them quickly they can claim the data was just in their other pair of trousers.
No, the timespan is 20 years. Within 20 years the results will either be sunk without a trace, disproven or replicated. A fourth option is very unlikely.
For example, I doubt the original measurements of superconductivity are still around. If they are, they'd be interesting from a historical perspective, but you could replicate the results yourself with
Re: (Score:2)
In most cases I would agree. However there are some large tables of data that are at least that old that are referred to currently. And they have not been replicated since.
Further, the tables are themselves not raw data but modified data with the raw data and methodology no longer available.
Its just entropy (Score:3)
I still have the raw data (Score:2)
that I used for my paper 15 years ago. It is on a tape, that is somewhere in a drawer, that I have no tape drive for. On the other hand, the LaTeX file and the C and FORTRAN programs I used to evaluate and create the data and write the paper are still on a hard drive that is running on a computer in my network and I can access it right now. I probably can*t compile the the program without change (was written for Solaris and DEC machines) and maybe not even run LaTeX on it without getting some of the include
is/are (Score:5, Interesting)
Whichever side of the "data is" vs. "data are" argument one falls on, I hope we can all agree that mixing both forms within the same sentence is definitely wrong.
Re: (Score:2)
On the off chance you're reading this: Wrong. Much of these data are... and is thus irreplaceable. You can argue that "is" refers to the "much" rather than the "data" -- it's ambiguous due to the inconsistency -- but "much" definitely refers to the "data," and "much" is singular.
Misleading figure caption (Score:2)
Some idiot sub-editor wrote a misleading figure caption here. The article (which I've read) says nothing about how data is lost with age. It only says something about how much data is lost for papers of a given age as of now.
In other words it does not mean that in 10 years time, 10 year old papers will have such drastic data loss. The world 20 years ago was a very different place in terms of communication, scientific practice, and data storage than it was 10 years ago or is now.
The Slashdot article repeats
And this study will be lost as well (Score:2)
a) because it's behind a paywall; and b) how can the original data even hope to be located when a majority of the population can't even read the paper?
Lost forever (Score:3, Interesting)
Or maybe impossible to regenerate (for certain values of impossible). I remember reading a classified technical report (dating from the 1940s) related to military life-jacket development, wherein the question arose as to whether a particular design would reliably turn an unconscious person face-up in the water. The experimental design used was to dress some servicemen (sailors, possibly, but I don't recall) in the prototype design, anaesthetise them and drop them in a large body of water, checking for face-down floaters to disprove the null hypothesis. Somehow, I don't think that those data are going to be regenerated any time soon. I hope to God not, anyway.
Dave...Dave... (Score:2)
I'm....losing...my..mind..Dave......Dave....Would you like me to sing a song?
And It Makes Me Wonder (Score:2)
The very fact that "Much of these data are unique to a time and place, and is thus irreplaceable, and many other data sets are expensive to regenerate.", makes me wonder if this could even be considered "scientific data" anymore. Since the data is unique to a time & place and irreplaceable, it would completely destroy the reproducibility aspect of the scientific process. Given that, should the lack of reproducibility mean that lost scientific data should be redefined as experimental data or hypothesis
Re: (Score:2)
The very fact that "Much of these data are unique to a time and place, and is thus irreplaceable, and many other data sets are expensive to regenerate.", makes me wonder if this could even be considered "scientific data" anymore. Since the data is unique to a time & place and irreplaceable, it would completely destroy the reproducibility aspect of the scientific process. Given that, should the lack of reproducibility mean that lost scientific data should be redefined as experimental data or hypothesis data? It also brings up the idea in my mind that scientific data has a half life since it can degrade back to hypothesis or experimental data if not properly stored.
Completely incorrect! How can you study "how X has changed over time" if you don't have data from other times? It is also impossible in many, if not most, cases to gather such historical data in the present time.
Re: (Score:2)
But you can't study "how X has changed over time" if you don't even have the original data that you'd be comparing it to?
Still, that's not really my point. I'm saying that without the original data (and remember this is data that cannot be gotten again even with effort), one cannot re-do the study and see if the results are reproducible. Therefore, the entire scientific process is impossible with studies that have lost & irretrievable data.
Re: (Score:3)
Since the data is unique to a time & place and irreplaceable, it would completely destroy the reproducibility aspect of the scientific process.
This gets tricky in some fields, however. I work in a field where generating the data is a notoriously difficult and haphazard process, subject to many non-experimental variables, such that the use of a different pipette or stock solution can make the difference - or even just the speed of the researcher's manual labor. Temperature and humidity play a role too,
Re: (Score:2)
The Rosetta Project: building a 10000 year library (Score:4, Interesting)
The Long Now Foundation has devised an interesting mechanism for storing important information which, although not optimal for machine readability, is dense and has an obvious format: a metal disk etched with microprinting, whose exterior shows text getting progressively smaller as an obvious way of saying "look at me under a microscope to see more":
http://rosettaproject.org/ [rosettaproject.org]
I highly recommend reading The Clock of the Long Now if you're interested in the theory and practice of making things last.
There's a bigger problem (Score:2)
Before science gets hot and bothered about the loss of data scientists need to do something about the quality of the data they produce to begin with. Frankly given the complete lack of quality controls that a lot of scientists use the loss of their data is probably for the best. Depending on the field as much as 60% of all scientific research cannot even be reproduced. Work that cannot be reproduced by another team is far from isolated to one field either:
http://online.wsj.com/news/articles/SB10001424052970 [wsj.com]
Best legitimate use of P2P (Score:2)
Universities should band together to distribute all data from published material on P2P networks so it's redundantly stored at mulitple locations. This has the side-benefit of making a legitimate use of P2P obvious.
Need extremely fine, fine print (Score:2)
As a former paper industry professional (recycled pulp), Paper is fine except that people limit its use to readable font. That is what led to Microfiche (which is now being dumped by the truckload at recycling stations as "obsolete tech"). If you printed a hard copy of everything either to microfiche or extremely small 1-point font, you could store the data in a type of seedbank or gene bank.
A salt mine may not be appropriate, but I'd like to start a business where everyone could send their hard drives t
WE ARE LIVING IN A FUTURE DARK AGE (Score:3)
[OP] "disappearing into old email addresses and obsolete storage devices, a Canadian study (abstract, article paywalled) indicated
Well so much for the study. Money changes everything. Eventually one hundred thousand copies of the abstract will exist on the Internet, but the authors' future descendants will find only only one actual link that leads to content, which terminates at a page saying "this domain is for sale".
You'd think that even science data of extremely low bit rate such as original weather station temperature data [blogspot.com] should be out there somewhere. A lot of other people did too... but all that is available now might be "value added" ajusted data. Not an evil conspiracy per se, it's human nature at it's best and worst.
A handy chronology of the history of data retention:
[2500BC] King Fuckemup boldly slew the enemy and I, Scribe Asskissus hath inscribed it in stone. He is an asshole who owes me back wages."
[1500] "With quivering quill [wikipedia.org] I will write mine own data."
[1866] "Data published at great expense into leather-bound volumes. Dust sold separately."
[1970] "This is really important. we should print it and store it in a binder."
[1971] They didn't.
[1983] "I'll write it to floppy disk with a notsosticky label"
[1985] "After a long and desperate search, the label has been found!"
[1987] "Unlabeled floppy disk keeps coffeemaker level."
[1995] "Roxio CD storage is forever, and Real Scientists don't close their data sessions."
[2003] "Microsoft Word has experienced a problem updating from an older document format and will now close. Save your work as soon as possible."
[2005] "I'll just email it to myself and shut the computer off immediately, then pick it up at work."
[2009] "Yes, three copies! In the safe. There was a fire. Yes, inside the safe. It was a fireproof safe, so no one noticed."
[2010] "This is really important. I should print it and store it in a binder. But my ink cartridge is dry."
[2013] "Our data has been uploaded to the Cloud where it will live forever."
[2500] "King Grapeape slew the primitive humans and buried their statue on the beach. I, Scribe Anthopoapologus hath incribed it in stone."
Perhaps the most mystiying data retention escapade of Modern Times [youtube.com] is the missing Apollo 11 SSTV moon tapes [wikipedia.org] which contained a multiplexed stream of raw telemetry and the original slow-scan TV signal broadcast from the moon. Not 'missing' really, rather we know they were re-used and recorded over because everyone assumed it was someone else's job to ensure that at least one copy was in a safe place. While the earth station operators dutifully sent their tapes to NASA where the sharpest signal of the moon landing was sure to be perserved for posterity (not), fortunately there were some librarians on duty, and you can aquire DVDs of the moonwalk [honeysucklecreek.net] with better quality than the recordings you've seen in countless movies -- an 8mm film camera pointed at an original SSTV monitor at Honeysuckle Creek, and the best quality scan-converted version.
In the Foundation series, Asimov envisioned Gaia [wikipedia.org], a world in which a telepathic network of sentient (and sensuous) beings kept a 'working set' retrievable data in-memory -- but also via access to progressively less and non-sentient objects, such as plants and even rocks -- a vast archive. Ask the mountain, it will answer in time, a long time.
Our own Earth has a Gaia storage mechanism, a record of its magnetic field over geologic time stored as polarization in crystallized lava floes [youtube.com]. But it i
The NIDDK was aware of this years ago. (Score:2)
The NIDDK was aware of this years ago and had commissioned a feasibility study on creating a storage mechanism that all grant paid research would have to use. Unfortunately after a successful feasibility study the reviewers for the follow up real grant responded with "I do not see the scientific value of this research" and the grant went away with Vanderbilt as the only applicant. I've heard through the vine that someone picked up a new similar grant to work on it, but I haven't seen anything from it yet
how this happens (Score:4, Informative)
Our scientific research system is built around the process of joining a lab, mastering the work there, and then leaving. There are very few long term research partnerships. The people who stay in place are the professors, who generally do not do the research work.
So you join a lab, produce a few terabytes of data a year, pull a few publishable nuggets out of that and then leave. I have a few backup hard drives that move around with me with what I consider my most important data, probably total 1/10 of the data I have taken. After a few years, this data is really unimportant to me as the labs I have left have done a good job of continuing the research and I have to spend my time and money on something else.
The original data is eventually overwritten by researchers a few "generations" removed from me and that's the end of it.
Yes and? (Score:2)
How is that different from the previous state of affairs?
Before digital age, Scientists would have work booklets that would get lost or destroyed when they change job, or when they become too numerous.
Drawning in an overflow of data is about as useful as having no data at all. It could be argued that forgetting is actually a good thing that puts forward important matter, those that we care to keep because they are valuable. Sure, some valuables get lost in the process, but anyway, who would go sort trough a
Magnificent Desolation - Behind the scenes (Score:3)
When John Knoll (yes, THE John Knoll, co-creator of Photoshop and VFX wizard extraordinaire) wanted to reproduce the Apollo moon landing in CG he ran into a small problem. He went to NASA to obtain the telemetry data for altitude and orientation but apparently the data had been tossed a long time ago. However, he was able to find physical prints of graphs of the telemetry channels. So he scanned them in, made them an underlay in a 3D modeling program, and painstakingly traced them by hand in order to extract the data. The results can be seen in Magnificent Desolation Apollo 15 landing sequence. And BTW, that's his modeling work for the lander too.
Ah memories... (Score:2)
And then I think about the actual project I was working on, which produced something like 1GB/hour every hour every day. Only a fraction of the raw data really made it through cooking, but if there turned out to be a flaw in that initial processing our a
that's the difference (Score:2)
We Need Legacy Support (Score:2)
We Need Legacy Support - I keep saying this and the little kids keep dissing me but we desperately need to maintain legacy support. In 30 more years what else will we have lost through rapid obsolescence?
Companies like Apple and Microsoft need to reach back and provide it all the way to their earliest systems forward. We need to be able to access our old data and that means being able to run our old applications.
Congress needs to put forth the legal framework that allows all software to be legal cross comp
floppy and laser disks (Score:2)
We just rescued some data (Score:2)
A couple of us just rescued some 20-year-old data that had been stored on 3.5 inch floppies. We actually had to go to one of our old retired colleague's houses because he was the only person we could find who had a computer with a floppy drive capable of reading them. Even so, some of the data was unrecoverable.
I know probably the best option right now for preservation in digital form would be several copies on CD/DVDs of the proper archival type, but I'm wondering if there are any free online services suc
It's a matter of organization (Score:2)
Re: (Score:2)
add revenue on data.. hmph..
Re: (Score:2)
Interesting and sad.
Those links would probably also be great for NSA & Google like data mining to monitize those users too.