Digitizing 100 Years of Astronomical Data

Maximum Prophet writes to mention that a collection of glass plates containing astronomical information from the late 19th century through the mid-1980s is being considered for digitization. "The accumulated result weighs heavily on its keepers on Observatory Hill, just up Garden Street from Harvard Square: more than half a million images constituting humanity's only record of a century's worth of sky. 'Besides being 25 percent of the world's total of astronomical photographic plates, this is the only collection that covers both hemispheres,' said Alison Doane, curator of a glass database occupying three floors, two of them subterranean, connected by corkscrew stairs. It weighs 165 tons and contains more than a petabyte of data. The scary thing is that there is no backup." I'm sure that anyone with a spare $5 million or so would be welcomed with open arms.

That's quite a bit.

165 tons of glass plates?

Sounds like a typical lunch clean-up after Rosie O'Donnel.

Sorry. I'm truly sorry.

That's at least...

a thousand minibuses full of magtape.

Glass plates will outlive the digital"backup"

now there is some irony.

Re:Glass plates will outlive the digital"backup"

Are you sure about the stability of glass plates? I hear a lot of people have real trouble with windows stability! Sorry, I'll go now....

Luckily glass isn't a liquid....

Luckily glass isn't a liquid so they won't distort.

Re:Luckily glass isn't a liquid....

Glass *is* a liquid (sort of) [wikipedia.org], but it does not flow, which is what I think you were getting at.

Re:

Glass does flow...a look at 80-150 year old windows will show this.

Please follow the link in the post you were replying to, and/or look this up on snopes. This is not true. 80-150 year old glass is simply warped due to less advanced manufacturing techniques, and often thicker at the bottom because window makers tended to

Re:Glass plates will outlive the digital"backup"

now there is some irony.

But currently they also makes them vulnerable to a single point of failure (as indirectly pointed out in the article). If you have some data that has any real value for you then having only one copy (or only one storage facility) isn't any real protection whatever method you use. In this case we have data that would be readily accepted for backup by organisations all around the globe and barring a worldwide upheaval the safety of the data would be much better than any single glassplate could offer.

Of course the ideal would be if we could develop a cheap digital permanent storage that had guaranteed physical longevity, say several millenia. That combination would allow easy dissemination of the data and safety by using a multiplicty of sources.

Re:

"Of course the ideal would be if we could develop a cheap digital permanent storage that had guaranteed physical longevity, say several millenia"

But would it be compatible with Office 5007 or OO v3452.23?

Re:Glass plates will outlive the digital"backup"

So what? Copy the digital version onto a second set of disks when it comes close to expiring.

Lossless copying means that given a little bit of maintenance, expiration of digital media is a nonissue.

Re:

Ever tried to maintain archival backups for a petabyte-worth of data? You would need a team of operators, millions of dollars worth of storage hardware, expensive climate-controlled facilities. This will not be a one-time expense either. The best way to ba

Re:

And your photographic copy would A) degrade over time and B) lose quality with each copy. IMHO that's not a very good archive. Moreover, in order to slow the inevitable decay that comes with time and reactive chemicals on paper/plastic/metal/whatever, you'

Re:Glass plates will outlive the digital"backup"

Ever tried to maintain archival backups for a petabyte-worth of data?

Yes, as a matter of fact. Definitely a lot of work is involved, but do you believe that you wouldn't need a team of document managers, millions of dollars worth of floor space, and expensive climate controlled facilities for archival of microfiche? You most certainly do. It's a lot of data. Period. No matter what you try to do with it, it's a lot of data. It's going to require a lot of resources. That's just a fact of life.

Anyway, noone in their right mind would choose microfiche for that type of data. If you're only storing plain text pages it's adequate (though I still don't think it would be the "right way to do it" in this day and age), but for photographic plates? Not going to work.

Microfiche is vastly overrated, in my opinion. My current project involves taking 2 floors worth of 30-50 year old microfiche and scanning it, OCRing it, and PDFing it. Yes it certainly does age. Quite poorly, in fact. The quality is absolutely terrible compared to the paper versions, some of it is stuck together, and indexing and cataloging it is a nightmare all of its own.

Yes, there are challenges in the digital world too, but most are easily surmountable given a little bit of common sense in understanding that digital is not magic. It doesn't mean you can "fire and forget". The documents will still require maintenance, cataloging, protection and monitoring. Format obsolescence is very nearly a nonissue, it is blown way out of proportion. That's where the "maintenance" comes in. The key benefit of digital is that you can and should losslessly upgrade your format whenever obsolescence is becoming a concern. Formats do not disappear overnight and suddenly everyone forgets what to do with them, you have plenty of time to make your transition if you're paying attention (which you must be: again, digital is not magic).

Re:

By that time, other techniques will be available to copy the digital archive over. Heck, it might even be possible to make a copy of the digital data on glass plates, complete with descriptions of the used protocol.

It's true that digitized data is more

I can't wait to see...

...all the alien porn I've been missing!

This sounds like a job for Google

Google provides views of the Earth, Moon and Mars, why not stars? If the information was made available for them to deliver to their users, they might be interested.

Re:

Don't worry, it's coming. I've seen previews of Google Sky at a couple of astronomical conferences so far. Also, check out partner number four for the Large Synoptic Survey Telescope [lsst.org].

A Million People With $5

This seems like a great opportunity for either corporate sponsorship, or a grass-roots donation drive. In all honesty, $5 million isn't a whole lot of money for the likes of any real corporation, and it probably wouldn't be that hard to raise it through small donations from individuals. Espectially if you could ascribe names to some or all of it. How would it feel to be able to personally identify which plates you paid to have scanned? (this image of the Crab Nebula brought to you by John Smith) I'm surprised Paul Allen or Richard Branson aren't all over this like stink on shit.

Google

I'm sure that a company like google would be MORE than willing to fund a project archiving these. The positive press, proliferation of their intended "do no evil/good guy/just another bunch of geeks" image, having their name on a major scientific project would easily be worth the investment.

Re:

I'm sure Microsoft has a way to store this data, no doubt with an "open" format. Things like this are too important to trust to anyone but Microsoft!

This is a job

for Google. Man if I had a spare 5 million I'd be all over that, I love data.

InfiniBytes

contains more than a petabyte of data

Glass photographic plates, especially from silver emulsion, are analog at extremely fine granularity. Effectively molecular, depending on how flat the glass surface was settled from its molten liquid state. The features of its silver oxide crystals, laid in place by individual photons arriving from vastly distant stars, could be meaningful at less than a nanometer. Especially when measuring extremely subtle influences, like the gravity from one distant star bending the light of another distant star, measured across a century in which those stars lost gravitational mass, for comparison.

There is a practically infinite amount of data on each of those plates, limited by our precision in measuring them. It's a smaller degree of infinity than that of the sky. But the original infinite sky is lost. While the plates' lesser infinities are impossible to replace, and all we'll get to use to look back across all the billions of years we saw in a long century of them.

Re:InfiniBytes

here is a practically infinite amount of data on each of those plates, limited by our precision in measuring them.

And limited by the lenses/mirrors, and limited by atmospheric effects, and inconsistencies in the glass, and the silver, and, and....

I can't testify to the quality of the glass negatives, but I can testify to the fact that as much as people like to believe, even the best modern analog capture sources aren't anywhere near practically infinite, even in the best laboratory conditions.

Re:

Well, the lenses/mirrors that are now lost to history do introduce noise. But the atmospheric effects, and inconsistencies in the glass and silver, and probably much of the "writing" noise from the optics do all hold the possibility of being filtered out.

Re:

Having worked with holographic media for decades (which is about as fine resolution as you can get optically) the maximum resolution is on par with the grain size 40 nm (Afga 8e75) and considerably worse both due to the wavelength of light and the expansio

Re:

The optics can be subtracted from the recording if their specific details are known. That's the beauty of analog recording: it is infinite.

And when the sampled phenomenon is as vast as all of interstellar space, that infinitude is relevant.

Re:

It's obviously not infinite. There is some number of atoms involved in the image, and they can only take on a finite number of states and still be an image on a plate. Plus of course the pigeon hole principle should make it pretty obvious that the image on

Re:

I said "practically infinite". That's because the data is not limited by the storage medium, but by our limited ability to read it.

It's funny how many people jumped on me for my pointing out how much more than a "petabyte" is on those plates. But no one ha

Re:

Ok, let's say that we've managed to get the original optics. We've discovered that, due to slight focusing error (which is fixed) and atmospheric effects (which are chaotic), each point on the photographic plate was blurred by 0.01 mm. How do you recover t

Re:

"Blurred" is an oversimplification. What we could get from the original optics is the convolution by what optics signal the recorded signal was changed. So we could deconvolve the data on the plates. In a hundred years, I expect our skills at such deconvol

Re:

In another hundred years that kind of data collection will probably be easy. But still extremely valuable, because the data recorded in them is irreplaceable.

If the astronomers who recorded these plates weren't anal, then astronomy wouldn't be advanced eno

sounds familar

anyone with a spare $5 million or so would be welcomed with open arms

That's what she said!

Planets et al...?

Maybe we'll get some data on possibly extra solarplanets from this?

All this stuff should be digitized and made public

When I completed my Astronomy masters access to publicly available data from various sources (most notably NASA data made free to the public) was a real boon. It meant we could do analysis on actual real data instead of artificial or sanitized textbook material. A couple of the students built on this to do some original research. (Sadly that's not the way I went, as my time was more limited).

There are also lots of amateurs out there running a wide variety of very specialized packages to do everything from discovering asteroids to keeping tabs on the brightness of stars and watching for supernovae.

Harvard can handle the burden

From here: http://www.hno.harvard.edu/guide/finance/index.htm l [harvard.edu],

This:
Harvard University's endowment, valued at $25.9 billion at the end of FY 2005, is a collection of more than 10,800 separate funds established over the years to provide scholarships; to maintain libraries, museums, and other collections; to support teaching and research activities; and to provide ongoing support for a wide variety of other activities. The great majority of these funds carry some type of restriction.

I think they can scare up the change.

Re:

Absolutely correct. According to records, Harvard saw their endowment fund appreciate over 16% in a single year (FY2005). Sixteen percent of $30 billion is nearly $5 billion which would allow them to quite easily fund this project. Even if Harvard has t

A great idea.

If they manage to standarize a century of these plates, it would significantly extend the time range of data to digitally extrapolate and detect objects previously missed. Just to speak of mapping our own cosmic backyard, a significant amount of slow moving, previously undetected Kuiper Belt Objects, for example, would more easily pop into view. Surely a bunch of comets, too.

Clyde Tombaugh captured Pluto several times during his three decades long hunt for the elusive Planet X, but failed to put the pieces together. If he had had digital technology, he would have shaved off at least a decade of effort. So imagine all the extremely useful raw data still stored in those plates.

More than just a flat scan

There is more to this than simply scanning a flat image. The emulsion on these plates is a three dimensional medium, and different data can be extracted depending on your focal depth into the the emulsion. I believe David Malin did much pioneering work on this kind of thing, including the use of different layers for unsharp masking.

There will be information in the plates that is not yet part of human knowledge, and a simple scan of one focal plane is not going to get it all.

Certainly it is worth taking backup images of these plates in any way we know how, but we should remain aware that, as of today, no technology exists that will make exact duplicates of them, so great care should always be taken to preserve the originals.

GoogleSky

Seriously, let Google index not only that collection, but any stellar image information and launch GoogleSky.

Re:Why does it have to cost so much?

Those plates as well as being old and delicate contain a LOT more information then a piece of paper. Considering that something less then 1/4 the size of the period on the end of the sentence is important your scanning at a much higher resolution.

Re:Why does it have to cost so much?

> I spearheaded a "digital backup" of around 90 filing cabinets of papers ...
> It took 2 years and way WAY WAY less than $5,000,000 to do it

500,000 plates. Over 2 years, assuming 50 wks/yr means just 5000 plates need be scanned per week. 1000 plates per day. 125 plates per hour. And this is large, fragile glass with really high data density, so you have to be a) careful in handling and b) use slow high-res scanning.

Let's take a guess that it takes only 10 minutes per plate (to fetch, tag, load, scan, and return). So we need only 20 people to scan 125 plates/hour.

Well, assume 20 scanning people and 1 IT guy handling the sysadmin work for the petabyte storage. Also one scientist/manager. Take a low intern/grad student $35k, 1 sysadmin at $65k, 1 PM/sci at $85K. All x2.5 for overhead, for 2 years. That's $4.25 mil in salaries.

There's also buying a redundant petabyte and all the necessary gear. I'm amazed they figure $5mil can do it.

Re:

The storage hardware wouldn't need to be stored onsite - all they would need is a high-speed data connection through a fibre-optic link wired up to the optical scanner. For redundancy, you would want the digital backup stored somewhere else
as well as onsit

Re:

Well, they could work on a cost-recovery basis at first, and prioritize 'client' files on demand, adding them to the master DB as they go. If two or more universities want the same palte or same series of plates, they can form an alliance to split the cost

Re:

haha, man, I run into people like you and wonder "Have people always been unable to think."

You are comparing 90 filing cabnets of paper to this.
The fact that A the paper isn't at all the fragile B it's not nearly as much data, 3 that they need special scan

Re:

In addition to the other points made, remember that the plates are fundamentally analog. Last year's memos, even if they are typewritten, are discrete. A smudge that makes a letter hard to read hardly matters, but it could be the one item you are looking

Re:

Holy crap dude, you just won the asshat of the year prize. Do you have any idea of the magnitude, delicacy, or importance of the data you're talking about? To say nothing of the needed precision when scanning.

"I scanzord 90 filing cabinets of paper into t

Re:

How about we make a backup of the backup on glass plates...

Ack! Put down that knife!

Re:

Google might do it just because it would be un-evil, and worth quite a few brownie points with scientists around the globe, not to mention that it would be cool archive to search.

Re:Would Google archive it, perhaps?

Google Universe (beta)
Searchable in Lat/Lon/time/intensity
that would be awesome...

Re:data/mass ratio

It depends on the number of pounds in a ton, but if it's short tons then

165 short tons = 149,685,482 grams
1e15 / 149,685,482 = 6,680,674 bytes per gram

A quick check of amazon turns up a 1TB drive which weights 2.4 pounds.
That's 1,089 grams which is 918,592,757 bytes per gram.

Unless I've messed up my math, it looks like hard drives store 137 times more information per gram. That's not as large a multiple as I had imagined though. The whole thing should still be between 1 and 2 tons when put on hard drives.