Tracking Near-Earth Meteors With a 1.1 Petabyte Database

Lucas123 writes "The latest and most ambitious attempt to detect 'near-Earth objects' (NEOs) is the Panoramic Survey Telescope and Rapid Response System, or Pan-STARRS. When it's fully operational several years from now, it will have four telescopes, each with a 1.4-gigapixel camera. The system is expected to be able to track virtually all NEOs larger than 300 meters in diameter as well as many smaller ones. Rather than turning to an expensive supercomputer equipped with hundreds or thousands of processors, Pan-STARRS will use a cluster of 50 PC servers connected to 1.1 petabytes of disk storage via fast Infiniband networking gear."

That's nothing

[Enderandrew]

Back on my C64 I used to track incoming meteors and asteroids, and then blast them into smaller polygons that were much safer, but still a nuisance.

Re:

[Ethanol-fueled]

Did you ever spot any flying saucers?

Re:

[Enderandrew]

NASA likes to call them weather balloons.

Re:

[lena_10326]

NASA likes to call them weather balloons.

Oh they are indeed weather balloons... weather balloons from Betelgeuse carrying a payload suspended from the balloons, which is piloted by grey aliens.

Re:

[WormholeFiend]

Did you ever spot any flying saucers?

not yet [sonotaco.com]

Re:

[Mattsson]

Wouldn't call them safer. =)
As long as they where large, all you had to do was fly around them.
It was the myriad of fragments that killed you in the end.

Personally, I preferred the Vectrex [wikipedia.org] version over the C64 one though. =)

good

[FudRucker]

just let me know what a planet buster is about to arrive so i can mix a really strong drink and get blitzed one last time before mass extinction, at least i wont have to worry about a hangover...

Re:

[Enki X]

Pan Galactic Gargle Blaster: The effects of which are like having your brain smashed out by a slice of lemon wrapped 'round a large gold...meteor?

Near-Earth Meteors ?

[mbone]

Meteors have to hit the Earth's atmosphere, so they are by definition near-Earth. The article is talking about near-Earth asteroids, most of which are big enough that we'll have big problems if they ever turn into meteors.

Re:Near-Earth Meteors ?

[SEGT]

IANAA however it is my understanding that an object is categorized a meteor or asteroid based on its size and not its presence within our atmosphere. Lets check out what Wikipedia has to say on the matter. http://en.wikipedia.org/wiki/Meteor [wikipedia.org]

Larger than a meteoroid, the object is an asteroid; smaller than that, it is interplanetary dust.

And since this is an article dealing with NEOs...

The NEO definition includes larger objects, up to 50 m in diameter, to this category.

Re:

[SEGT]

My bad, I was reading 'meteor' as 'meteoroid'.

Re:

[RockDoctor]

IANAA however it is my understanding that an object is categorized a meteor or asteroid based on its size and not its presence within our atmosphere. Lets check out what Wikipedia has to say on the matter. http://en.wikipedia.org/wiki/Meteor [wikipedia.org]

Larger than a meteoroid, the object is an asteroid; smaller than that, it is interplanetary dust.

Good homework, as far as it goes. Unfortunately, you missed the bit at the top of the page that said

"Meteor" redirects here
[...]
A meteoroid is a small sand to boulder-sized

Re:

[foobsr]

that we'll have big problems

Indeed [geosociety.org] :) [nationalgeographic.com]

CC.

Re:A 50 PC cluster that processes 1.1Pb of data?

[antirelic]

So what is worth tracking that is 1.1Pb of data? Are there really that many NEO that are to be concerned about? 1.1Pb is a LOT of data to manage, even with a cluster of "50 PC's". Will this data be used for modeling or just for tracking or a combination of both? I'm interested in the technical explanation for needing that large of storage.

Re:

[La Camiseta]

It probably has to do with storing massive amounts of 1.4 gigapixel images in an uncompressed format. It should roughly be about 2-4GB per image depending on what's in the image and if they do any non-lossless post compression, and considering that they're going to be receiving data from 4 telescopes, that's a lot of storage that they'll be using up in just one night's worth of images, let alone years worth of data.

Re:A 50 PC cluster that processes 1.1Pb of data?

[La Camiseta]

Never mind, should have read the website [hawaii.edu]

Each raw image from a single Pan-STARRS camera will contain 2 Gbytes (2 bytes per pixel). In full survey mode, typical exposures last 30 seconds, so the raw data rate is several terabytes per night for the full telescope. The amount of data produced by Pan-STARRS is so large that it will not be practical to archive every image. Software techniques are therefore being developed to extract the important information from the images, while allowing less crucial information to be discarded.

Re:

[imsabbel]

Well. This actually is kind of obvious:

If you want to track something, you have to know that there _is_ something.
If you already know the asteroid, and have the trajectory calculated for a comparison, fine. But then you wouldnt need the survey. Its to find new/changed stuff.

So they need to have at least 1 full reference copy of the sky of the survey on storage to compare against, to look for changed "star" positions or new "points".
Most likely they would need a lot more than that, because of various reasons

Re:

[zappepcs]

I'm with you on that one AND isn't it a bit old hat this week to refer to mere 1.1PB storage arrays as big? Don't we have news of actual databases that big or bigger?

When we have 1.5 TB drives being made ready for market isn't a PB storage system rather mundane, or at least getting there? Simple RAID-5 Terabyte arrays are common. When you split the storage space across 50 systems, meh, it doesn't really sound like that much. Sure, it's a lot of hardware, and I bet it makes some fan noise but is it really n

Re:

[Anonymous Coward]

They are storing (almost) raw data to analyze. You can probably spot the position of some rocks in one "frame." To find their speed since the last frame, you need at least another frame. To predict their future position, a shitload more frames (remember that we can't solve meteoroid orbital equations precisely, we need many more points so the orbits can be fitted to a bunch of aproximations with lots of empirical constants).

Since now we are talking probabilities, you need a large number of measurements to i

Data Analysis

[js_sebastian]

So what is worth tracking that is 1.1Pb of data? Are there really that many NEO that are to be concerned about? 1.1Pb is a LOT of data to manage, even with a cluster of "50 PC's". Will this data be used for modeling or just for tracking or a combination of both? I'm interested in the technical explanation for needing that large of storage.

My dad is currently in Hawaii, working on the data analysis software for Pan-STARRS, so I have a vague idea of what it is about.

It's not the NEO that use 1.1Pb of data, it's storage of the raw (high resolution) data from the telescope. In fact, I'm not even sure they CAN archive all the images.

The point of having it is finding NEO, computing their orbits, so we can be able to predict our doom (and maybe do something about it before it happens).

Actually I think the system also has civilian space-flight appl

In case of Armageddon

[runlevelfour]

Good thing that the meteors are being tracked so meticulously. That way if a big "planet ender" comes close enough we can send a crack team of entertaining miners up to plant nuclear devices on it. Oh and they better shell out a bit extra for a bitchin' soundtrack to go with it.

Re:

[Enderandrew]

Don't forget the animal crackers!

Re:

[maxume]

Do animals have good security?

Re:

[daveime]

I think I'd rather drill Liv Tyler than the meteor to be honest (although I'm sure my wife would disagree).

Re:

[XHIIHIIHX]

all the good high precision math junk is in c++.

"...Near-Earth Meteors..."

[John Hasler]

Perhaps you might want to read up on just what a meteor [wikipedia.org] actually is. Hint: all meteors are "near Earth".

SQL eh?

[needs2bfree]

If i ever discover a NEO, im nameing it 17234 XKCD); DROP TABLE asteroids;

Re:

[Enderandrew]

No THX-1138 XKCD); DROP TABLE asteroids;

Re:

[mbone]

Asteroid 17234 has the temporary name of 2000 EL11, so maybe for a suitable donation to the discoverer you could get your wish.

Hang on

[CrackedButter]

Its more important to focus on lenses rather than megapixels, having good glass is more important in order to gain a better quality image... oh wait!

Re:

[hvm2hvm]

No, this is the right way to do it because they are not looking at distant objects like stars or galaxies. NEOs reflect enough light to be seen easily. The problem is with having a large enough field with a large enough resolution that you can scan the sky fast and process the data automatically. A camera with a huge resolution is perfect for the job.

Re:Hang on

[dierdorf]

Actually, Pan-STARRS is the el cheapo quick and dirty version of the near-Earth survey. Look up the LSST (the Large Synoptic Survey Telescope), currently under construction. http://www.lsst.org/ [lsst.org]

• Ten times as sensitive (an 8-meter mirror) so it can detect down to 100m objects -- thirty times as small.
• A 3.2 Gpixel camera.
• An image every 15 seconds, doing a complete raster scan of the sky every three days.
• 30 TBytes of data PER NIGHT, and they plan to keep it all for ten years.

Google volunteered to be involved in the data handling, and Bill Gates and Charles Simonyi have contributed 30 million dollars to the construction. During the initial design, the astronomers actually said, "By the time the LSST goes online (2014) we expect that Moore's Law will allow us to process the data stream."

Re: LSST will just be the latest in a line.

[Shag]

Sure, but by the time LSST goes online (2014), PS1 (the prototype on Maui, assuming that the full PS4 system is never even built) will have tiled the entire visible sky (90N to I dunno, -60 to -75S, I'd imagine) something like 120 times... so LSST can just look for whatever's left. :)

And it's not like these are the first, of course, NASA JPL's NEAT (Near-Earth Asteroid Tracking) system has been running for years and years.

One thing to consider, though - automated survey systems like these don't just find th

This is so misguided

[Al Al Cool J]

I'm sorry, but with 1.1 Petabytes of storage and 1.4-gigapixel cameras, they should be focussing on porn-stars, not Pan-STARRS.

Re:

[maxume]

Porn is pretty much not a huge fan of HD. The human body doesn't look all that good up-close and in high-res. Or at least, not as good as the fantasy that fills in over lower res.

Re:

[TooMuchToDo]

Agreed. I've worked with an adult entertainment company (we host their sites) and shooting the shit the other day over a beer, they told me that they don't want to do stuff in HD because reality sets in (that even porn stars have average joe bodies up close).

Re:

[Kjella]

I'm sorry, but with 1.1 Petabytes of storage and 1.4-gigapixel cameras, they should be focussing on porn-stars, not Pan-STARRS.

Most of them, I'd rather not. In high definition, all the blemishes come out such as zits, scars, lousy makeup and so on. There are exceptions but I'd say the number of potential porn actresses drops by about 90% to only include the really, really stunningly beautiful girls. The cheap porn flick I'd rather pay to not see in HD...

Re:

[Sponge Bath]

cheap porn flick I'd rather pay to not see in HD...

Where all the participants (men and women) look and
sound like Carl from Aqua Teen Hunger Force.
Now in 1080p!

Re:

[Al Al Cool J]

Doesn't have to be HD. Think massively widescreen orgy.

Individual boxen?

[kramulous]

The only advantage I can see of using 50 boxes for the computational and file serving would be that you could power cycle those that are not on demand. But if your recording terabytes of images and you're going to run some image processing/data analysis routines over them, would you be better off with a compute cluster such as a rack of Altix or Blue Gene? Easier to manage, lower administration, maintenance and ongoings? I also have to question Microsoft SQL Server. Storing and retrieving images sure, but when it comes to serving for analysis and storing/collating results, it would be a little too slow? How much can you tune a closed source solution on a tight budget as opposed going for one that you can tinker with to gain performance.

Re:

[Lennie]

It's in the article: they got the SQL-server on an academic-license and 50 computers for the same shoe-string butget.

Databases blow for storage

[wisty]

Scientific data is write-once, read-many (not read-write financial transactions), so pretty much everything in the database is overhead.

1.1 Petabytes!

[OakDragon]

1.1 petabytes? 1.1 petabytes? Great Scott!

Great Related Sci-Fi

[saberworks]

If the topic interests you at all, check out Michael Flynn's Firestar series. Near-future science fiction. I contend that it's the best sci-fi series ever written. No, these aren't affiliate links.

http://www.amazon.com/Firestar-Michael-Flynn/dp/0812530063/ref=sr_1_1?ie=UTF8&s=books&qid=1218256538&sr=8-1 [amazon.com]

http://www.amazon.com/Rogue-Star-Firestar-Michael-Flynn/dp/0812542991/ref=pd_sim_b_1 [amazon.com]

http://www.amazon.com/Lodestar-Firestar-Michael-Flynn/dp/0812542967/ref=pd_sim_b_1 [amazon.com]

http://www.amazon.com/Falli [amazon.com]

So it can track those which cant be avoided ...

[tkjtkj]

So, it will be able to track those objects of such size or greater that would , unavoidably, sterilize our planet ... yet be unable to track those ( dia 300m ) whose paths we actually might be able to deflect ... but it is a start and is to be applauded....

Re:

[js_sebastian]

So, it will be able to track those objects of such size or greater that would , unavoidably, sterilize our planet ... yet be unable to track those ( dia 300m ) whose paths we actually might be able to deflect ...

but it is a start and is to be applauded....

Who says we can't deflect a 1km object? The point is, you can't do it Armageddon-style at the last minute. But you can give it a small push in some direction 10 orbits (or 30 years) before it hits us. That's why orbit predictions need to be 50 years ahead.

Re:

[tkjtkj]

True, yet the odds are enormously larger that the number of objects smaller than 300m is considerably greater than the number that can be detected. This, logically, makes it relatively pointless to search for 'the biggies' when the risk of a cataclysmic sterilizing event secondary to a smaller, less than 300m , object has to be considerably higher. but yes, as i commented, huge objects, even the size of earth itself, can theoretically have their orbits significantly changed, given enough time .. Didnt i

Re:

[js_sebastian]

True, yet the odds are enormously larger that the number of objects smaller than 300m is considerably greater than the number that can be detected. This, logically, makes it relatively pointless to search for 'the biggies' when the risk of a cataclysmic sterilizing event secondary to a smaller, less than 300m , object has to be considerably higher.

A 300m object will make a big BOOM, but it won't kill many people unless it lands on Manhattan (which is unlikely). If we do a little game theory and use the number of human lifes lost as cost, we have to multiply the probability of an event by the number of people it will kill.

If the big asteroids (of the kind that made dinosaurs go extinct) happen once every 70 million years on average, and we assume they kill all of us 7 billions people when they hit, then the big asteroids kill 100 people a year on aver

Re:

[tkjtkj]

you must be joking! To say that an asteroid having a diameter 3 times as long as a football field, and travelling at 18,000 miles per hour and colliding with earth would merely destroy an area the size of Manhattan?? Do you have data for such a misguided assertion?

S&T article on these guys

[edremy]

This month's issue of Sky and Telescope has a nice article on Pan-STARRS and a few other enormous survey telescopes. The linked article missed one of the most interesting bits of the camera- it's using an Orthogonal Transfer Charge Coupled Device.(See http://pan-starrs.ifa.hawaii.edu/public/design-features/cameras.html [hawaii.edu]) An OTCCD can transfer built up charge from one pixel to another, so you can compensate for atmospheric distortion by simply moving stuff on the chip rather than trying to do it with a flex

Wasted equipment.....

[IHC Navistar]

I used to use an Atari 2600. Pretty simple set up. Small, compact and plugged right into my TV. Plus, with the addition of a cheap little switch, I could blow up incoming asteroids during commercial breaks. .....sigh..... How I miss my Atari.

Real men don't need controllers with a dozen buttons to beat games. They only need a joystick and a button.

Re:

[mikael]

Haven't you seen the Jakks Atari classics 10 games in a joystick? They have miniaturised the console electronics so that you get a whole Atari 2600 system plus 10 games (Gravitar, Asteroids, Real Sports Volleyball, Centipede, Adventure, Pong, Missile Command, Breakout, Yars' Revenge and Circus Atari) all in a single joystick [amazon.com].

Re:

[mikael]

Better still, Atari Flashback 2 [amazon.com].

It totally amazed me that these console systems are still on sale after 30 years, but make great entertainment for a 80's themed party.

The GrayWulf cluster from Microsoft

[viking80]

From TFA:
A new feature allowing big distributed databases with MS Server 2008. This setup is called a GrayWulf cluster.

I wonder if this was the idea of the new PR company MS hired after the marketing failure of Vista?

Re:

[Bazman]

I can't really see the phrase 'Imagine a GrayWulf cluster of these!' catching on on Slashdot. Nice try, Microsoft.

Mil Sats? Govt. Censorship?

[wisebabo]

So anything that can detect, track and determine the orbits of NEOs efficiently has got to be good at picking up all sorts of things in space. Like military satellites.

Will (any) governments be able to "edit" the database so that things they don't want noticed will remain so? On the other hand, will they be providing this project with satellite data to prevent false positives? Also, will the database be available to the public/amateur astronomers so that things like the recent "ghost" image can be found

Re:

[IdleTime]

The voice came from nowhere and everywhere around us, it was soft and feminine and reassuring in some strange way, our intergalactic cruiser was floating in space...
"The Earth is currently closed for business due to a reboot, we will be back on-line in...ohh let's see.. another 4.5 billion years. Thank you for waiting!"