How Data Science Powered the Search for MH370 (hpe.com) 133
"In the absence of physical evidence, scientists are employing powerful computational tools to attempt to solve the greatest aviation mystery of our time: the disappearance of flight MH370." Slashdot reader Esther Schindler shared this article from HPE Insights:
Satellite communications provider Inmarsat announced it had found recorded signals in its archives that MH370 had sent for another six hours after it disappeared. The plane had been aloft and flying for that whole time -- but where had it gone? As Inmarsat scientists examined the signals, they saw that what they had was not data such as text messages or location information. Rather, the signals contained metadata: information about the signal itself. This was recorded as the satellite automatically contacted the plane's communications system every hour to see if it was still logged on. Bafflingly, whoever had taken the plane hadn't used the satcom system to communicate with the outside world, but had switched it off and then on again, leaving it able to exchange hourly "pings" with the satellite. Some of the metadata related to extremely subtle variations in the frequency of the signal. "We're talking about changes as big as one part in a billion," says Inmarsat scientist Chris Ashton.
Nobody had tried to use this kind of data to try to locate an airplane before. At first, Ashton's team didn't know if the attempt would work. But painstakingly, over the course of weeks, the team figured out how the movement of the plane, the orbital wobble of the satellite, and the electronics within the satcom system all interacted to create the data values that had been received. "We had to create the model from scratch," Ashton says. Their work revealed that the plane had flown into the remote southern Indian Ocean. They didn't know where exactly. But since there are no islands in that part of the world, it was impossible that anyone could have survived. For the first time in history, hundreds of people were declared legally dead based on mathematics alone.
Then mathematician Dr. Neil Gordon led a team from the Defense Science and Technology Group "to extract a path from a subset of the Inmarsat data called the Burst Timing Offset. This measured how quickly the aircraft responded each time the satellite pinged it, and was used to determine the distance between the satellite and the plane." They ultimately generate "a probabilistic 'heat map' of the plane's most likely resting places using a technique called Bayesian analysis. These calculations allowed the DSTG team to draw a box 400 miles long and 70 miles across, which contained about 90 percent of the total probability distribution.
Nobody had tried to use this kind of data to try to locate an airplane before. At first, Ashton's team didn't know if the attempt would work. But painstakingly, over the course of weeks, the team figured out how the movement of the plane, the orbital wobble of the satellite, and the electronics within the satcom system all interacted to create the data values that had been received. "We had to create the model from scratch," Ashton says. Their work revealed that the plane had flown into the remote southern Indian Ocean. They didn't know where exactly. But since there are no islands in that part of the world, it was impossible that anyone could have survived. For the first time in history, hundreds of people were declared legally dead based on mathematics alone.
Then mathematician Dr. Neil Gordon led a team from the Defense Science and Technology Group "to extract a path from a subset of the Inmarsat data called the Burst Timing Offset. This measured how quickly the aircraft responded each time the satellite pinged it, and was used to determine the distance between the satellite and the plane." They ultimately generate "a probabilistic 'heat map' of the plane's most likely resting places using a technique called Bayesian analysis. These calculations allowed the DSTG team to draw a box 400 miles long and 70 miles across, which contained about 90 percent of the total probability distribution.
Very userful (Score:5, Insightful)
Amazing stuff.
Re:Very userful (Score:5, Interesting)
This is an old story, and I think the analysis has shown it to be more conjecture that a real model. There is simply too many assumptions that need to be made, the variables are too complex.
The analysis was interesting, and if it had worked would have a great feat of data analysis. But over a year ago drift analysis of the wreckage indicated that any probably location was nowhere near the IMSAT estimated locations.
Re:Very userful (Score:5, Insightful)
But until the plane is actually found, there's no point arguing which is correct. We can't draw any conclusions until the plane is found. And it probably will never be found. Even if the search area indicated by the satellite signals is accurate, finding it there was always going to be a long shot (after the pingers stopped after 30 days). Given the relative sizes of the plane and the search area, finding a needle in a haystack is child's play by comparison. This is like trying to find a needle in field of haystacks.
If they wanted to test the accuracy of their satellite analysis, they should be running it on planes on regular flights. They can calculate a plane's position at certain times based on similar satellite ping times, then check it against the plane's actual flight path. Do it enough times and you can figure out just how accurate the methodology is.
Re: Very userful (Score:5, Funny)
Use a really big magnet.
Technology makes things easier.
Re: Very userful (Score:5, Insightful)
Last time I tried, I couldn't grab aluminum with a magnet.
Re: Very userful (Score:5, Informative)
I can. Just need to induce an electric current in it, and bingo. They are known as Eddy Current Separators, and are used extensively to separate non ferrous metals from recycling waste, usually after removing the ferrous metals first using a magnet.
Where I live we put all our metal and plastics in the same bin, because the metal can be separated from the plastic with ease automatically.
Re: (Score:2)
Re: (Score:1)
Not very efficient. Simply announce a line of new Green Needles, a sustainable product made from dried herbaceous plant matter. Now you've reduced the problem to finding a needle in a needle stack.
Re: (Score:2)
If they wanted to test the accuracy of their satellite analysis, they should be running it on planes on regular flights. They can calculate a plane's position at certain times based on similar satellite ping times, then check it against the plane's actual flight path. Do it enough times and you can figure out just how accurate the methodology is.
You shouldn't need to test the satellite's accuracy until you ensure that you have an accurate way to find the location. In other words, what you should be questioning is the accuracy of data analysis (look at the quote from TFA below). The satellite collected only the time difference when it pinged the plane. That would give you a uniformed error distance (if the data is inaccurate) each ping. Then you should calculate for the satellite's location when it pinged and received the signal. Sadly, these inform
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
Yeah, everyone except for the conspiracy theorists who think it was landed on a remote jungle island or flown under the radar to some particular country, or abducted by aliens or... etc.
Re:Very userful (Score:4, Funny)
CNN is still looking.
Re: (Score:3)
The funny thing is everyone knows that the plane went down in the ocean but don't know where exactly.
Everyone except those that look at the alternate interpretations of the same data.
Again, if the data analysis had actually led to something that would be one thing; but it hasn't. There's still only a high likelihood that MH370 ended in the Indian Ocean. And yes - I'm aware of the components found; consider that if if it went on the Northern Arc (again - a path predicted by the data) that it would have been in the interest of parties to make it look like it went on the Southern Arc by giving some evidenc
Re: (Score:2)
How big is the area? 400 x 700 square miles. How big is the aircraft in miles (73 meters x 4 meters = 0.043 miles x 0.0024 miles). Simplifying the shape of the aircraft to a box 73 meters x 73 meters, and assuming that these are aligned in a regular grid, then there are 100,000,000 possible grid squares. Submersible sonar systems can scan up to 10,000 meters, but they trade resolution for depth. but there are very few of them.
I do wonder whether weather satellites over the Indian ocean would have picked up
Re: (Score:2)
2800 quartic miles?
Re: (Score:2)
"draw a box 400 miles long and 70 miles across"
where did the 700 come from?
Re: (Score:2)
How big is the area? 400 x 700 square miles.
Well according to Wikipedia the Indian Ocean is (https://en.wikipedia.org/wiki/Indian_Ocean) 70,560,000 km2 (27,240,000 sq mi) (approximately 20% of the water on the Earth's surface) - about 2 orders of magnitude higher than your calculation - which only gives 280,000 square miles.
Now figure you're looking for things that are more like 1 meter x 1 meter in size or smaller. Yeah - it's not going to be easy; you might be able to find large parts (tail section, wings, etc) but most things are going to be qu
Re: (Score:2)
Full title: Nancy Guerrero, Dircetor of Creimer Amebas
Well... (Score:5, Informative)
Re: (Score:2)
https://en.wikipedia.org/wiki/... [wikipedia.org]
No religion needed to kill people.
Re: (Score:3)
all that science worked well!
Back in the old days, we called it math and statistical analysis. "data science" sounds cooler I suppose.
Comment removed (Score:5, Funny)
Re: (Score:2, Interesting)
I think /pol/ is getting bored of ruining Shia's life. A week ago he put the flag on top of some music hall in France and all they've done so far is launch a flaming drone at it in a half-hearted attempt to set it on fire.
Re: (Score:1)
You had one thing to do (Score:1)
Perhaps you shouldn't gloat about how great your Data Science is, if you haven't actually achieved your one goal.
Re: (Score:2)
The "ringing phones" isn't a thing. [cnn.com]
Re: (Score:1)
CNN you referenced CNN, c'mon at least try to treat it a little seriously, the Corporate Nonsense Network, that's a worse smack down than referencing Wikipedia in a doctoral thesis ;D (no seriously, it's a bit of a pill to filter out news searches but you kind of have to especially for main stream US media).
Re: (Score:2)
Yeah, CNN isn't the greatest. Tell you what - supply your "source" and we'll compare reputations :)
Re: (Score:2, Insightful)
Re: (Score:2)
CNN is well known as a middle-of-the-road, trustworthy source.
You sound a like a nutter sheep. Bahahahahahaha little sheep bahahahahahaha! Just bleet out whatever the newsvertainment outlet told you to.
Re: (Score:2)
Re: (Score:2)
If you're not going to go whole-hog and measure a metric ton of horse shit every day to filter out the very small amount of source data implied by it, then yeah, you're best off just not believing anything.
If you disbelieve everything you're almost there. The problem is, people want to believe something, but they don't want to admit that everybody is full of shit and that it would be a huge amount of work just to figure out what was actually claimed, and what was implied through phrasing but wasn't actually
Re: (Score:1)
RWNJ ALERT, Cunt hates truth, whimpers fake news.
Re: (Score:2)
As I recall, there actually was a possible motive. He had a friend who was on trial for the crime of homosexual conduct. Yes, that's right, we're not talking 1940's UK here, we're talking about the 21st century. I'm by no means left-leaning and I think it's pretty dumb. I seem to recall but not very well that there was also something going on where his wife and kids had moved out.
And yeah, he was a flight simulator junkie and could easily have planned and practiced this from his own house. Of course flight
Re: (Score:2)
Re: (Score:1)
How about we just forego all this back patting until the plane is found?
Let's not forget what brought us here (Score:5, Insightful)
Everybody would not be needing to comb over minute pieces of data and vast esoteric computations if service providers had behaved better.
The satellite service was capable of gathering the gps data [extremetech.com] from the plane instantaneously and throughout its flight path. But the satellite company was charging for it, and Malaysian authorities did not want to pay for it presumably because it cost too much.
If the gps location service had been available for this flight, one can't help but wonder if there was a possible intervention that could have been undertaken when the plane would have been discovered wildly off course, and even though it appears the crash was not survivable, the quick crash site discovery and possible apprehension of possible criminals involved (if there are any).
As it is, everybody was chintzy all the way around at the expense of the safety of the flying public.
Re: (Score:2)
As it is, everybody was chintzy all the way around at the expense of the safety of the flying public.
No lie. What year is it? It's really not acceptable that we don't know where all airlines are, all the time.
Re: (Score:2)
Re: (Score:2)
As it is, everybody was chintzy all the way around at the expense of the safety of the flying public.
At some point I would hope that people stop throwing money at the already ludicrously safe experience of travelling via aircraft and instead spend those savings on maybe making my car drive to the airport safer given I'm far more likely to die there on the road than in a aircraft crash.
And that goes double, triple and then some for driving or just living in a city like Kuala Lumpur.
Where were the editors? (Score:2)
Where were the editors? HPE's not Slashdot's.
Then mathematician Dr. Neil Gordon led a team from the Defence Science and Technology Group...
Given that this is a proper noun the article's spelling is incorrect even in the US. The rest of the world is constantly making allowance for US spelling but it seems that the favour is not being returned.
Has theoretical model been validated? (Score:2)
While the theoretical model has been carefully studied (See for example, http://epubs.siam.org/doi/pdf/... [siam.org] ), I'm not aware if any entity ever validated the model by actually flying an aircraft along one of the potential flight paths and comparing the ping times and doppler offsets from the theoretical model with an actual flight path. Does anyone know if that was ever done? Second best would be to compare the metadata from some other known flight with that flight's actual path.
Re: (Score:1)
They used the data from planes in the air at the time of all of this to compare the metadata. That is why they feel they have it correct. They were able to reproduce the results across a variety of aircraft and figure out arcs for them, using this method.
MH370 is STILL missing.... (Score:5, Insightful)
I'd be a whole lot more impressed about the performance of "Big Data" if the submarine ROVs had found any trace of the aircraft. Right now, what they have is a big fat NOTHING. Some control surfaces washed up on islands a thousand miles away are not indicative of the performance of any sort of data analysis.
Re: (Score:1)
Given that the black box has enough battery to ping away for 30 days, a nuclear sub would take less than a week to get there and pin point it exactly. Maybe not get to the depth of the sunken plane, but would be able to sit right there above it and say it is exactly below us and have the coordinates.
Notice that no media has mentioned anything about manned submarines finding it. No sub found it as it was not there to be found. There are military subs all over the world and within 30 days, one would have foun
Re: (Score:3, Informative)
... because nuclear submarines, which are expensive and built to 24/7 stand alert, are just willy nillly sent off looking for a missing civil airliner instead of hunting enemy submarines and putting nuclear missiles off the enemy's coast.
Oh by the way, water affects sonar the same way for sonobouys, dippers, towed sonar arays and submarines. You still have to be relatively close.
-Dan
Re: (Score:2)
No, because you have searched the entire area of your bathtub with sensors that are capable of detecting the presence of aircraft debris. We can therefore be confident that MH370 did NOT crash in your bathtub.
Re: (Score:2)
Re: (Score:2)
HPE Insights (Score:2)
Solves nothing.
Don't declare victory until the airframe is found (Score:2)
Umm, hold up, folks.
Don't declare victory until the airframe is found.
.
MH370 huh? (Score:3)
Re: (Score:2)
Too early (Score:5, Insightful)
This post comes too early. First find the damn thing, then boast about how this or that method helped finding it.
Dead by mathematics? (Score:4, Interesting)
How about declared dead after they couldn't find the plane for over 9 months and no one had established contact?
This article is a load of crap. It's an example of how these data models have failed to achieve anything useful. Firstly after almost 2 years they announced that they were looking in the wrong place: http://www.aljazeera.com/news/... [aljazeera.com] and also that they were confident that after spending $200m the plane was not in the search area they established. https://www.theguardian.com/wo... [theguardian.com]
Good work big data!
That is not "Data Science" (Score:5, Insightful)
That is merely a bit more special RF signal analysis engineering and not so much different from other radio-location tasks, although you usually have more data. Calling this "Data Science" is nonsense.
Best Explanation I've heard... (Score:2)
There was a similar aircraft where the oxygen bottle for the pilots broke its valve top and shot out the side of the aircraft like a rocket, putting a decent-sized hole in the fuselage (A foot or two in diameter). Turning off all the electronics breakers is consistent with fighting an electrical fire. So the theory would be - the pilots' oxygen bottle, in the radio bay below them, starts leaking, producing an oxygen-rich environment underneath them in the electrical hold - an electrical spark then results
Data science so good that they didn't find it (Score:2)
I'm so impressed. With all this sophisticated data science they know exactly where it is. The only problem is that they haven't found the airplane. They know where it is. They really do. It's just that they haven't found it. Another :"WIN" for science!
Not accurate (Score:2)
Re: (Score:2, Insightful)
Re: (Score:3)