Extracting Audio From Visual Information 142
rtoz writes Researchers at MIT, Microsoft, and Adobe have developed an algorithm that can reconstruct an audio signal by analyzing minute vibrations of objects depicted in video. In one set of experiments, they were able to recover intelligible speech from the vibrations of a potato-chip bag (video) photographed from 15 feet away through soundproof glass.
Not surprising (Score:5, Insightful)
Measuring the vibrations of windows or other items was used already 40 to 50 years ago by spy agencies, so I wonder if this isn't something that has been re-discovered?
Re:Not surprising (Score:5, Informative)
To follow up, look at the Electromax Laser Listening Systems [electromax.com].
Re: (Score:1)
Bet that works really well on video.
Re:Not surprising (Score:5, Insightful)
The countermeasure for laser listening was to install the windows inside a pipe *frame* and play music in the pipes. Using an object inside the building to extract audio defeats that countermeasure. This is 2014, do not expect any privacy, especially from government agencies...
Re:Not surprising (Score:5, Funny)
(At least until they discover the guy spying on the basement dwellers with sophisticated seismometers)
Re: (Score:3)
From Top Secret America: the rise of the surveilance state [google.com]
As important to a man's self image as the power of his car's engine or his motorcycle's rumble, SCIF size had become a symbol of status. "In DC, everyone talks SCIF, SCIF, SCIF," said Bruce Paquin, owner of a construction company that builds SCIFs for the government and private corporations. "They've got the penis envy thing going. You can't be a big boy unless you're a three letter agency and you have a big SCIF.
(A SCIF [scifsolutions.com] is a room that has been certified to be impenetrable to various types of surveillance techniques.)
Re: (Score:2)
Re: (Score:3)
That's on a need to know basis, and you don't have a need to know.
Re: (Score:2)
Re: (Score:2)
If they can place the laser at an angle where they can see more than the ceiling then any rigid object needs countermeasures, not just the outside window.
Re: (Score:2)
I was thinking the same.. Or more to the point, about putting hidden messages in video so a plant for instance would play Pantera and a bag of chips could play some canned messages about the government spying or some shit. Perhaps even make a contest to see who can decode the most and have all the different messages combine to be code for something else entirely.
Either way, I'm wondering what can be picked up on some of those Youtube vids where they dubbed a narration and blanked out the camera's sound. Fo
Re:Not surprising (Score:5, Insightful)
Re: (Score:1)
I heard about this system sometime in the 1980's -- I'm sure it was "secret" back then, but dudes seem to leak "cool stuff" -- so there you go.
The counter to this technique is to put two speakers set to different radio stations and aim them at the window -- or go into a shower stall and whisper in your hand with the water turned on (no, really).
I hope I don't get on a "Snowden" watch list, but as a ten-year-old, I think I could figure out a counter to every possible spook tool. It's not my fault they aren't
Re:Not surprising (Score:5, Informative)
There is a very significant difference: this involves detecting vibrations in images of objects in a video recording rather than the objects themselves. However, not just any video will do; it requires a very high frame rate.
Re: (Score:2, Funny)
When high frame rate cameras are outlawed, only outlaws will have high frame rate cameras..............
Re: (Score:2)
This needs to be put on a T-shirt.
Re: (Score:3)
The method is the same, it's just a different tool involved on the way.
It's enough to measure the image of an object, you don't need to record it first and you actually don't need a laser either, even though it may help.
Re:Not surprising (Score:5, Informative)
FTFA
In other experiments, however, they used an ordinary digital camera. Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as it was with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.
They don't go into detail on the algorithm but reading between the lines it seems that they are using the spatial nature of video and the fact that not every pixel is captured at exactly the same moment (let alone each line) to ferret out higher frequency information. I have other guesses, but they are wild speculation. Either way VERY cool.
Re: (Score:2)
Seems to me that a very high frequency lightsource in the room (WAAY faster than the 60-120hz fluorescent type) would totally bork this system, especially if there were multiple light sources in the room that were not in good phase agreement.
The camera takes images at 5000 frames per second, or there abouts. So, we need to degrade the camera's ability to take good images by screwing with the light levels at a higher speed, which will then introduce noise into the images captured. A normal human in the room
Re: (Score:2)
I don't know but I bet a Google Glass wearer might be the first to find out.
Imagine that, now not only do you have to worry about your pic and video of you at the bar hanging with the guys being posted all over face book by someone you don't even know, but now your conversations about who you would like to bag might make it back to your wife or significant other.
Expect the cops and government to use this. I imagine a drone flying over at 15,000 feet with a telescopic lenses and listening in on a terrorist o
Re: (Score:2)
However, there's very little similarity, aside from the use of available objects rather than specially designed surfaces; between using an interferometer to measure vibrations and using a machine vision algorithm to do so.
Re: (Score:2)
Agreed, this is more of a bench mark scale capability I think. Also uses video capture, not lasers which is more passive technology.
Re: (Score:2)
Clarify, video is the more passive tech obv.
Re:This can not work with most video (Score:2)
This works with very high framerate video. Normal video has way too low of a frame rate to capture vibrations in the (phone company defined) band of 300 HZ to 3000 HZ. At normal vidoe rates of about 24-50 FPS, the sample rate is below the lower fundemental frequency of voice.
Re: (Score:2)
Actually, I was terribly unimpressed, too, until they did something clever that overcomes this limitation. Watch the last few dozen seconds of the video, as they pull sound out of the video recorded by a normal, consumer DSLR: no high speed video needed!
Re: (Score:2)
And what happens if someone holds up a document to the window? Perhaps that might be a problem, too. If you want good security, don't have windows.
Re: (Score:2)
I would still say that it's the same basic principle just another method.
Re: (Score:2)
I'm not sure if this is true.
Definitely not for the coherent bit, trying to setup a second equally long beam path to be able to do interferometry would be an absolute pain and a half (needs to be pretty much exactly as long, coherence length of lasers is never going to be large enough to be able to interfere it without path length matching). Much easier to just intensity modulate the laser and measure the doppler shift of the modulating signal.
Not sure about the angle either, the window won't have much diff
Now that's a bit disturbing.... (Score:2)
Puts the potato chip bag back into his lunch bag.
(How much did DHS pay for this research?)
Been there done that (Score:3, Funny)
Sorry but that is so 2004.
- NSA
Re:Been there done that (Score:5, Funny)
Are they not doing this already in CSI? I'm sure I saw them enhance an office security video of a post-it note, reflected off a monitor screen, magnified a couple of times, and there they had it; complete dialog in stereo, with accompanying analysis of voice stress so they knew who was lying. Isn't science wonderful?
Re: (Score:2)
Er, that's so 1984
Possible NASA method (Score:2, Funny)
Could this be used by NASA to look for intelligent life on other worlds by measuring objects in the same fashion?
Re: (Score:1)
Depends, does it work on objects of sub-pixel size?
Re: (Score:2)
Depends, does it work on objects of sub-pixel size?
Just to make life more fun, a planet with life would likely have an atmosphere and those make a bit of a mess of light that passes through them(which might be handy if we are looking for atmospheres; but substantially less so if we are trying to look through them).
Re: (Score:2)
This is best used at very high frame rates (50,000 frames per second I think) - and the "pictures" of alien planets are made with exposures of hours.
Re: (Score:2)
In theory... However you will need to get a really good video image of a planet.
Right now most of the planets outside of our solar system is extrapolated mathematically not actually seen directly with a camera.
Then you will need to get a really really good resolution (To a point where you can probably see the life on the planet anyways) Then you will need to send it back, to earth for translation. So by the time we say hello. to them it may just be a lost language, or at least sounding quite out of place.
Re: (Score:2)
I think it depends on whether the intelligent life has developed potato chips and bag in which to hold them while being eaten...under the assumption that potato chips are not toxic to the intelligent life as they sometimes are to terrestrial humans.
Scary (Score:3, Interesting)
This is cool, yet scary stuff.
I wonder how loud the original audio has to be in order to be recovered in this manner? It sounded to me like the spoken words were being shouted, and we have no way of knowing how loud the music was played. I didn't see any mention of that in the linked article.
The linked article has additional technical(ish) information that's not in the video.
Now my tin-foil hat... (Score:5, Funny)
...Needs a tin-foil hat!
Re:Now my tin-foil hat... (Score:5, Insightful)
The hat is a trick!
The reason they want you to wear foil is so that the sound can bounce off it.
Re:Now my tin-foil hat... (Score:5, Insightful)
Unfortunately, walking around with what appears to be a small F-117 attached to your head offers limited visual camouflage potential and may prove counterproductive in your attempts to avoid Their surveillance.
Re: (Score:1)
What's that, a hockey joke?
To properly secure your tin foil hat, you need to cover it with cork. Quarter inch should do it.
Requires a very high speed camera (Score:5, Interesting)
Re:Requires a very high speed camera (Score:5, Insightful)
Re: (Score:3)
No, it could work. He wants to capture different information from different parts of the bag. This is a multi-channel problem so you can go below Nyquist. Also you might have a model for speech and you can use to reduce the amount of required information. Finally, you co not need perfect recovery.
Re: (Score:1)
The problem is that you would be able to pick up a maximum frequency of 15Hz.
It doesn't matter how many channels you have if all of them are completely out of your frequency range.
Re: (Score:2)
Another possibility is if there are multiple video streams from the same even
Re: (Score:2)
They were able to get decent voice from a 60 Hz camera. I'm guessing the camera didn't record all pixels simultaneously, and so the differences between when each pixel sampled were enough to work with.
Re: (Score:3, Informative)
No, you can pick up something higher than Nyquist, as long as you understand your sources of information and noise. It will alias down into the measurable range, and you can extract useful information from the alias. We have a system that operates up to 1 MHz using a 1.8 MHz ADC. When we know the signal is at 1 MHz, we extract the information at 800 kHz and use that.
What the GGP was talking about, though, was finding resonance on the bag where unique 30-Hz-width bands higher frequencies were being natura
Re: (Score:3)
Re:Requires a very high speed camera (Score:4, Insightful)
Re: (Score:2)
Re: (Score:3, Insightful)
Oh dear. You even linked to Wikipedia (although not to the Wikipedia page "Nyquist Rate"). Does it not occur to you that OP understands those things better than you do?
To start with you need to understand what the Nyquist rate means. Sampling is like wrapping a signal around a cylinder. Just because parts are overlaid ("aliasing") doesn't mean you can't untangle the original signal. For instance, if a single audio source contains only pure harmonics, so the frequencies are known to be N, 2N, 3N, 4N, an
Re: (Score:3)
That assumes that you only are getting one sample per frame. FTFA
In other experiments, however, they used an ordinary digital camera. Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as it was with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.
Remember that video has two spatial dimensions with 3 channels (which themselves are in different spatial locations within each pixel) each and that each line isn't captured at the same instant. There is a lot more information there than a single sample at a given rate. Nyquist doesn't apply to the frame rate here. Nyquist is stil lrelevant to the problem, of course! They didn't break Nyquist, they just found a way to get more information than
Re: (Score:3, Insightful)
reconstruct from standard 30 fps video
Dear sir: what you are asking is impossible.
Sincerely yours,
Harry Nyquist
Re: (Score:2)
It's not "impossible", and he even told you how to do it. Incidentally, your ear works the way he suggested.
Re: (Score:2)
If you use a 1 kHz ADC to measure a 1.1 kHz signal, what do you measure at 900 Hz?
Re: (Score:2)
If you use a 1 kHz ADC to measure a 1.1 kHz signal, what do you measure at 900 Hz?
450Hz. Just like, in your example of 1.1kHz measured by a 1kHz ADC, your measurement would be .650kHz or 650Hz.
Re: (Score:2)
Requires a very high speed camera (Score:1)
You say "little different from a laser mic". Yes, innovation is incremental, this is an increment.
Re:Requires a very high speed camera (Score:5, Informative)
For some reason, the person who posted the article or the Slashdot editors linked to a bad knock-off video that removed 3/4 of the details instead of the actual researchers' video [youtube.com]. The real video makes it clear that they can also get results from a standard DSLR 60 FPS video by taking advantage of the rolling shutter effect. There's a fidelity loss, but it's a lot better than I would have expected.
Re: (Score:2)
Re: (Score:2)
Radar! What is the general saying?
Resolution and sensor noise (Score:3, Informative)
The sensor and optics must have been ridiculously high quality and resolution for this to work. Sensor noise alone would almost certainly rule this out for any COTS consumer package. They certainly aren't doing it with CNN footage or old CCTV surveillance tapes.
In which case, it's of no practical value since a laser mic would be far cheaper and more discrete.
Cool from an academic perspective that they can use DSP now, but it's just more fun with a laser mic, same principals and theories, new less workable application.
Re: (Score:2)
This seems to work through soundproof glass... On the other hand, how big would be a camera able to record at this resolution and frame rates, and how close it must be?
Re: (Score:2)
This seems to work through soundproof glass...
Glass that is pretty CLEAN.. You need really fast frame rates (6Khz will get you phone quality audio) and pretty high optical resolution. I'm just guessing, but you are going to need 3-4 pixels for any kind of reasonable S/N ratio that's listenable, so if the object you are looking at only moves a few nanometers with the sound, that means you need a minimum of two pixels per nanometer. To do that kind of resolution at say 10 feet, is going to require some pretty good optics. The resolution of the video
Re: (Score:2)
Being able to do this from a recording is the magic. There are many places where video is recorded but not audio. Being able to recover the audio much later would be special. Also worrying, in places where you can record video but not audio without a warrant.
Re: (Score:2)
The problem is that the video is recorded in too low of resolution and too low of framerate to be useful. Any place just recording video and not audio is doing it with cheap ass bargain basement gear, hence where there is no audio gear already recording the sound they are missing
Re: (Score:2)
Cheap cameras only get better. They were able to reconstruct voice from a good 60 Hz camera. If a policeman wanted to game the system, he could use a good camera to avoid the need for a warrant. Much like the NSAs "we capture everything, but only look at the data with (later) probable cause" excuse, someone might try an equally disturbing "we capture video, and only reconstruct the audio with (later) probable cause" in an attempt to game the system similarly. Uggh.
Re: (Score:2)
Or Wolf Blitzer broadcasts. It is a fact that Wolf can actually suck information out of the ambient room and it is never seen nor heard from again. Think of him as an acoustical black hole.
Re: (Score:2)
except that a laser mic is ACTIVE monitoring and could be detected, whereas video is PASSIVE monitoring and is undetectable. It would also allow for monitoring in hostile environments, i.e. some guys around a campfire, where there may not be a nice object to laser against.
Yeah, only if one speaks in extremely low tones... (Score:2)
Re: Yeah, only if one speaks in extremely low tone (Score:2, Funny)
Because if your target is eating SunChips you'd risk hearing loss.
Re: (Score:3)
Because your emitting something sending that IR laser to do it. This is completely passive.
Re: (Score:2)
Re: (Score:2)
And with good optics the camera could be just/nearly as far away. Laser is old know tech so places will have countermeasures and detection setup. Miniaturization can also come into place where you can shove the camera into a cell phone etc.
Re: (Score:2)
Miniaturization can also come into place where you can shove the camera into a cell phone etc.
The laws of physics say otherwise. You can't get the resolution/quality out of a cell phone sized camera to do it at any difference, there just isn't enough in the lenses to do it.
Then couple in the need for 6k frames per second being about the bottom line requirement for getting voice frequencies using this technique, its pretty useless in any situation that an alternative wouldn't work better.
Re: (Score:2)
From the looks of it a 1d array might work rather well and get the frame rates required. I didn't say generic cell phone just fitting in the form factor or close enough to not be suspicious. This could fit in existing security camera form factors (the 18 ish inch long all weather enclosures commonly used) that are so common as to be forgotten.
Re: (Score:2)
Why should a high speed camera look any different from a regular one?
Re: (Score:1)
RTFA. They are using a high speed camera.
Why not bounce an IR laser off that potato chip bag? Because you may not be able to. Windows have IR-reflective coating, the bag is not very reflective, the surface may be at the wrong angle. This is useful.
High speed cameras are not that expensive. You can get 1000fps in a consumer camera for around $1000 these days.
Re: (Score:2)
You can get 1000fps in a consumer camera for around $1000 these days.
Not one with a telephoto lens that will zoom in from yards away to resolve a potato chip bag. Your consumer camera spy equipment is a pipe dream...
You'd be better off hiring a lip reader. Of course, all of this could be averted by closing the blinds in front of the window where the subject wishes his conversation to be private... I'm filing this one under "Dumb ideas".
Re: (Score:2)
I'm usually a cynical bastard but... (Score:2)
...that's pretty effin' amazing. From video, 15 feet away. Not using a laser, FROM VIDEO! Lol.
It's curtains for privacy (Score:1)
The minimalist architects are in league with the spooks!
Breached! (Score:1)
The Cone of Silence has been breached Max!
Re: Breached! (Score:2)
only because the Adobe software was subject to a buffer overflow encoded in the Ruffles halftone.
Stop talking (Score:1)
The first thing (Score:2)
This is the first thing I though of when reading the title.
SFW
https://www.youtube.com/watch?v=6i3NWKbBaaU
Class and Security (Score:2)
Not only is it classier, but now serving your potato chips in a nice bowl is more secure.
Um... (Score:2)
How did they isolate the speech from the crunch of potato chips???? And, if this possible, there is no hope for anyone with the munchies!
tl;dr: (Score:2)
Re: (Score:2)
Laugh if you want, but if you yell MARY HAD A LITTLE LAMB at your rhododendrons, the NSA will know about it.
laws on recording (Score:1)
The reason I find this interesting is that there are a number of locations (in the U.S.) where it is legal to record video but not audio. I happen to be against some of the laws forbidding audio recordings, because I think that it usually is used to protect against the recording of corruption. (try to open a bar in new york city, with live music, see what kinds of visits you get, and notice the correlation with fire, police, and food inspection after varying responses to those asking for money.) Would
Observers (Score:1)
It's nothing more than laser mics (Score:2)
Otherwise, they're just sampling motion at 2000-6000fps. It seems ridiculously processor intensive for something which could be better achieved using a high performance light meter.
Eagle Eye (Score:2)
This is what I instantly thought of: Eagle Eye [wikipedia.org]. The scene with the soundproof room.
Link to youtube? (Score:2)
We totally /.'d Youtube now - at the time of writing this, the video has been watched 2k times. After several hours of being on the /. front page.
Re: (Score:2)
Which significantly tones down the 'omg they will hear what we say in the background of youtube videos' effect.
Not to mention that you need fairly high resolution images to be able to detect the movement too. If you cannot do high resolution at reasonable frame rates you are not going to have anything to hear...
We tend to scoff at the beliefs of the ancients (Score:2)
"We tend to scoff at the beliefs of the ancients. But we can't scoff at them personally, to their faces, and this is what annoys me."
- Jack Handey