Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Science Technology

Extracting Audio From Visual Information 142

rtoz writes Researchers at MIT, Microsoft, and Adobe have developed an algorithm that can reconstruct an audio signal by analyzing minute vibrations of objects depicted in video. In one set of experiments, they were able to recover intelligible speech from the vibrations of a potato-chip bag (video) photographed from 15 feet away through soundproof glass.
This discussion has been archived. No new comments can be posted.

Extracting Audio From Visual Information

Comments Filter:
  • Not surprising (Score:5, Insightful)

    by Z00L00K ( 682162 ) on Monday August 04, 2014 @09:42AM (#47599079) Homepage

    Measuring the vibrations of windows or other items was used already 40 to 50 years ago by spy agencies, so I wonder if this isn't something that has been re-discovered?

    • Re:Not surprising (Score:5, Informative)

      by Z00L00K ( 682162 ) on Monday August 04, 2014 @09:45AM (#47599099) Homepage

      To follow up, look at the Electromax Laser Listening Systems [electromax.com].

      • by Anonymous Coward

        Bet that works really well on video.

      • Re:Not surprising (Score:5, Insightful)

        by Hamsterdan ( 815291 ) on Monday August 04, 2014 @10:00AM (#47599181)

        The countermeasure for laser listening was to install the windows inside a pipe *frame* and play music in the pipes. Using an object inside the building to extract audio defeats that countermeasure. This is 2014, do not expect any privacy, especially from government agencies...

        • by fuzzyfuzzyfungus ( 1223518 ) on Monday August 04, 2014 @10:18AM (#47599297) Journal
          Clearly, if your work is that important having a window office becomes a sign of extremely low status and institutional nonimportance, rather than professional advancement...

          (At least until they discover the guy spying on the basement dwellers with sophisticated seismometers)
          • From Top Secret America: the rise of the surveilance state [google.com]

            As important to a man's self image as the power of his car's engine or his motorcycle's rumble, SCIF size had become a symbol of status. "In DC, everyone talks SCIF, SCIF, SCIF," said Bruce Paquin, owner of a construction company that builds SCIFs for the government and private corporations. "They've got the penis envy thing going. You can't be a big boy unless you're a three letter agency and you have a big SCIF.

            (A SCIF [scifsolutions.com] is a room that has been certified to be impenetrable to various types of surveillance techniques.)

        • If they can place the laser at an angle where they can see more than the ceiling then any rigid object needs countermeasures, not just the outside window.

      • Re:Not surprising (Score:5, Insightful)

        by timeOday ( 582209 ) on Monday August 04, 2014 @10:08AM (#47599233)
        Well, even a normal microphone is "just" measuring the linear displacement of a membrane over time, so clearly the important distinction is how you measure it. A laser range-finder is different from a microphone, and a video camera is different from a laser range-finder.
      • by Anonymous Coward

        I heard about this system sometime in the 1980's -- I'm sure it was "secret" back then, but dudes seem to leak "cool stuff" -- so there you go.

        The counter to this technique is to put two speakers set to different radio stations and aim them at the window -- or go into a shower stall and whisper in your hand with the water turned on (no, really).

        I hope I don't get on a "Snowden" watch list, but as a ten-year-old, I think I could figure out a counter to every possible spook tool. It's not my fault they aren't

    • Re:Not surprising (Score:5, Informative)

      by JazzHarper ( 745403 ) on Monday August 04, 2014 @10:02AM (#47599195) Journal

      There is a very significant difference: this involves detecting vibrations in images of objects in a video recording rather than the objects themselves. However, not just any video will do; it requires a very high frame rate.

      • Re: (Score:2, Funny)

        by Anonymous Coward

        When high frame rate cameras are outlawed, only outlaws will have high frame rate cameras..............

      • by Z00L00K ( 682162 )

        The method is the same, it's just a different tool involved on the way.

        It's enough to measure the image of an object, you don't need to record it first and you actually don't need a laser either, even though it may help.

    • The general notion that all sorts of things will vibrate in the presence of ambient noise is definitely not new. Even perfectly ordinary mics depend on it, though they bring their own specialized vibrating surface in order to make the problem considerably easier.

      However, there's very little similarity, aside from the use of available objects rather than specially designed surfaces; between using an interferometer to measure vibrations and using a machine vision algorithm to do so.
    • Agreed, this is more of a bench mark scale capability I think. Also uses video capture, not lasers which is more passive technology.

    • This works with very high framerate video. Normal video has way too low of a frame rate to capture vibrations in the (phone company defined) band of 300 HZ to 3000 HZ. At normal vidoe rates of about 24-50 FPS, the sample rate is below the lower fundemental frequency of voice.

      • Actually, I was terribly unimpressed, too, until they did something clever that overcomes this limitation. Watch the last few dozen seconds of the video, as they pull sound out of the video recorded by a normal, consumer DSLR: no high speed video needed!

    • by nwf ( 25607 )

      And what happens if someone holds up a document to the window? Perhaps that might be a problem, too. If you want good security, don't have windows.

  • Puts the potato chip bag back into his lunch bag.

    (How much did DHS pay for this research?)

  • by Anonymous Coward on Monday August 04, 2014 @09:45AM (#47599109)

    Sorry but that is so 2004.

    - NSA

  • by Anonymous Coward

    Could this be used by NASA to look for intelligent life on other worlds by measuring objects in the same fashion?

    • by Anonymous Coward

      Depends, does it work on objects of sub-pixel size?

      • Depends, does it work on objects of sub-pixel size?

        Just to make life more fun, a planet with life would likely have an atmosphere and those make a bit of a mess of light that passes through them(which might be handy if we are looking for atmospheres; but substantially less so if we are trying to look through them).

    • This is best used at very high frame rates (50,000 frames per second I think) - and the "pictures" of alien planets are made with exposures of hours.

    • In theory... However you will need to get a really good video image of a planet.
      Right now most of the planets outside of our solar system is extrapolated mathematically not actually seen directly with a camera.

      Then you will need to get a really really good resolution (To a point where you can probably see the life on the planet anyways) Then you will need to send it back, to earth for translation. So by the time we say hello. to them it may just be a lost language, or at least sounding quite out of place.

    • by gtall ( 79522 )

      I think it depends on whether the intelligent life has developed potato chips and bag in which to hold them while being eaten...under the assumption that potato chips are not toxic to the intelligent life as they sometimes are to terrestrial humans.

  • Scary (Score:3, Interesting)

    by Anonymous Coward on Monday August 04, 2014 @09:48AM (#47599121)

    This is cool, yet scary stuff.

    I wonder how loud the original audio has to be in order to be recovered in this manner? It sounded to me like the spoken words were being shouted, and we have no way of knowing how loud the music was played. I didn't see any mention of that in the linked article.

    The linked article has additional technical(ish) information that's not in the video.

  • by Anonymous Coward on Monday August 04, 2014 @09:51AM (#47599137)

    ...Needs a tin-foil hat!

  • by tepples ( 727027 ) <tepples.gmail@com> on Monday August 04, 2014 @09:51AM (#47599139) Homepage Journal
    The YouTube video captions state that this technique requires a camera capable of a few thousand frames per second. Thus this is pretty much using a camera to follow the vibrations, little different from a laser mic [wikipedia.org]. What would impress me more is if they were able to pick up different frequencies from different parts of the bag with different resonant frequencies and reconstruct from standard 30 fps video using the bag as a transducer.
    • by interiot ( 50685 ) on Monday August 04, 2014 @10:10AM (#47599241) Homepage
      30 Hz is far below the Nyquist rate [wikipedia.org] (6800 Hz, going by POTS specs), so no, that wouldn't be possible without some fundamental changes in our understanding of information theory and physics.
      • by Uecker ( 1842596 )

        No, it could work. He wants to capture different information from different parts of the bag. This is a multi-channel problem so you can go below Nyquist. Also you might have a model for speech and you can use to reduce the amount of required information. Finally, you co not need perfect recovery.

        • by Anonymous Coward

          The problem is that you would be able to pick up a maximum frequency of 15Hz.
          It doesn't matter how many channels you have if all of them are completely out of your frequency range.

          • Well, it might be theoretically possible - but you'd need to get the bits from somewhere. Think of an ocean wave, and you want to measure the height of the water at a given point in time. But waves on water move in fairly predictable ways, so a single picture will tell you both the height of the water at the time the picture was taken, as well as a good approximation of what it was for a short time before and after the picture.

            Another possibility is if there are multiple video streams from the same even
            • by lgw ( 121541 )

              They were able to get decent voice from a 60 Hz camera. I'm guessing the camera didn't record all pixels simultaneously, and so the differences between when each pixel sampled were enough to work with.

          • Re: (Score:3, Informative)

            by SydShamino ( 547793 )

            No, you can pick up something higher than Nyquist, as long as you understand your sources of information and noise. It will alias down into the measurable range, and you can extract useful information from the alias. We have a system that operates up to 1 MHz using a 1.8 MHz ADC. When we know the signal is at 1 MHz, we extract the information at 800 kHz and use that.

            What the GGP was talking about, though, was finding resonance on the bag where unique 30-Hz-width bands higher frequencies were being natura

      • 30 fps would allow a maximum frequency of 15 Hz.
      • Re: (Score:3, Insightful)

        by Anonymous Coward

        Oh dear. You even linked to Wikipedia (although not to the Wikipedia page "Nyquist Rate"). Does it not occur to you that OP understands those things better than you do?

        To start with you need to understand what the Nyquist rate means. Sampling is like wrapping a signal around a cylinder. Just because parts are overlaid ("aliasing") doesn't mean you can't untangle the original signal. For instance, if a single audio source contains only pure harmonics, so the frequencies are known to be N, 2N, 3N, 4N, an

      • That assumes that you only are getting one sample per frame. FTFA

        In other experiments, however, they used an ordinary digital camera. Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. While this audio reconstruction wasn’t as faithful as it was with the high-speed camera, it may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers’ voices, their identities.

        Remember that video has two spatial dimensions with 3 channels (which themselves are in different spatial locations within each pixel) each and that each line isn't captured at the same instant. There is a lot more information there than a single sample at a given rate. Nyquist doesn't apply to the frame rate here. Nyquist is stil lrelevant to the problem, of course! They didn't break Nyquist, they just found a way to get more information than

    • Re: (Score:3, Insightful)

      reconstruct from standard 30 fps video

      Dear sir: what you are asking is impossible.

      Sincerely yours,

      Harry Nyquist

      • by silfen ( 3720385 )

        It's not "impossible", and he even told you how to do it. Incidentally, your ear works the way he suggested.

      • If you use a 1 kHz ADC to measure a 1.1 kHz signal, what do you measure at 900 Hz?

        • by aitikin ( 909209 )

          If you use a 1 kHz ADC to measure a 1.1 kHz signal, what do you measure at 900 Hz?

          450Hz. Just like, in your example of 1.1kHz measured by a 1kHz ADC, your measurement would be .650kHz or 650Hz.

      • Unless thirty frames per second isn't actually thirty scalar samples per second, right?
    • From your own link:

      However, countermeasures exist in the form of specialized light sensors that can detect the light from the beam. Rippled glass can be used as a defense, as it provides a poor surface for a laser microphone.

      You say "little different from a laser mic". Yes, innovation is incremental, this is an increment.

    • by blincoln ( 592401 ) on Monday August 04, 2014 @11:48AM (#47600149) Homepage Journal

      For some reason, the person who posted the article or the Slashdot editors linked to a bad knock-off video that removed 3/4 of the details instead of the actual researchers' video [youtube.com]. The real video makes it clear that they can also get results from a standard DSLR 60 FPS video by taking advantage of the rolling shutter effect. There's a fidelity loss, but it's a lot better than I would have expected.

  • by BitZtream ( 692029 ) on Monday August 04, 2014 @09:53AM (#47599145)

    The sensor and optics must have been ridiculously high quality and resolution for this to work. Sensor noise alone would almost certainly rule this out for any COTS consumer package. They certainly aren't doing it with CNN footage or old CCTV surveillance tapes.

    In which case, it's of no practical value since a laser mic would be far cheaper and more discrete.

    Cool from an academic perspective that they can use DSP now, but it's just more fun with a laser mic, same principals and theories, new less workable application.

    • This seems to work through soundproof glass... On the other hand, how big would be a camera able to record at this resolution and frame rates, and how close it must be?

      • This seems to work through soundproof glass...

        Glass that is pretty CLEAN.. You need really fast frame rates (6Khz will get you phone quality audio) and pretty high optical resolution. I'm just guessing, but you are going to need 3-4 pixels for any kind of reasonable S/N ratio that's listenable, so if the object you are looking at only moves a few nanometers with the sound, that means you need a minimum of two pixels per nanometer. To do that kind of resolution at say 10 feet, is going to require some pretty good optics. The resolution of the video

      • by lgw ( 121541 )

        Being able to do this from a recording is the magic. There are many places where video is recorded but not audio. Being able to recover the audio much later would be special. Also worrying, in places where you can record video but not audio without a warrant.

        • The problem is that the video is recorded in too low of resolution and too low of framerate to be useful. Any place just recording video and not audio is doing it with cheap ass bargain basement gear, hence where there is no audio gear already recording the sound they are missing

          • by lgw ( 121541 )

            Cheap cameras only get better. They were able to reconstruct voice from a good 60 Hz camera. If a policeman wanted to game the system, he could use a good camera to avoid the need for a warrant. Much like the NSAs "we capture everything, but only look at the data with (later) probable cause" excuse, someone might try an equally disturbing "we capture video, and only reconstruct the audio with (later) probable cause" in an attempt to game the system similarly. Uggh.

    • by gtall ( 79522 )

      Or Wolf Blitzer broadcasts. It is a fact that Wolf can actually suck information out of the ambient room and it is never seen nor heard from again. Think of him as an acoustical black hole.

    • except that a laser mic is ACTIVE monitoring and could be detected, whereas video is PASSIVE monitoring and is undetectable. It would also allow for monitoring in hostile environments, i.e. some guys around a campfire, where there may not be a nice object to laser against.

  • After all, video normally has an update rate of 24 - 30 fps. The sampling rate will be half that at about 15 Hz. If you have to have a video camera that can take pictures at audible sampling rates (very expensive), why not just bounce an IR laser off that potato chip bag?
    • by Anonymous Coward

      Because if your target is eating SunChips you'd risk hearing loss.

    • Because your emitting something sending that IR laser to do it. This is completely passive.

      • Yeah ... except that your spy van in your target's parking lot housing the high speed camera and its zooming lens will be obvious. With an IR laser you could do it from blocks away and nobody in the room would be the wiser.
        • And with good optics the camera could be just/nearly as far away. Laser is old know tech so places will have countermeasures and detection setup. Miniaturization can also come into place where you can shove the camera into a cell phone etc.

          • Miniaturization can also come into place where you can shove the camera into a cell phone etc.

            The laws of physics say otherwise. You can't get the resolution/quality out of a cell phone sized camera to do it at any difference, there just isn't enough in the lenses to do it.

            Then couple in the need for 6k frames per second being about the bottom line requirement for getting voice frequencies using this technique, its pretty useless in any situation that an alternative wouldn't work better.

            • From the looks of it a 1d array might work rather well and get the frame rates required. I didn't say generic cell phone just fitting in the form factor or close enough to not be suspicious. This could fit in existing security camera form factors (the 18 ish inch long all weather enclosures commonly used) that are so common as to be forgotten.

        • by mpe ( 36238 )
          Yeah ... except that your spy van in your target's parking lot housing the high speed camera and its zooming lens will be obvious.

          Why should a high speed camera look any different from a regular one?
    • by Anonymous Coward

      RTFA. They are using a high speed camera.

      Why not bounce an IR laser off that potato chip bag? Because you may not be able to. Windows have IR-reflective coating, the bag is not very reflective, the surface may be at the wrong angle. This is useful.

      High speed cameras are not that expensive. You can get 1000fps in a consumer camera for around $1000 these days.

      • You can get 1000fps in a consumer camera for around $1000 these days.

        Not one with a telephoto lens that will zoom in from yards away to resolve a potato chip bag. Your consumer camera spy equipment is a pipe dream...

        You'd be better off hiring a lip reader. Of course, all of this could be averted by closing the blinds in front of the window where the subject wishes his conversation to be private... I'm filing this one under "Dumb ideas".

        • I just thought of using microwaves to get a reflection from something metallic and vibrating. Wouldn't that work even through the blinds? (Or even walls?)
  • ...that's pretty effin' amazing. From video, 15 feet away. Not using a laser, FROM VIDEO! Lol.

  • The minimalist architects are in league with the spooks!

  • by Anonymous Coward

    The Cone of Silence has been breached Max!

  • by Anonymous Coward
    that glasshole is staring at your crisps.
  • This is the first thing I though of when reading the title.



  • Not only is it classier, but now serving your potato chips in a nice bowl is more secure.

  • How did they isolate the speech from the crunch of potato chips???? And, if this possible, there is no hope for anyone with the munchies!

  • Yelling MARY HAD A LITTLE LAMB, ITS FLEECE WAS WHITE AS SNOW at a houseplant, bag of chips, and glass of water is now research.
    • Laugh if you want, but if you yell MARY HAD A LITTLE LAMB at your rhododendrons, the NSA will know about it.

  • by Anonymous Coward

    The reason I find this interesting is that there are a number of locations (in the U.S.) where it is legal to record video but not audio. I happen to be against some of the laws forbidding audio recordings, because I think that it usually is used to protect against the recording of corruption. (try to open a bar in new york city, with live music, see what kinds of visits you get, and notice the correlation with fire, police, and food inspection after varying responses to those asking for money.) Would

  • Reminds me of technology used in the show Fringe by the Observers. Surely, no one will use this tech for nefarious purposes. Don't worry about it! Now, where's that Carbon Monoxide generator?
  • All they did was remove the laser to make the ease dropper unobservable. The tech is useless unless the camera has an insanely fine pitch resolution or the speech is so loud it causes large vibrations.

    Otherwise, they're just sampling motion at 2000-6000fps. It seems ridiculously processor intensive for something which could be better achieved using a high performance light meter.
  • This is what I instantly thought of: Eagle Eye [wikipedia.org]. The scene with the soundproof room.

  • We totally /.'d Youtube now - at the time of writing this, the video has been watched 2k times. After several hours of being on the /. front page.

egrep -n '^[a-z].*\(' $ | sort -t':' +2.0