Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
AI Science

New Imaging System Creates Pictures By Measuring Time (phys.org) 63

An anonymous reader writes: Photos and videos are usually produced by capturing photons -- the building blocks of light—with digital sensors. For instance, digital cameras consist of millions of pixels that form images by detecting the intensity and color of the light at every point of space. 3-D images can then be generated either by positioning two or more cameras around the subject to photograph it from multiple angles, or by using streams of photons to scan the scene and reconstruct it in three dimensions. Either way, an image is only built by gathering spatial information of the scene. In a new paper published today in the journal Optica, researchers based in the U.K., Italy and the Netherlands describe an entirely new way to make animated 3-D images: by capturing temporal information about photons instead of their spatial coordinates.

Their process begins with a simple, inexpensive single-point detector tuned to act as a kind of stopwatch for photons. Unlike cameras, measuring the spatial distribution of color and intensity, the detector only records how long it takes the photons produced by a split-second pulse of laser light to bounce off each object in any given scene and reach the sensor. The further away an object is, the longer it will take each reflected photon to reach the sensor. The information about the timings of each photon reflected in the scene -- what the researchers call the temporal data -- is collected in a very simple graph.

Those graphs are then transformed into a 3-D image with the help of a sophisticated neural network algorithm. The researchers trained the algorithm by showing it thousands of conventional photos of the team moving and carrying objects around the lab, alongside temporal data captured by the single-point detector at the same time. Eventually, the network had learned enough about how the temporal data corresponded with the photos that it was capable of creating highly accurate images from the temporal data alone. In the proof-of-principle experiments, the team managed to construct moving images at about 10 frames per second from the temporal data, although the hardware and algorithm used has the potential to produce thousands of images per second. Currently, the neural net's ability to create images is limited to what it has been trained to pick out from the temporal data of scenes created by the researchers. However, with further training and even by using more advanced algorithms, it could learn to visualize a varied range of scenes, widening its potential applications in real-world situations.

This discussion has been archived. No new comments can be posted.

New Imaging System Creates Pictures By Measuring Time

Comments Filter:
  • by gweihir ( 88907 ) on Thursday July 30, 2020 @10:40PM (#60350191)

    The idea is about 60 years old.

    • honestly this looks like LIDAR :
      pulse of laser light to bounce off each object in any given scene and reach the sensor. The further away an object is, the longer it will take each reflected photon to reach the sensor.

      whats unique in this compared to conventional LIDAR ?

      • by narcc ( 412956 )

        whats unique in this compared to conventional LIDAR ?

        It's significantly less useful:

        Currently, the neural net's ability to create images is limited to what it has been trained to pick out from the temporal data of scenes created by the researchers.

        • I'm guessing that this method, once refined, will allow for better results from a limited data set.

          Or not, I'm just guessing, but that would make this novel compared to LIDAR.

        • This could give someone's AI "baby" eye beams [wikipedia.org], so it can grow up with Platonic senses.
      • it's just a single flash and recording a stream of returning photons.

        then using some algorithm to turn that into a depth map? which I guess was trained using sample sets? I mean it could distinguish a face in front of it to be a face from a banana in front of it, that I can imagine rather easily from the data it gathers. but I'm kinda confused how it can even with ai know the orientation of the banana if the summary is correct at all, because if it's a single flash of light and a single point sensor (the s

        • by q_e_t ( 5104099 )
          It might be using shape from shading under ideal lighting for the conventional photography. Not that I've read the article! I'm just quickly thinking what I might do given the parameters described. The problem would be shape from shading isn't perfect.
        • by aturpin ( 7094807 ) on Friday July 31, 2020 @09:11AM (#60351111)
          You got it perfectly! The system can tell the orientation of the "banana" (as you say) because it uses not only information about the banana itself but about all objects appearing in the scene. This allos breaking the symmetry you comment. Having a single sensor and no scanning parts makes that the system works orders of magnitude faster than a LiDAR (that scans the scene). We used only one to demonstrate that that's enough to form an image. We don't claim that this is better than LiDAR in terms of resolution, but creating an image using only time information is a conceptual change on what the requirements for imaging are. This means that now you can use any pulsed source, like a wifi antenna, to create an image, for instance. Of course, adding more sensors will improve the image, imagine a camera where every pixel has this capability!
          • by Shotgun ( 30919 )

            Except, it doesn't seem that you created an image. It seems that you mapped out that there are a set of surfaces, each at various distances from the surfaces. Then you used "AI" (ie, glorified filter) to categorize the "surface distance" readings into categories to matched known physical configurations of people and things.

            With a picture, if you replaced the person with a dog, a person viewing the output would be able to determine that it is a dog. This is not a "picture". No one would be able to deter

      • whats unique in this compared to conventional LIDAR ?

        They designed it like a fish finder, and then instead of programming the formulas, they trained an AI, so it is crappier but easier to make.

        • They stuck some crummy DeepFake for home furnishings on top of the LIDAR

          It is as if they stuck pictures of Billy Bass over the fish finder output and claimed they captured the image

      • In lidar, the scene is illuminated by a scanner (a laser beam that moves up & down, back & forth) which lights up the one tiny area of the scene (one pixel) at a time. With their approach the whole scene is illuminated at the same time with wide flashes of light.

      • by rednip ( 186217 )
        I'm no excerpt on LIDAR, but doesn't it work by shooting out a laser, getting back the reflected light and doing the math on distance and time? This, as part of an image capture, somehow measures distance from the photons naturally emitted, seems to be a lot better way of doing things.
        • by rednip ( 186217 )

          It seems that I was a little quick on what it does. A second look seems to show that it uses LIDAR data to 'train an AI' to recognize individual objects.

          Offhand of course, I'd claim it was more incremental than revolutionary, but could see how this could be useful for self driving cars. By not relying on video, it could reduce some privacy issues (some) and be useful in a number of applications.

      • That LiDAR requires to scan the whole scene point by point to create a point cloud and here you don't have to. You get the image in a single-shot :)
        • by Shotgun ( 30919 )

          Not quite. You don't get an image. You get a graph of photons returned per time period. You now know that there are surfaces out there and particular distances, but you don't know what directions the surfaces are.

    • I imagine dolphins, whales, and bats have been doing it for a lot longer than 60 years.

      Without RTFA, I suspect the difference here is that they're doing it with a single pulse of light. LIDAR is typically scanned (so are CHIRP radars and sonars). You send a laser out in a certain direction, measure the time for the return signal, rotate a bit and send the laser out in the new direction, etc. to build up a 3D map. If you have a phased array of receivers (and a ton of processing power), you can send out
    • First off it should be obvious that's not the case given that it's published in a decent journal AND it's a given that everyone has heard of Lidar. Second, if you bother to read the article (instead of relying on the rather misleading description) you'll see that their method doesn't work the same way as LIDAR.

    • by AmiMoJo ( 196126 ) on Friday July 31, 2020 @07:04AM (#60350829) Homepage Journal

      This does seem to be novel and not simply just another LIDAR.

      Current LIDAR systems using a single point detector scan the environment in some way, usually by having the light pulses scan over the area so that one vector can be measured at a time. That's why you see spinning optics on LIDAR systems, they are scanning the laser pulses both horizontally and vertically. An alternative is to use a series of special patterns that when combined allow you to get a value for each vector.

      These guys have found a way to eliminate the scanning. They flood illuminate the scene and then make a temporal histogram from the returns. Using AI that is trained to recognize what the histograms of various different objects and poses look like they can turn it into an image.

      That could be very useful for things like industrial machine vision where you want to check if something has been manufactured or installed correctly, for example. They are also looking at possible consumer applications such as adding 3D data to smartphone cameras.

      • by gweihir ( 88907 )

        What you describe is a "LIDAR scanner". LIDAR is in principle just the ranging idea.

        The usual bastardization of the language at work.

      • by idji ( 984038 )
        LIDAR contains directional and depth information in the results, because at any one instant it receives light from only one, known direction.
        This technique receives light from the entire scene at the same time- there is no directional information, just a histogram of depth. The AI reconstructs the directional information from the depth histrogram. The potential here is amazing.
        This could produce color as well by using 3 detectors with RGB filters.
    • by ceoyoyo ( 59147 )

      Sure. If LIDAR used an omnidirectional light source instead of a laser.

      To be honest, LIDAR is a dumb idea because the Phonecians invented eyeballs in 2500 BCE.

    • by pz ( 113803 ) on Friday July 31, 2020 @07:58AM (#60350927) Journal

      The summary is a bit obscure on the point, but the article is clear (and open-access): whereas LiDAR uses a scanned beam and time-of-flight to develop a 3D image of a scene, this research uses pulsed flood illumination, not unlike a camera flash. That is to say, there is no raster scanning of the scene. They collect no spatial information but are able to reconstruct the scene structure just by time-of-flight information alone. In contrast, LiDAR, with a scanned beam controlled by the system at all times (just like RADAR), uses the position of the beam as a critical part of 3D reconstruction. This research is different, and because of that is pretty impressive. They also specifically mention LiDAR in their article as a means to develop ground truth for the training algorithm, so it isn't like they don't know about what LiDAR is and how it works. This research is closer to scene reconstruction through echo-location, but with light.

  • by josquin9 ( 458669 ) on Thursday July 30, 2020 @11:24PM (#60350261)

    Having worked to develop several Ag Tech lidar products over the last decade, I can assure you that this is not "new" technology. The price keeps coming down, and the quality keeps going up, but the technology has been out there for decades. It's the heart of many of the collision detection systems in new cars, so you may actually already own a lidar sensor without being aware of it.

    • Actually TI had a chip which does this already, it is a 3D imaging chip with Laser illumination. Can be pretty accurate also.
    • This works different than lidar, it doesn't have a scanning light source, instead it use a flashes of light that illuminates the whole scene simulataneously.

    • by AmiMoJo ( 196126 )

      I doubt this particular technology has existed for decades as it's only fairly recently that AI systems have got good enough to do this kind of scene/object recognition with the accuracy and in the time frames they are talking about.

      Also all car anti-collision systems use radar, not LIDAR. Well, a few of the experimental self-driving systems use LIDAR, but all consumer cars use radar. The first LIDAR equipped car will be released by Volvo next year.

  • Either way, an image is only built by gathering spatial information of the scene.

    Except for holograms. They work by creating an interference pattern on/in the detector(film). Pedantic, I know, but let's be accurate.

    • by ceoyoyo ( 59147 )

      Time of flight sensors usually do this as well. It's super expensive to actually time a light pulse, so most use interference between a transmitted and a reference pulse.

  • Hmm... (Score:4, Insightful)

    by CaptainLugnuts ( 2594663 ) on Thursday July 30, 2020 @11:55PM (#60350317)

    What you're saying is they've invented a much shittier Kinect?

  • I am quite impressed by the idea, and the optics part (which is my area) is pretty sound.
    But I don't know what to think of the image reconstruction through AI. All they have is the distance data. As they explain in their publication any object in the same distance gives the same signal.
    This means of the whole scene is rotated it would produce the same signal. Same if parts of the scene in a certain angle range are.
    This seems to be just too limited data to determine what is in front of the detector. A lo
    • I mean, the system gets no directional information. Just the distance of reflecting objects, not in what direction they are. If it senses the reflection of an arm the arm can be pointing in many different directions, as long as it keeps the same distance to the sensor. The algorithm just compares it to the arms it has seen before and selects something that fits.
    • by aturpin ( 7094807 ) on Friday July 31, 2020 @09:20AM (#60351141)
      As a matter of fact, different objects give different signals, even if they are at the same distance. The peak position of the signal might be at the same place, but the signal itself is different. Even the signals for two different people at the exact same place are different: https://www.nature.com/article... [nature.com] One of the keys is that the scene has moving objects and a fixed background, and we exploit that to recognize where objects are and shape do they have. But you're correct by saying that a lot of information comes from what the algorithm has seen before, otherwise the problem would be not possible to be solved.
  • I'm not expert with this technology, but I try to share what I know. Please correct me if you see any mistakes.

    With LIDAR, you shoot multiple lasers to different locations. This is not LIDAR.

    With this technology you first need to collect background 3D data, e.g. with LIDAR. Then you shoot pulses or light, which you collect from single point and with this to my understanding you can create a 3D movie of moving people. With top equipment I think you can get into resolutions of 10 cm.

    There is also older techno

    • You're right that LiDAR shines a laser at different directiosn to create an image. Here we don't collect first the background 3D data; we shoot light pusles and collect them with the single point while another 3D sensor collects the same scene. With this data we train an AI algorithm and after it is trained, we remove the 3D sensor and only collect with the single point. The data from the single point + AI algorithm is what creates the 3D image. The TED presentation you shared is stunning! However, they d
  • by Sqreater ( 895148 ) on Friday July 31, 2020 @03:13AM (#60350549)
    In reality it is only just guessing about reality from the temporal data and what it has stored. It's accuracy depends on how close the stored data is to the temporal data it acquires. Who wants that? It even sounds dangerous.
    • Any mathematician can tell you it would require more information to get a true 3D picture. Distance from a single point doesn't give you any direction - in this case the AI may already know the layout of the lab from the training data, but without that you'd have nothing.Two points would almost be enough to locate positions in 2 dimensions (other than whether an object is left or right of the line connecting the points), etc.So, the new thing here is the AI inferring positions from its surroundings - which
      • What happens if you move a table in a lab? Doesn't that invalidate the database and therefore greatly diminish the guess about position?
        • If you change completely your environment, it will stop working, at least with the algorithm we have used at the moment. We have tried to move some objects (like the white box you can see in the figures of the article) from the back of the image, and the algorithm still works, but changing the whole room will certainly break the imaging.
    • Note that we do not store any data, we train an AI algortihm that learns from examples, which is different to compare new data to stored data. As many technologies, one always look for a compromise. It won't be possible even in the near future to have a 3D sensor with megapixel resolution that works at 1000fps. But for many applications you don't even need that. If you have a static scenario, for instance a fixed location in a public spacfe where you want to obtain images or tell how many people are there,
      • The data is virtually stored in the learning of the AI. If the environment were to change, it's learning based on the previous examples would be in error. And how would you know that if you are trusting that the environment is truly static and the examples it learned from are still valid ones.
  • So basically, its fixed point LIDAR, passed through a potentially non deterministic AI filter. I guess over time it will be improved to use passive photons from a point in space measured from two differe... oh, wait....
  • Its it really measuring time? Or is it really measuring object distance utilizing time as a measurement?
  • This sounds like a somewhat interesting problem. If you ignore the AI aspect and think about how you could solve for "what is in the room" from the timing frame data with direct mathematical/computational approaches, I can see how you could probably make some reasonable estimates as long as you are allowed to make a few assumptions.

    For example:

    Start with a mostly empty room, assume it is a normal "rectangular" room with 90 degree angles, etc, and that you can filter out / ignore "secondary" reflections

    • It is also somewhat similar to the pysical setup for getting sythentic aperature radar (SAR) data: https://en.wikipedia.org/wiki/... [wikipedia.org] In the sense that it it is working with a single-dimensional signal of data.

      But differences include:

      • * Highly doubtful it measures timing of visible light to the fraction of a wavelength necessary to include phase information.
      • * Doesn't necessary correlate multiple pulses from different positions together, and especially not totalling with phase information to cancel out i
  • People are comparing this to LIDAR. While it is similar, this is not LIDAR.
    LIDAR actually captures a 3D image from its perspective.
    This sounds like it captures where objects are and their orientation.
    This Time Measurement system has an advantage where you can look at the picture from any angle and you will see any object that was in the original "picture" in full detail from that perspective.
    With LIDAR the image quickly starts to fall apart if you are only imaging from 1 camera.
    Even imaging from 2 ang

No man is an island if he's on at least one mailing list.

Working...