Creating 3D Computer Graphics From 2D HDTV Camera 45
photon jockey writes: "Everyone knows that holograms are cool! But these three dimensional images are hard to make and need special conditions to view. A group from The University of Tokyo have taken a step toward 3D displays with this paper in Optics Express.
Using a HDTV camera they effectively capture the light rays passing through a plane from a lit scene and then reconstruct the three dimensional geometry of the scene. Some pretty movies are available from the same page to show this. The paralax is limited by the size of the CCD and the distance to the object.
From the paper: In the field of computer graphics (CG),the use of real images has been attracting
attention as a method of attaining image synthesis that results in a more photo-realistic
quality. This field of technology is called image-based rendering. The authors have
attempted to solve this problem, by applying the approach of 3-D display technology,
in which light rays are the most primitive elements of visual cues.
"
Re:What's new? (Score:1)
Again, sorry for the thread drift,
treker
A way of simplifying the concept of holography (Score:2)
I can't remember the maths of the Fresnel zone plate, but basically, if you interfere a point light source with a parallel light source, take the resulting pattern of light and dark rings and etch it onto a surface, light reflected off that surface is diffracted in such a way that it is focused, as if it has passed through a lens with its focal point at the same distance behind the plate as the point source was in front. This creates a virtual image of the original point source. You can view the object as being composed of such light sources each generating their own zone plate.
Like I said, I can't remember the maths, and the extrapolation from one point source to a whole object isn't exactly mathematically trivial, but I remember that the reason the Fresnel zone plate worked was very straightforward to grasp conceptually and helped the holography concept to fall into place. I always preferred to have a way to visualise something like that even when the maths clearly works out fine.
Re:What's new? (Score:1)
This has been another useless post from....
Comment removed (Score:3)
HDTV comming of age (Score:1)
But even though the HDTV consortium contains 19 formats it looks like there is an emerging universal mastering format 1080/24P [discreet.com] 1980*1080 pixels 12 bits per channel (48 bits per pixel) 24 frames per second. (you can most likely find this resolution in your display settings). The bandwidth needed for this format is about 220 megs per second. we would need a fire wire cable capable of more then 1760 Mbit to be able to hare real-time transfer (the current fire wire is capable of 400 Mbits). The point is that HDTV is not that far away. I mean this is uncompressed!
My computer is ready for it, and so is my display, but so far there are no consumer cams out there and renting a sony HDTV cam will cost you about 1700$ a day! even though many digital still cams can take substantially higher resolution images (i guess its the bandwidth....) it is going to be interesting to see who will be first whit a consumer HDTV cam... sony? canon?
But the thing that i am waiting the most for is 12bits per pixel, its not HDRI (High Dynamic Range Images) [isi.edu] but its a lot better then 8 bits.
Now let me rant a bit about resolution
It has taken about 15 years to get to this point (this tells you a bit about how serious tv people are about formats). And the really silly part is that a mpeg [cselt.it] image does not have a set resolution. the resolution is just a part of the header as a "please un compress this data to this resolution" statement. So since all DTV(digital tv) will be mpeg there is no need for a fix resolution. you could in fact just broadcast the data an let the receiving tv set, unpack in in any resolution! This would also make it possible for different programs to be broadcaster in different aspects. every one seams happy whit the fact that "wide screen" is 16:9, but nothing is filmed in 16:9, most films are made in 1.85:1, 2:1, 2.1:1 or even 2.34:1 so we will still get borders on our tv sets, borders that will be broadcasted and take up bandwidth! why cant i buy a TV set whit the aspect and resolution i want and then just de-code the tv signal the way I want?
Eskil
Re:Could Someone explain this to me (Score:1)
Nice URL (Score:1)
Nice outage Slashdot (Score:2)
Bad Links - Bad article (Score:1)
Image based Rendering rules (Score:1)
If you want to see a nice application of image based rendering, take a look at Elixir Studios [elixir-studios.co.uk]. They are using a technique called relief texture mapping to produce a game with really photorealistic graphics. The screenshots on that site are absolutely awesome. Forget Quake 3 and Unreal!
CU,
AC
Re:Porn! (Score:1)
Bingo Foo
---
Integral photography (Score:2)
This group is using the technique to extract depth using a single HDTV camera. That makes sense, although the approach is somewhat low-res. Depth extraction from stereo images is commercially available [ptgrey.com], and is an alternative to this approach.
No Quake application, sorry. (Score:2)
Software creating 3D from 2D (Score:1)
Synapix(www.synapix.com) is developing software for Windows NT/2000 called Synaflex, which creates a 3D model from a 2D camera path, then allows you to place 3D objects created in Maya and 3D Studio Max into the created scene.
Trully revolutionary stuff. Unfortunately, this product is aimed at strictly production houses, and no it's not going to be open source. The people who plunked down the backing money wouldn't be too pleased with us giving the technology away.
BTW...why wasn't the Slashdot downtime mentioned at all? Not newsworthy enough? If Microsoft goes down, we hear about it for weeks, but if Slashdot goes down...nothing?!?! Maybe I'll submit it as a story to Taco or Hemos.
Re:HDTV comming of age (Score:1)
The major difference being that they use an analog system and our system (american, titled DTV) uses a proprietary digital system that is completely different from any standard in ANY other country.
[sarcasm] Of course, since hollywood is so smart, our system will have intimate piracy protection systems built in so that those nasty pirates that steal movies off of the air (and thus degrade our capital and king, hollywood). Which incidentally will produce movies and shows that can only be seen on compatible sets, forcing all other countries to fall into stride. After all, they need our protection from those nasty pirates.. Even the casual pirates that want to watch a show more than their alloted one time. [/sarcasm]
-since when did 'MTV' stand for Real World Television instead of MUSIC television?
Re:Software creating 3D from 2D (Score:1)
Textures (Score:1)
--
"I'm surfin the dead zone
Re:HDTV comming of age (Score:1)
Bladerunner (Score:1)
Re:Textures (Score:1)
Re:What's new? (Score:1)
Re:blender (Score:1)
Re:More 3D models from 2D cameras (Score:2)
This article seems to be a technique to recover 3d information from a large number of slightly offset pictures.
The paper shows an illustration in which the camera lens is covered by a large number of circular/fish-eyeish lenses in a grid pattern.
By integrating the slightly different view points of each one, the 3d information of the scene can be recovered. The Japanese researchers (and this is where I started skimming...) seem to have several heuristics for performing this transformation in realtime. The application is that HDTV sets equipped with appropriate decoding circutry could allow the user to rotate the scene very slightly at will. Of course, this comes at a significant resolution cost... almost so much so that I wonder if it wouldn't be more efficient to send 9 or so exponentially placed pictures in one (reducing resolution by 3) and using simple morphing to simulate free movement between them.
Just a thought.
Re:Could Someone explain this to me (Score:2)
After developing the film, if you illuminate the film with the laser light from the same angle and of the same wavelength as before, then a 3-dimensional image of the object is created within the plane.
The cool thing about the reconstructed image is that it is 'true' 3-d. That means, as you move your head from side to side as you are looking at it, you can actually see behind the objects in the image.
To understand how this works, I struggled through the mathematics of how the light beams pass through the developed film as well as how the original image exposes itself on the film. I don't understand it conceptually, but the mathematics involved nothing more than trig.
My personal disclaimer: I didn't read the article that the abstract refers to (no PDF viewer), but based upon the abstract, it sounds like they are using a techniques similar to what I described.
Wow ! (Score:2)
----------------------------
Re:Could Someone explain this to me (Score:1)
Theoretically you could put a plane of microlensis onto the (TV-) screen, transmit a composition of the pictures taken, and the lenses will "select" the right image for your POV. - But then either the resolution is very low, or the bandwith really has to improve.
What's new? (Score:1)
Re:Wow ! (Score:1)
----------------------------
More 3D models from 2D cameras (Score:2)
You might have seen it already, but a forerunner of this technique more suited to getting a 3D model of architecture is described at Paul Debevec's Home Page [isi.edu], with the famous Campanile film. It's pretty amazing what can be done - perhaps one day digitised actors can stroll around a digitised building, with various other additions made to it...
Ford Prefect
Re:Could Someone explain this to me (Score:5)
It basically depends on polarization. All light is polarized, meaning that the electric wave and the magnetic wave that make up a photon are orthogonal (at right-angles within a plane) to each other. Most light is randomly polarized... that is, it bounces around at random with no structure to it. That's why lasers are commonly used in holography; it provides a polarized constant.
A traditional hologram is made by bouncing polarized light off an object (possibly from several different angles) and then exposing a piece of film to both the original, highly polarized light, and the light that is reflected off the object. When light is reflected, you change it's polarization to be (typically) parallel to the incident of reflection.
This makes miniature "grooves" in the image... they're virtual grooves, meaning they have no height, but all the same they selectively reflect only light of certain polarizations. Then, by shining the same type of polarized light on the exposed image, different angles of viewing select different polarizations, meaning different angles of viewing on the target object.
As for how this technology works, from what I can tell they're capturing the color and polarization of all the photons. This, combined with the width of the CCD, allows you to capture 3D information about the subject matter. If you were to add a source of polarized light to this thing, you could probably through the use of mirrors capture EVERY angle, just like a traditional hologram.
As a matter of fact, it doesn't even have to be visible light. Infrared will work fine, though you'll only get a rough gray-scale. But then, you don't need to be shining red/green/blue laser light around everywhere...
Won't the matrix people be mad at this! They spent Some Great Value Of Hard Earned Cash (SGVOHEC) to develop bullet-time, and now they can just use what turns out to be existing technology, making that expenditure of SGVOHEC a moot point.
Oh well, maybe the superbowl people will get with it next time so that my super-zooming rotating image of the QB won't jerk around like a 10-year-old computer trying to run quake...
Re:Wow ! (Score:1)
Re:HDTV comming of age (Score:1)
The upshot of this is that the borders caused by improper aspect ratios will only get transmitted once, since they will presumably be black till the end of program or commercial break.
Just my $.02
Re:Software creating 3D from 2D (Score:2)
I can see the 3d placement along the 2d camera path -- basically, you are deriving how the virtual camera should move and then blue screening that onto the real footage.
But: are you able to deduce accurate environment maps? Occulusion of the 3d virutal elements by real world items? This would seem to require true scene intepretation, while just deducing a virtual camera path from the real world footage sounds much more doable. Do you also deal with zooming and depth of field?
Microsoft Did It Already (Score:2)
See an example of some raw footage of a canyon [microsoft.com].
Then see the 3D model of the canyon he recovered from it. [microsoft.com]
Bullet time will still be used (Score:1)
What is more interesting to me is the possibility of using this system to lower the cost of motion capture: even if you don't use the footage in your movie it could be very useful for generating and compositing special effects.
Re:More 3D models from 2D cameras (Score:1)
Re:More 3D models from 2D cameras (Score:1)
Actually, if you read his thesis [isi.edu], you'll discover that a large proportion of the modelling process was done by the computer. A number of photographs from different angles were used as the source, and lines traced on them by a human operator. The actual three-dimensional structure and proportions were then determined from those by the computer, and relevant, direction-dependent textures recovered and applied.
Pretty interesting reading - a very different procedure to that described in the posted article, but who cares.
Ford Prefect
Oops... (Score:2)
reconstruct the three dimensional geometry of the scene
meant what it said.
Video Games (Score:1)
It was a sit-in F1 style racer, with a hige monitor up front with this weird plastic lens covering the monitor, and from the driver's seat, the 3D effect was pretty damn good.
this was around the same time (give or take a year or so) that another 3D game came out that used digitized footage of live actors in a game that was done like Dragon's Lair (view footage, hit joystick or button at the right time to progress to the next scene) however this game used a round circular table, with a spherical mirror behind it to project a "hologram" onto the playfield
anyone else remember these two games?
-- kai
Verbing Weirds Language.
Incorrect - you don't need polarization (Score:1)
You want to render a scene from a new viewpoints. If you know how all the light in a cube is moving, you can render the scene in that cube from any viewpoint by putting a virtual "camera" in the space and intersecting light rays with it (through a virtual "aperature", of course). Think of it as a parameterization problem, and you can see that this will become a big table lookup (with interpolation).
Now, for the most general case, you'd need to parameterize the space in 5D to get the data you need for unique-view rendering (3D position, plus 2D for which direction you're looking), but you can simplify things a bit by assuming that:
a) There's a box aroung what you're looking at.
b) you're outside the box.
The surface of the box is 2D, and since you're outside of it, you only need data at the surface. At each point on the surface you need 2D of parameterized information (angle and azimuth), so 4D all together.
In practice, the parameterization most people use is "two-plane", in that you have two parallel planes, with your object between them (well, that part's optional), and you store the rays by the two sets of 2D coordinates of where they hit the two planes.
Now, what this paper does is use Integrated Photography, which is just photography with one big lens (on the camera) and a bunch of really tiny lenses in a grid (in front of the camera). The camera, through its big lens gives one set of 2D coordinates, and each little lens creates another set of coordinates in its own very tiny space (look at figure 2 in the paper for what I'm talking about). So you end up with a sort of 2-level hierarchical paramterization.
Each of the tiny images, one from each integrated lens, is basically a little picture of the scene from the location of that lens. So you can turn this into a two-plane parameterization: the location of the lens determines where a ray hits the first plane (the integrated lens plane), and the position within the tiny image from that lens determines the direction the ray is going, and therefore the eventual point where it strikes the second plane. You need to know exactly how your integrating lens affects light, which they get by precalibration, from my reading of the paper. (You don't need the two-plane parameterization to render images. The authors of this paper work directly from the lens image. It's just important to realize that they are equivalent in concept, if not in the details.)
This page could be quite useful to see these concepts with images:
http://graphics.lcs.mit.edu/~aisaksen/projects/
BTW, the paper by Isaksen really should have been referenced in the paper this discussion is about, since it mentions integrated lenses in the 4D light field context, and even has an image of a synthetic integrated photo. I think the main contribution of this paper is probably the use of HDTV to do it in real time. That's pretty cool.
Finally, also on the MIT graphics web page is how to improve your rendering by getting a better approximation to your geometry. See the work under publications titled "Iamge Based Visual Hulls". This is another way of tackling the 3D video problem.
Bonsai Dogs (Score:2)
First we're squishing cats into jars, and now dogs between panes of glass, my god people, what are we doing to our pets?
Could Someone explain this to me (Score:2)
Mmhh, (Score:3)
Porn! (Score:4)
--
From this announcement... (Score:3)
Bryan R.