Content-Aware Image Resizing

An anonymous reader writes "At the SIGGRAPH 2007 conference in San Diego, two Israeli professors, Shai Avidan and Ariel Shamir, have demonstrated a new method to shrink images. The method is called 'Seam Carving for Content-Aware Image Resizing' (PDF paper here) and it figures out which parts of an image are less significant. This makes it possible to change the aspect ratio of an image without making the content look skewed or stretched out. There is a video demonstration up on YouTube."

nice!

[White Shade]

It seems like a little bit of work is left to make it as completely automated as you would need to have it just "always work" on any platform or device, but it seems like they're already working on that...

Other than that though, that's pretty awesome... I'm sure there's more instances where it doesn't look right than what they showed, but it's definitely cool how well it works as it stands!

I can imagine it would be extremely useful for ex-boyfriends or ex-girlfriends; just load up all their photos of them and their ex, wave the magic eraser, and *boom* you don't have to delete all your old vacation shots ;)

I wonder how well it would work for the porn industry too; nice automatic resizing of breasts without ruining the picture! Fetishists will be SO happy! :)

Slightly Strange

[JamesRose]

Okay, I get that they remove the pixels with least energy, so the unimportant information is lost when shrinking, it kinda works, looks a bit strange, but it's okay. however, when they make an image larger they also add the least information so you end up with a large image- but the useful information is the same size and the extra/useless low energy or background gets duplicated- to me, I think thats kinda pointless, I mean, you're adding stuff you've analysed and found NOT to be the focus of the picture. This may work for pictures with no obvious background, but lanscapes like one of the examples, have such an obvious background that only that gets enlarged and just gives you more background. You may aswell just add a nice blue frame round the edge of the picture to make it fit.

DP Approach

[xquark]

This method is quiet interesting, though it falls over in situations where the detail level
or entropy of the background is as great as the foreground. Also the paper doesn't go into
too much details about the dynamic programming approach they used to find the path of least
energy, I guess that aspect of it is patentable. Another thing they could investigate is the
use of diagonal seams instead of just staggered vertical and horizontal seams.

All in all a very interesting read.

Re:Slightly Strange

[Anonymous Coward]

when they make an image larger they also add the least information so you end up with a large image- but the useful information is the same size and the extra/useless low energy or background gets duplicated- to me,

According to the video, the added background information is actually the averaging of the extra "low energy" information around it. So it's not quite duplicated.

Re:nice!

[Allicorn]

It seems like a little bit of work is left to make it as completely automated as you would need to have it just "always work"

Completely right, yes. The images in the video have been selected to show this technique in the best possible light. There's a great variety of images that'll really not work quite right with a completely automated treatment. Speaking from experience having implemented this last week.

That said, as pointed out in the paper there's plenty of room for a higher level of analysis over the top of the basic seam-carving procedure. The function used to calculate the energy of a given pixel is easily swapped out with any one of dozens of different approaches. A more user-friendly implementation could attempt seam-carving based on a number of different feature maps and work out which is likely to produce the least distortion for a given image.

Anyhow, cracking bit of work IMHO, with boatloads of potential applications.

Alli

Gimp!

[larry bagina]

Although they demonstrated on Windows, a friend of mine is one of their graduate students and was peripherally involved. He said it was originally developed as a GIMP plug in, but moved to a separate Windows app to show off the realtime resizing, etc. Hopefully they'll release the GIMP plugin? More likely Adobe will write them a check and license it to make sure that never happens.

Re:The Commissar Vanishes

[larry bagina]

It could be worse.

In December 2001 The New York Fire Department unveiled plans for a statue based on the photograph [wikipedia.org] to be placed at the Brooklyn headquarters. In an effort to be politically correct, the statue was to include black, white, and Hispanic firefighters. However, it was cancelled in an outcry about rewriting history -- the depicted firefighters are white.

My Implementation

[The New Andy]

I thought it was pretty cool, so I made my own version after seeing the video. It obviously won't be as awesome as their one, but if you want to play around with it, you can get my C source [ultra-premium.com] and have a play around. It is GPL3.

Re:A picture speaks a thousand words...

[random735]

while this is technically true, you're also rearranging the relative positioning of those pixels. cropping something out doesn't change the relationship of what is left in the photo (though it may remove critical details).

if you have 3 people in a picture and you crop it down to 2, you've erased a person, but you haven't changed who is seated next to whom. if you use this method and the middle person is erased, you make it appear as though the outer two people were in fact seated next to each other when they weren't.

we are used to the idea that a picture can be cropped (mentally considering what might be just outside the frame). We aren't yet used to the concept that the photo has effectively been cut and pasted together to create new relationships between the objects in the photo (though of course photoshop is getting us there).

to continue your analogy, if we take:
the quick brown fox jumped over the lazy dog

and drop letters, we can create:
the cow jumped over the dog

whereas "cropping" might let us say:
the quick brown fox jumped

I think it's clear that one of these is more misleading than the other, though in both cases you're just removing information. (in one case, some of that information happens to be spaces between letters/words)

some code

[Arthur B.]

Too much caffeine in the blog, couldn't sleep... I can't get my hand on the paper but the youtube presentation was extremely clear and I just wrote this C code based on libgd2. Basically it lowers the height of an image by 1 pixel, you can run it multiple time to remove more line.

http://rafb.net/p/jinioy45.html [rafb.net]

(yeah my coding sucks but it produces awesome results and I reversed engineered the algorithm from youtube so please grovel...)

I'll improve it soon to remove an arbitrary number of line, horizontally or vertically
  - no recalculation of gradient, only the gradient near the line needs to be recomputed
  - precomputes a file that store the order of the pixel needing to be removed

I need help with something though, I understand how the algorithm can precompute a file which says in which order pixel should be removed, but I don't see how this can work in *both* direction. Suppose you want to reduce vertically and horizontally at the same time, the horizontal change should completely break the precomputed vertical changes. How would you handle that?

Re:some code

[Arthur B.]

here's an example produced by that code (this is Maine)

original
http://img96.imageshack.us/my.php?image=testxq4.jp g [imageshack.us]

somewhat reduced
http://img361.imageshack.us/my.php?image=outew8.pn g [imageshack.us]

very reduced
http://img484.imageshack.us/my.php?image=outas2.pn g [imageshack.us]