Getting Around in GIMP - Heal Selection (Resynthesizer)



Missing Tower! (from Eiffel Tower by Terrazzo on Flickr)

I've been reading through a ton of hyperbole about Adobes Photoshop "Content Aware Fill" for some reason lately ("magical", "incredible", "amazing" and others are fun to read).

I don't think I would be too far off in assuming that Adobes implementation is likely based off the wonderful work of Dr. Paul Harrison. Dr. Harrison's PhD thesis just happened to yield the code that gave GIMP users the Resynthesizer plugin, which has provided us a "Content Aware Fill" for quite a while now (since before 2005).

If you've ever spent any time with the Clone Tool or Heal Tool in GIMP, and haven't tried out Heal Selection with Resynthesizer, then you might find the results of this filter very helpful.

The rest of my GIMP tutorials can be found here:
Getting Around in GIMP

I have since looked at a newer algorithm for doing this, Inpainting [patch-based] in G'MIC.
I think it’s now a better option for me, check out the post here:

Getting Around in GIMP - G'MIC Inpainting (Content Aware Fill)

What does it do?

The basic premise behind Resynthesizer is that it will sample image textures from a given area, and will build a new texture from that data. The texture will be combined from segments of the selection in a random orientation. The plugin will also clean the feathering between segments to maintain a relatively consistent contrast, thus minimizing its visual impact on the result. (This is a very simplistic explanation of what's actually going on under the hood - there are some neat pixel neighborhood comparisons going on, actually).

So to illustrate, here is a small 100 × 100 px sample of some water:


A small water sample

Now running Resynthesizer on a larger canvas area (450 × 450) will yield:


After Resynthesizing

This produces a pretty nice new texture! Random enough that there are no discernible repeating patterns that would give it away as a rebuilt texture.

An interesting use of Resynthesizer for seamless textures explored by GimpChat.com user lylejk is described in this thread, for example.

Now, creating good seamless textures is handy, but what does it have to do with healing selections in our image? Well - this same code can be used to fill-in areas of our image from data nearby. This is exactly what the companion Script-Fu "Heal Selection" will do, with some neat options...

Heal Selection

So let's have a look at healing scenes from an image. Once you've installed the plugin and associated Script-Fu, the option will be available from:

Filters → Enhance → Heal Selection...

Before we fire up the command, though, we just need to look at a couple of things. I'm going to use a cropped portion of a larger image to demonstrate:


Let's remove this person from the beach

Now, the normal way you might use Heal Selection to remove this person would be to create a selection around them, and then just run the command. So let's first create a selection around the object we want to remove.


I've colored in the selection to make it more visible

With our selection made, we could then start the Heal Selection dialog.


Before we run this, command, though, let's have a quick look at the options and what they mean.

The Context sampling width will allow you to choose how far out from the edges of your selection to sample for textures to fill with. In my example I am saying to use the surrounding image up to 75 pixels away from the edges to create the new texture that will be used to fill up my selection.


Selection to Fill and the 75px surrounding area to get the texture from.

If I run the command now, then what will happen is that the script will use the surrounding 75px area to build a texture to fill in my selection, and it will randomly fill the selection area for me.

So let's see how good of a job it will do with these (mostly) default settings:


Holy crap! (mouseover for original)

Well, wow. This has certainly done an admirable job in my opinion! The rebuilt portion of my selection contains enough randomness to blend very nicely without any artifacts to give it away.

The thing you really want to take away from this section is the use of the Context sampling width to adjust how far away from your selection to sample for the new texture. If you make the value too small for a large selection area, it may end up looking funny (due to having a small amount of pixels to sample from in order to fill a large area).

Well, we certainly can't stop here - so let's experiment even further with the entire image:


I have already removed the person in the lower right frame of the image, and I will also remove the people in the water, and the man on the shore. I'll leave the small girl there for aesthetic reasons.

You can also select multiple areas to have Heal Selection fix them all at once if you'd like.

For instance, I will choose the man in the red shirt and the people in the water at the same time (don't forget to remove their shadows and reflections in the water):


I used a 50px sampling width, and selected both the people in the water and the man in the red shirt at the same time (mouseover for original)

If you want to be able to quickly visualize the sampling width, once you've made your selection you can grow it to the width you want using:

Select → Grow

then just Undo the Grow to get back to your original selection.

Heal Selection works wonders provided that you have a relatively consistent type of surrounding textures to work with. It even does wonders where you might not think it would at first.

While playing around for this tutorial, I said "What if I wanted to remove the house in the upper right of the image?"

Well, turns out Heal Selection did a dandy job of fixing that right up for me:


Wow. No more house! (mouseover for original)

One other quick example, a nice serene beach scene (try guessing where the changes were made before you hover your mouse over it):


A nice quiet(?) beach scene... Beach Umbrellas by Randy Pertiet on Flickr

A Few Tips

There's a few things to keep in mind when using Heal Selection that might help you to achieve better results.

First is that depending on the geometry that you want to heal, the Sample from options will dictate from where around your selection you want to source pixels. The default will be All around, but the other two options can come in handy: Sides and Above and below.

This will restrict Heal Selection to choosing pixels only from either the sides, or above/below the current selection. I've found this helpful especially in places where there is a hard delineation in an image (trying to remove an object that crosses over the horizon, for instance).

I've used these options oftentimes in order to only partially remove an object from a scene (as part of a longer workflow). This is very handy around areas with a hard delineation.

For instance, remove the lower half of this telescope post to rebuild the brick wall behind it:


As part of a workflow, I only want to remove the post in front of the brick at the moment

Here is using the default Sample from: set to All around:


Notice the inclusion of part of the post in the lower shadow area of the selection. (mouseover for original)

Now if I restrict the sampling area to only the sides of the post, I will only get rock textures to create my fill textures with, which is much more desirable here:


Restricting the sampling to only the sides of the selection (mouseover for original)

This yields much better initial results for me to work with. The same method can be used for vertical edges as well (in which case you'd only sample from above and below).

A second tip that might be helpful is that you can run Heal Selection multiple times if needed in an area. Because the textures are being sampled from the image nearby, the overall grain/quality of the results will be close enough to the original to not stand out (though this entire process is very destructive to an image - so keep that in mind).

For instance, if I wanted to touch up the results from removing the post just a little more, I might want to remove the shadowed rock along it's top edge. No problem - just re-select a new area, and run Heal Selection one more time:


Second healing run to remove the shadow on the stone (mouseover for original)

Healing Panoramas


Of course, just like Adobes "Content Aware Fill", we can also use Heal Selection to extend areas of panoramic images where we are lacking source material. For instance, this is not an uncommon result of some panoramas:


Using Heal Selection, we can have Resynthesizer rebuild the missing data for the sky and grass, using the nearby image texture data. In this case I just selected the black background areas, then grew the selection a bit to overlap the actual image.

Then, because I wanted to just sample from areas directly above the grass, and directly below the sky, I set Sample from to Above and below.


Heal Selection with sampling from above and below

This can be especially helpful to those creating 360° panoramas, and either don't capture a full, clean nadir or need to remove a tripod leg... (I had previously written about this image in my post On Little Planets).


I had captured the full nadir, but needed to remove tripod legs...

With the nadir frames missing, I can just run Heal Selection on the center of this panorama (2048 squared, set the sample width to 75px), resulting in:


Rebuilding grass for this nadir is a strong point of Resynthesizer

Rebuilding a random grass texture to fill in this nadir is an especially strong feature of Heal Selection, and it shows here.

Another note is that Heal Selection will not fill an area with texture if it is transparent. So if you have a layer with transparency in areas that you want to fill, you'll have to remember to remove the transparency from that layer first:

Layer → Transparency → Remove Alpha Channel

In Summary


Heal Selection/Resynthesizer is a killer filter. Due to the nature of GIMP plugins and scripts, it might not be immediately obvious to a new user that this is even an option. (We really need a better method of managing scripts and plug-ins in GIMP).

So here is a small list of links that might help:

[Update 2012-12]
Over on the GimpChat forums, I walked through how I would approach an example by "slicing" up the area to heal based on surrounding textures. I thought it might be helpful to others who read this post, so check it out here!



Where it Works Well

Anywhere you have a need to fill from a rebuilt texture, and have at least something nearby to sample from. This could be mitigated through creative use of layers to bring appropriate textures closer to your healing area.

Large areas of similar texture, where you want to remove objects. Sky, grass, walls, etc. work very very well to resynthesize textures to heal your selections.

Where it Needs Help

Edges can be problematic to get an automated healing around. This can be mitigated by creative selections and directions for sampling, though. If your only other option is a lot of time with the clone and heal tool, then this method shouldn't be any harder or take any longer. (ie: try this first to help you get on your way to a clean heal).

If you've had a chance to try this filter out, please feel free to share the links with us in the comments below!

Help support the site! Or don’t!
I’m not supporting my (growing) family or anything from this website. Seriously.
There is only one reason I am writing these tutorials and posts:
I love doing it.
Technically there is a second reason: to give back to the community. Others before me were instrumental in helping me learn things when I first got started, and I’m hoping to pay it forward here.

If you want to visit an ad, or make a donation, or even link/share my content, I would be absolutely grateful (and tickled pink). If you don’t it’s not going to affect me writing and posting here one bit.

I’ll keep writing, and I’ll keep it free.
If you get any use out of this site, I only ask that you do one thing:
pay it forward.


12 comments:

  1. Excellent! You should post this in the Digital Photography School site. They don't have many Gimp tutorials and this should be very welcome :)

    ReplyDelete
    Replies
    1. Thanks, I'll head over there and see if I can get it posted! :)

      Delete
  2. Pat, I'm having a tough time getting some older extensions to work in GIMP 2.8. Is there an updated version of Resynthesizer? I've used this before in 2.6, to extend/contract some areas without distortion, but I had no idea you could do content aware actions, a la PS.

    Thanks for another outstanding tutorial. I agree with Leandro, this needs to be published somewhere. You are contributing to the new GIMP Magazine, aren't you?

    ReplyDelete
    Replies
    1. Partha has a nice collection of really, really useful plugins compiled for Win7 (x86, x64) as well as OSX:

      http://www.partha.com/

      I would recommend his plugins pack (you can also just download the .7zip if you don't want an installer - so you can pick and choose which/what to install).

      If you're looking at scaling areas in an image, have you had a look at Liquid Rescale?

      http://liquidrescale.wikidot.com/

      The seam carving stuff is just amazing...

      I had contacted the GIMP magazine crew, and might try to either put something together, or re-work some older stuff for them. If I can stop being lazy, I might be able to get something into issue 2 or beyond.

      Delete
    2. Liquid Rescale; that's actually the extension I was thinking of. I'd heard of resynth before, but I don't think I've actually put it to work. Thanks for the update info; I'll definitely try it.

      Delete
  3. Hi just installed resynth and practiced it worked great. The problem i have (and this is going to sound really stupid but am VERY new to all this) but how do i get rid of the marching ants around my selection?

    ReplyDelete
  4. Hi there,

    Heal Selection does not work the same way for me in Gimp version 2.6 on the Mac. I downloaded the first picture from your demo of the woman on the beach and used the same parameters as in the example. My healed selection becomes a blurred section of colours that seem to be taken from the woman (selected figure). For example, the pattern from her pants gets replicated throughout her image. Or her hands will be moved to a different area of her image. Any ideas what is wrong?

    ReplyDelete
    Replies
    1. This is likely due to the different versions and scripts that have been floating around... I know that previous version of the "Heal Selection" script approached things in a different way (and newer versions of Resynthesizer had new options).

      Perhaps you could try a newer version again and see if that alleviates the problem? (There are pre-built native OSX versions you can now use without requiring X server - try one of the Partha builds in the links above!)

      Delete
  5. It wasn't the version of the script, it was the version of Gimp. I installed version 2.8.2 of Mac Gimp from partha.com and ran resynthesizer again on your demo picture. I now get the same result as you do.

    Thanks!

    ReplyDelete
  6. Hi, I have 2.8 and I have Resynthesizer, but can't seem to make a picture larger like you do in the first picture here.

    ReplyDelete
  7. Hi Pat,
    Amazing tutorial. Thanks for the detailed instructions and the samples.

    Thanks,
    Jo

    ReplyDelete
  8. Wow, I loved the way you explained things. I've been using the heal selection tool for over a year now, however you explanation gave me a little more insight that will help me immensely. Thanks for taking the time to share your experience and knowledge on this topic.

    ReplyDelete