Professional Photographer Magazine Web Exclusives

Masking Software, a Closer Look

By Stan Sholik
[This feature elaborates on "Hideaway," the masking software review by Stan Sholik in the March issue of
Professional Photographer magazine.]

Every image that requires masking in order to composite it with another image, or to create a layer mask for image enhancement, seems to present a unique set of problems. Some of these problems can be handled with the tools in Photoshop, but the tougher ones are generally handled quicker and easier with third-party software optimized for the needed task.

onOne Software Mask Pro 4

This wedding image was made with available light in the church, but I want to eliminate the busy background and replace it with an Old Masters digital background from Owen's Originals. The problem is the original image has a lot of the same hues in the areas that I want to keep and the areas I want to drop. What I need to do is outline the couple with the Pen tool rather than try to use “keep” and “drop” colors. Mask Pro’s Magic Pen tool is perfect for this.

Open the image in Photoshop, unlock it by double-clicking and rename it. Then make a copy of it as a precaution. It’s always best to work on a copy rather than the original. I add a Layer Mask at this point and leave it highlighted. Mask Pro will create a layer mask. If you don’t create a layer mask, Mask Pro will eliminate the background in the image file itself. If you choose to work without a layer mask, be sure to work on a copy!

I also create a layer with my new background. When the mask is completed in Mask Pro, the software will then allow you to preview it against this background.

All images ©Stan Sholik; click for larger view.

Once the image is set up in Photoshop, selecting Filter>on One>Mask Pro opens the file in Mask Pro’s interface. The third tool on the left in the tool palette is the Magic Brush.


The Magic Brush really is magic. By clicking on points along edges, it follows the contours and creates a path. In really tough areas where there is no contrast between colors, such as under the bride’s chin, you drag the Magic Brush to create the path. The tool options allow you to adjust the brush size, how strongly it is attacted to an edge and the softness of the resulting mask.

After closing the path, the brush turns into a hammer and one click erases within the area. I erased all around the couple to eliminate the background and it took less than five minutes. Mask Pro used only a few seconds of that time to create the mask.


I did a quick check to see how the image looked against the new background, and there was still a slight halo of background color in areas, particularly the groom’s hair and the bride’s veil. These I eliminated with Mask Pro’s Chisel tool, sixth tool down on the left.


I did another check of the couple against their new background in Mask Pro and it looked great.


So I clicked File > Save/Apply and was immediately brought back to Photoshop with the layer mask in its proper place.


Vertus Fluid Mask 2.0

With this image of a fall scene in Connecticut, I want to replace the boring cyan sky with one that is more interesting. The problem is that the sky shows through lots of places between the branches and leaves of the trees. Fluid Mask is uniquely equipped to deal with this problem.

After opening your image file in Photoshop, it is a good idea to unlock the image and make a copy before working on it. Fluid Mask will make permanent changes to the image and this always leaves open the possibility of accidentally saving over your original file by mistake.


Selecting Filter > Vertus > Fluid Mask opens a proxy of the original in the Fluid Mask interface. Several seconds elapse while the software finds edges between areas of color contrast. When the image opens, it looks like a paint-by-number copy of your original.


Selecting the Drop Area tool, second one down in the right column, I was able to fill the open area of sky on the right. But that leaves all those areas among the trees.


Fluid Mask has an easy procedure to deal with this problem. After filling the Keep areas with green using the Fluid Mask Edit > Auto-Fill command, I drew a polygon around the problem area. This brings up a palette of all the colors in the selected area. I adjusted the Number of colors slider in the palette to show a smaller number of colors, and chose to display the blue values since those were the ones I wanted to eliminate.


By selecting the blues and moving them from the Keep tab to the Delete tab, Fluid Mask removes all of the sky areas between the trees. By clicking on the Camera icon and zooming in, you can see how successful Fluid Mask is.


Everything looks good, so you click on the Process icon at the bottom of the tool palette, and Fluid Mask renders the mask in just a few seconds. Clicking the Output tab shows the final result. You can zoom around in this window and easily clean up little areas to your liking.


When you are happy with the result, you exit Fluid Mask and are back in Photoshop with the masked image. Note again that Fluid Mask does not create a layer mask, it permanently deletes the masked area from the image layer.

Now it’s a simple matter of opening the sky image and adding it to the layer stack. I’ve also added a green layer so you can see what a good job Fluid Mask did in eliminating the sky.

Here’s the image with green:


And the final image with the new sky:


EZ Mask

In this image the problem is removing the background while preserving the detail in the feather boa so that a new background will show through.

After opening an image in Photoshop, the EZ Mask requires you to unlock the image. It’s always a good idea to make a copy of the image before working on it. Apply a layer mask to the image, then click on the layer’s color channels thumbnail to activate it rather than the layer mask thumbnail.

If you plan on compositing this image with another, open the other image and place it on a layer below the image you are masking. EZ Mask will allow you to preview the composite if you do.


Select Filter > Digital Film Tools > EZ Mask and the image opens in the EZ Mask interface.


For this image, there is a lot of fine detail we want to preserve in the feathers when we drop out the background. With this type image, the best way to proceed is to use the Paint Unknown brush (third icon down on the left) to highlight the feathers and edge.


Now you fill the Keep areas using the Paint Bucket with the green Keep color and the Drop areas using the Paint Brush tool with the red Drop color.


When this is done, you can preview the mask by clicking on the third icon from the right in the top row. The diamond icon next to it gives you three different settings for preview quality. The higher the quality, the longer it take EZ Mask to generate the preview. The preview below is done at the highest quality and took 45 seconds on this 36MB file.


If you are satisfied with the mask, click the Done icon next to the Preview icon and EZ Mask exits you back to Photoshop and renders the final layer mask. The rendering took just over 20 minutes for this image.


At times the progress bar and elapsed time indicator stopped completely. Don’t panic. The process is still continuing. If you are patient, your reward is an extremely accurate mask. I have added a red background in this 1:1 view to make it easier to see the quality of the mask.


Click for 2,245x1,796-pixel view.

Each of these programs can handle all the easy masking jobs, like separating a yellow flower from the blue sky. But each also has features that make it capable of handling some masking chores better than other programs handle them. Your choice of program will depend on the type of work you do and the number of times you need to deal with difficult masking problems.