Professional Photographer Magazine Web Exclusives

Essential Selection: Excerpted from "Adobe Photoshop Masking & Compositing"

Explore one of the most powerful tools in Photoshop for making and perfecting accurate selections. 

Excerpted from “Adobe Photoshop Masking & Compositing,” Second Edition, by Katrin Eismann, Seán Duggan, and James Porto. Copyright © 2013. Used with permission of Pearson Education, Inc. and New Riders.

Working with Adobe Photoshop can be a lot like a daily commute, and it can seem like you're in a rut and going over the same territory. When making selections, most people simply grab one of the familiar selection tools from the toolbar and hope a quick drag or click will get the job done. To achieve professional results, relying on the standard selection tools may create disappointing results.

Making selections in Photoshop is such a fundamental part of working in the program that an entire menu is devoted to them, the Select menu. Here we'll take a closer look at the amazing power of the Refine Edge dialog.

REFINE EDGE: In the Options bar for the Marquee and Lasso selection tools is a setting for feathering the selection. Feathering creates a softer edge with a more gradual transition between the selected and nonselected areas. The main problem with choosing a Feather setting in the Options bar is that you cannot see the result and must guess at what number might be appropriate. Fortunately, there is a better way to apply edge feathering, as well as other modifications, to a selection and that is to use the Refine Edge dialog.

Refine Edge can be accessed either in the Select menu or via a button in the Options bar when a selection tool is active. In addition to feathering, the Refine Edge dialog includes a number of other very useful controls for modifying the edges of a selection. This section will primarily be a detailed exploration of the possibilities offered by the Refine Edge dialog, not a strict step-by-step exercise. To properly cover all of the options in Refine Edge, however, we need to start with a basic selection so we have a selection edge to modify; for that we'll use the photo of the curious dog (Figure 1).


Figure 1: This image resource is available to download at A great majority of the files used throughout the book are available for download at the resource link referenced in the book’s Introduction.  ©Seán Duggan

1. Choose the Quick Selection tool with the Auto-Enhance option selected, and set the brush size to 100 pixels. Start the selection by dragging diagonally down from the top of the dog's left ear. Next, drag down from the right ear to complete the selection of the dog's head. Continue dragging over the dog's body until the selection is expanded to cover the entire dog. A few drags with the Quick Selection tool should do it.

2. Zoom in to make sure that you are not missing any areas, such as by the ring on the dog's collar or the bottom edges of the front feet (Figure 2). If you see areas that should be selected but are not, just click on them with the Quick Selection tool (for fine work, make the brush size smaller by tapping on the left bracket key).


 201301we_Fig 7-32b.jpg


Figure 2: Selecting the dog with the Quick Selection tool and fine-tuning the selection around the dog's collar and feet.

3. With the dog selected, click the Refine Edge button in the Options bar or choose Select > Refine Edge.

View Mode

The Refine Edge dialog is divided into four sections: View Mode, Edge Detection, Adjust Edge, and Output. Click the small disclosure triangle next to the View menu to see the different ways to view the selection in the main document window. Each view has a single-letter keyboard shortcut. Knowing these shortcuts makes switching back and forth between the different views much easier than constantly returning to the menu. Some are more useful than others:

● Marching Ants (M) is the least useful, because it just looks like an ordinary selection, and the marching ants cannot show the subtle edge refinements you can make.

● Overlay (V) view is like seeing a selection in Quick Mask mode. A transparent color overlay (the default is red) covers the areas that are not selected. Option-click (Alt-click) on this view mode to access the Quick Mask options to change the color of the overlay.

● On Black (B) is the same as the Black Matte view in the Color Range dialog. The selected area is placed on a black background. This view is useful if you’ll be placing the selected object into another image and need to see how the edges blend against a darker background.

● On White (W) is the same as the White Matte view in the Color Range dialog. The selected area is placed on a white background. This view is useful if you’ll be placing the selected object into another image and need to see how the edges blend against a lighter background.

 Black & White (K) view shows the selection as a black-and-white mask, like the grayscale view in the Color Range dialog. In this view you can really see all the subtleties of the edge modifications (Figure 3).

● On Layers (L) view shows the selected area surrounded by transparency, as if it were a separate layer with a layer mask. If there are layers underneath the image, you’ll see how well the selection modifications are working in the entire composite.

● Reveal Layer (R) shows the entire layer with no masking applied.


Figure 3: Overlay, On White, On Black, and Black & White mask view.

For making a selection, the views we use most are Overlay (V), which shows us the Quick Mask view of the selection, and Black & White (K), which shows the selection as a mask channel. If the image you’re working on is already part of a composite and additional image layers are below it, the On Layers (L) view is also quite handy. The usefulness of the On Black (B) or On White (W) matte view really depends on how you’ll be using the selected element in another photo. You’ll also see the Show Radius and Show Original check boxes in the Refine Edge dialog. We’ll discuss these in the following section when we take a closer look at Edge Detection.


Edge Detection

Edge Detection is truly the heart of the selection mojo offered by the Refine Edge dialog. In fact, for some selections, many of the edge adjustments you make can be handled entirely by the Radius slider and its accompanying Smart Radius check box (Figure 4):


Figure 4: The Edge Detection controls.

● Radius slider. This determines the size of the selection border in which edge refinement occurs. Essentially, it tells Photoshop how far out from the current selection edge to look for additional areas that may need to be added to the selection. Low radius settings are best for objects with hard and sharply defined edges, whereas a larger radius setting works best for elements that have softer or more irregular edges.

● Smart Radius. Selecting this option will adjust the radius for the different characteristics found in a selection that has both hard and soft edges. It is less effective if the edge that you are selecting is uniformly hard or soft, or if you want more control by using the Radius slider and the Refinement Brushes, which are found under the Brush tool icon to the left of this section of the dialog.

● Show Radius (J). This check box is located in the View Mode section, but because it pertains to the Radius slider, we’ll discuss it here. As mentioned earlier, the Radius slider tells Photoshop how far out from a selection edge to analyze the image as it refines the edge. Turning on this option shows you the size of the radius in relation to the image. The actual view of the radius will depend on which overall View Mode you are using at the time. The keyboard shortcut J makes no alphabetical sense, but such is Photoshop life!

The Black & White view (K) is ideal for evaluating the quality of a selection edge and how it is affected by the different controls. When the Radius slider is set to 10, the edges of the dog, including very subtle details on his coat, are revealed with more accuracy (Figure 5).


Figure 5: Radius at 0 (left). When Radius is set to 10, the fine edges of the dog’s coat are rendered with more accuracy.

With the Show Radius check box selected, you’ll see how large the radius setting is in relation to the actual edge. You might want to check out Show Radius using the Overlay (V), On Black (B), and On White (W) views, just to get a sense of which are most useful. Show Radius is generally an option you will use for a quick check on the size of the radius, not an option you will have turned on when making fine adjustments.


In our quest for a more perfect edge, let’s return to the Black & White (K) view, increase the radius to 20, and turn on Smart Radius. As the radius setting was increased, more pixels surrounding the dog were added to the selection, although at very low opacities. Throughout this process we would occasionally view the image in Overlay, On Black, and On White to see how well the edge of the dog was being defined. For the final adjustment in this section, let’s lower the Radius to 8 with Smart Radius turned on (Figure 6).


Figure 6: Three views of the Edge Detection adjustments. Radius is set to 8 and Smart Radius is on.


Refine Radius tools

To the left of the Radius slider and Smart Radius check box is a Brush tool icon. Click and hold on this icon to display the menu showing the Refine Radius tool and the Erase Refinements tool (Figure 7). Both tools can be used to help refine the edges of a selection by telling Photoshop where it should look for tricky edge transitions. These tools are useful if there were slight errors in the selection when you first entered the Refine Edge dialog. If you are already used to editing layer masks using the Brush tool, these tools can take a bit of getting used to because they do not behave in the same way.

201301we_Fig 7-43.jpg

Figure 7: The Refine Radius and Erase Refinements tools.

Refine Radius tool. The Refine Radius Tool, or as Katrin calls it “the Look Again Brush,” analyzes the area you brushed over and tries to determine where the edge should be. It does an excellent job of creating partially selected pixels (gray areas) where there are subtle edge transitions but is not as well suited for creating more distinct edges of fully selected pixels. If larger areas need to be added to the selection, it is probably best to exit the Refine Edge dialog and add them using one of the selection tools, and then reenter Refine Edge to continue your edge enhancements.

Erase Refinements tool. This tool removes any refinements created with the Refine Radius tool and restores the edge to what is was before you used either tool. It does not work like an eraser or as a regular brush when painting with the opposite color (a common task in mask editing); it merely restores the edge to what it was before the Refine Radius tool was used.

TIP: To change the size of either of the radius refinement brushes, you can use the Brush Size slider in the Options bar, or tap the left or right bracket keys to make the brush cursor smaller or larger.

To show how these tools work, we created a selection with a small edge error on the side of the dog’s head (Figure 8). Brushing over it with the Refine Radius tool did a pretty good job of modifying the edge to more accurately reflect the actual side of the dog’s head (Figure 9), although a few areas that should have been black were rendered as dark gray. If the edge refinements are output to a new layer with a layer mask, this can easily be fixed with further mask edits (the output options for this dialog are discussed a bit later in the “Output” section). Brushing over the newly refined edge with the Erase Refinements tool restores the original, inaccurate edge (Figure 10).


Figure 8A selection “error” viewed in Overlay and Black & White views.


Figure 9The Refine Radius tool does a pretty good job of fixing the selection error and creating a more accurate edge.


Figure 10: The Erase Refinements tool restores the original edge.

Once you’ve found a Radius amount that works well, carefully inspect the edges to check for any areas where the Refine Radius tool might help. You can hold down the spacebar and press and drag to scroll through the image. On some areas, this tool will work well, such as a small triangular area between the dog’s front and rear legs on the left side of the image (Figure 11). On other areas, you may need to edit the mask using other methods outside of the Refine Edge dialog.


Figure 11: The Refine Radius tool is used to fix a small error on the dog's legs. 


Adjust Edge

In comparison to the Edge Detection controls, the Adjust Edge sliders are not nearly as refined or delicate. These enable you to modify the characteristics of the selection edge, but they do not analyze the actual selected pixels like the Edge Detection sliders do. Still, for some types of selections, these more generalized edge adjustments may be just what you need:

Smooth. Evens out jagged, irregular areas to create a smoother edge (Figure 12).

Feather. Softens the edges of a selection with a uniform blur. Although feathering the edges of a selection is a common task that can help to soften hard, abrupt edges, for more refined edge adjustments, the Feather slider is best used in conjunction with the Radius control, Smart Radius, and the Shift Edge slider (Figure 13).

Contrast. Increases the contrast of a selection edge, making softer areas appear harder and more abrupt. This essentially has the opposite effect of the Feather slider (Figure 14).


Figure 12: The Smooth slider smoothes a selection edge and removes rough, jagged spots.


Figure 13: The Feather slider softens and blurs a selection edge.


Figure 14: The Contrast slider increases contrast or hardness on a selection edge.

Shift Edge. Shifts the entire edge of the selection inward or outward, which makes it a very useful control. Negative numbers will contract a selection, making it smaller. Positive numbers will expand the edge and make the selection bigger. This control works best with softer-edged selection borders and can be useful in removing unwanted background colors from the edges of a selection (Figure 15).


Figure 15: The Shift Edge slider moves a selection edge in or out, contracting the edge by small amounts. 



The Output section determines how and where the edge adjustments will be applied. It also offers a very useful feature for decontaminating edge colors to create a better selection, which will, in turn, make your life easier when you add the selected element into another image.

Decontaminate Colors. The Decontaminate Colors option replaces color fringes and color spill along the edge with the color of nearby pixels that are fully selected. The amount of the color replacement along the fringes is proportional to the softness of the selection edges. Due to the way this feature works, it cannot be applied to a selection or to a layer mask on the original layer; a new layer must be created. The default Output option when Decontaminate Colors is selected is to create a new layer with a layer mask. When this feature is turned on, the Amount slider becomes active. Depending on the selection you’re working with, it may be difficult to see the results.

Zoom in for a closer view and try viewing the image on either black or white, depending on the color of the fringe. Decontaminate Colors often works well in conjunction with the Shift Edge slider as well as the Radius slider in the Edge Detection section. When the dog image is viewed against the white matte, you can see some dark fringes in certain areas, such as the top of his head. With Decontaminate Colors turned on, the Amount set to 100, and the Radius slider set to 0.3 (Figure 16), there is noticeable lessening of the dark fringe.


Figure 16: Decontaminate Colors is off (left); Decontaminate Colors is turned on, Amount = 100, and the Radius is set to 0.3 (right).


Output Options

The default and the first option available in the Output To menu is to apply the changes and return you to a selection in the main image. But there are several other ways you can choose to apply the edge refinements:

Layer Mask. Creates a layer mask for the active layer, hiding the nonselected areas. If your file has a standard background layer, this option will turn the background into a layer with a layer mask. This is very useful for compositing because it is nondestructive. Of course, in and of itself a selection is also nondestructive but not if you use the selection to copy and paste an object into a new image. In that case nonselected pixels would be deleted. If your selection was perfect, that is not a problem. But if you still need to refine the edges a bit more, using the Layer Mask option is the better and highly recommended way to go.

New Layer. Makes a copy of the selected area, deletes the nonselected pixels, and places the selected element on a new layer surrounded by transparency.

New Layer with Layer Mask. Copies the layer to a new layer, and a layer mask is used to hide the nonselected area. No pixels are deleted.

New Document. Creates a new file with the nonselected pixels deleted and the selected element on a layer surrounded by transparency.

New Document with Layer Mask. Creates a new file with the nonselected pixels hidden by a layer mask. No pixels are deleted.  


Remember Settings

The final option in the Refine Edge dialog is the Remember Settings check box. As its name suggests, it will remember the current settings and use them the next time you work with this dialog. The Remember Settings also apply to the Refine Mask dialog, which is nearly identical to Refine Edge but is used for modifying the edges of layer masks. Setting the Refine Edge (or Refine Mask) dialog to remember a specific configuration of settings can be very useful if you are working on a compositing project where you need to select the same, or very similar, objects in different images. For example, you might be using several photos of the dog to create illustrations for a children’s book or a humorous calendar on the secret life of dogs. Being able to quickly recall the settings that worked well on one image of the dog is a big time-saver, particularly if you have photographed the dog with consistent lighting and with the same background.


Refining the Dog Selection

It’s time to finish the selection of the dog. Try to refine the edges so they are as accurate as possible. Because you’ll be outputting these refinements to a layer mask, you’ll have additional opportunities later to apply further improvements.

1. Using your newfound knowledge of how the controls in the Refine Edge dialog work, make adjustments that will result in an accurate edge for the selection of the dog.

2. You can also use our settings: Set the Radius to 8 and select the Smart Radius check box. Select the Decontaminate Colors check box and set its Amount slider to 70. With Decontaminate Colors turned on, the Output menu should be set to New Layer with Layer Mask. Click OK to apply the changes.

3. Now you have a new layer of the dog with the nonselected areas hidden by a layer mask. Save this file in the Photoshop format so that you can use it again. Make sure that the Layers check box is selected in the Save As dialog.

TRY IT How might you use the masked dog photo in a composite image? When deciding on another photo to add the dog to, what would you need to consider to create a convincing scene (even if that scene was obviously a surreal one)? See Figure 17 for some clues.


Figure 17: Guard dog. ©Seán Duggan

At the heart of making accurate selections are tools that give you a good “starter selection” and features that let you apply more precise fine-tuning with visual feedback so you can judge the effectiveness of your edits. The Color Range command, the Refine Edge dialog, and its companion for editing layer masks, the Refine Mask command are three of the most powerful tools in Photoshop for making and perfecting accurate selections. Spending time to get to know them and know them well is time well spent!