Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Masks and Compositing

Select and Mask, Edge Refinement and Matching Layers

Cut out hair and complex edges with Select and Mask, remove halos, and match color, light and grain so composited layers look like one photograph.

A composite fails for two reasons: the edges look cut out, or the pieces do not match. This lesson covers both: Select and Mask for edges, then the checks you make so the colors, light and grain agree.

The Select and Mask workspace

With a selection tool active (Quick Selection, Object Selection, Lasso, Marquee) choose Select > Select and Mask, or press Ctrl+Alt+R. Photoshop opens a dedicated workspace with:

  • Tools on the left: Quick Selection, Refine Edge Brush (for hair and fur), Brush, Object Selection, Lasso, Hand and Zoom.
  • View Mode in the Properties panel: Onion Skin, Marching Ants, Overlay, On Black, On White, Black & White, On Layers. Switch between them with F (cycle) and look at the edge against a color that makes problems obvious.
  • Edge Detection: Radius sets the width of the band in which Photoshop decides what is subject and what is background. Tick Smart Radius to let it vary around the edge. Use a larger radius around hair, a small one around sharp edges.
  • Global Refinements: Smooth (removes jagged edges), Feather (blurs the edge), Contrast (tightens the edge back up) and Shift Edge (negative values contract the selection to reduce halos).
  • Output Settings: Decontaminate Colors replaces the edge color fringe with color from nearby subject pixels, and Output To lets you choose Selection, Layer Mask, New Layer, New Layer with Layer Mask, and so on.

Recommended workflow for a portrait:

  1. Make a first selection with Select Subject (Select menu) or the Object Selection tool.
  2. Open Select and Mask and set the View to On Black or On White, then compare with the target background color.
  3. Use the Refine Edge Brush on hair so Photoshop widens its analysis there.
  4. Set Shift Edge to about -10 to -30% to remove the old background's halo.
  5. Tick Decontaminate Colors and output to New Layer with Layer Mask, keeping the original untouched.

Always output to a layer mask rather than a plain selection, so you can finish the edge by painting.

Python Stand-in: feathering a hard mask edge

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A feather turns the single step into a ramp. Contrast then pulls the ramp back toward a steeper edge, which is why the two sliders are often used together.

Fixing fringes and halos

A light halo around a subject usually means pixels from the old background remain at the edge. Remedies, from simplest:

  • Reduce Shift Edge in Select and Mask.
  • Use Layer > Matting > Defringe (set 1 or 2 px) or Remove White Matte / Remove Black Matte.
  • Paint on the mask edge with black using a small brush.
  • Add a clipped adjustment layer (Hue/Saturation) with a brush sampling the edge color.
  • Use Alt+click on the mask thumbnail to check the mask against black and white.

Matching a subject to a new background

Even a perfect cut-out looks fake if it does not belong in the scene. Check these in order.

1. Direction and softness of light

Study the shadows in the background. Which way do they fall, and how hard are they? The subject needs the same light direction. Flip it horizontally (Edit > Transform > Flip Horizontal) if the light comes from the opposite side, and paint or add a shadow on a Multiply layer with the same angle and softness.

2. Color and contrast

Add a clipped Curves or Levels layer to the subject and match the black point and white point with the background. A subject lit by warm light needs warm highlights.

Image > Adjustments > Match Color can transfer the color cast from a source to a target. It works on the target layer; pick the source image and layer, adjust Luminance and Color Intensity, and tick Neutralize if necessary. It is quick, but use it at partial strength because it can crush detail.

Python Stand-in: match mean and spread (the idea behind Match Color)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a simplified one-channel version of "make this layer's brightness statistics resemble that one's". Photoshop's Match Color works in three channels with more care.

3. Edge light and color spill

A subject in front of a blue sky picks up blue edges. Create a new layer clipped to the subject, set it to Color or Soft Light, and paint the sky color along the edges with a soft low-opacity brush.

4. Perspective, scale and sharpness

Match the horizon line and camera height. A subject photographed from above looks wrong on a ground-level background. Keep sharpness consistent: blur a subject that should sit in the distance (Filter > Blur > Gaussian Blur as a Smart Filter) or sharpen one that looks soft.

5. Grain and noise

Cameras add noise. A clean cutout over a noisy photo shows its seams. Add noise to the subject with Filter > Noise > Add Noise (Gaussian, Monochromatic, about 2-4%) as a Smart Filter, or add a noise layer over the whole composite at the end so everything shares the same grain.

Final unifying touches

A few steps that make layers feel photographed together:

  • A Color Lookup or Gradient Map adjustment layer at low opacity over everything.
  • A soft vignette with a Curves layer and a mask.
  • A small amount of blur or chromatic aberration (Filter > Lens Correction) across the entire stack.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which Select and Mask setting removes a light halo by shrinking the selection edge?

    A negative Shift Edge contracts the edge so leftover background pixels fall outside the mask.
  2. 2. Why output Select and Mask to a layer mask rather than a selection?

    A layer mask preserves your work and can be adjusted later; a selection disappears when you deselect.
  3. 3. What does Decontaminate Colors do?

    It removes the color spill from the old background that clings to the edge pixels.
  4. 4. A cut-out subject looks too clean over a grainy photo. What helps?

    Matching grain is one of the quickest ways to make a composite believable.

Summary

  • Use Select and Mask with Smart Radius, Refine Edge Brush, Shift Edge and Decontaminate Colors, and output to a layer mask.
  • Remove halos with Shift Edge, Defringe and careful painting.
  • Match light direction, color, edge light, perspective, sharpness and grain so the composite reads as one photo.
  • Finish with unifying adjustments across the whole stack.

Next: Actions: Recording and Replaying Steps