Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 54 ยท Edges, Contours and Shapes

GrabCut: Foreground Extraction and Background Removal

Cut out an object with cv2.grabCut: start from a rectangle or a mask, understand the four mask values and the GMM model, refine with strokes, and apply the result.

Thresholding works when an object is much brighter or darker than its surroundings, and color masks work when it has a distinctive color. But what about a person standing in front of a busy street, or a flower against leaves? For these jobs you need to cut out an object whose colors overlap with the background. OpenCV's cv2.grabCut does this with very little help from you: you draw a rough rectangle around the object, and the algorithm works out which pixels belong to it.

GrabCut is an interactive segmentation method. It is widely used for background removal, for making sticker-like cutouts and for preparing training masks. In this lesson you will learn what the algorithm does, the exact inputs and mask values of cv2.grabCut, how to start from a rectangle or from a mask, how to refine the result with strokes, and how to apply the final mask. The OpenCV calls are read-only; NumPy models let you see the key ideas run.

How GrabCut works

You do not need the mathematics to use it, but the intuition explains every parameter.

  1. Initialization. You give a rectangle. Everything outside it is treated as definitely background. Everything inside it is labeled "probably foreground".
  2. Color models. The algorithm fits a Gaussian Mixture Model (GMM) to the colors of the background pixels and another to the foreground pixels. A GMM is a flexible description of "what colors appear and how often", usually with five components each. For any pixel, you can now compute how likely it is to be background and how likely to be foreground.
  3. Graph and min-cut. It builds a graph in which every pixel is a node. Each pixel is linked to a "foreground" and a "background" terminal with weights from the GMM likelihoods, and to its neighbors with weights that are strong when the colors are similar. A minimum cut of this graph splits it into two sets with the lowest total cost: pixels that are likely foreground stay together and edges (where colors change) are the cheapest places to cut.
  4. Iterate. The new labels are used to re-estimate the two GMMs, the graph is rebuilt, and the cut is recomputed. A few iterations (usually 5) converge.

So the algorithm combines color statistics with a smoothness preference for clean boundaries. That is why it can handle objects with colors that appear in the background too, as long as the rectangle makes most of the background clearly separate.

The function

Python
cv2.grabCut(img, mask, rect, bgdModel, fgdModel, iterCount, mode)
Argument Meaning
img An 8-bit, 3-channel color image
mask A uint8 array of the same height and width. In rectangle mode it is filled by the function; in mask mode you provide the starting labels.
rect (x, y, w, h) of the box around the object. Used only in rectangle mode.
bgdModel, fgdModel Temporary arrays of shape (1, 65) and type float64 that hold the two GMMs. Create them with np.zeros and do not touch them. Reuse the same ones when you refine.
iterCount Number of iterations, typically 5
mode cv2.GC_INIT_WITH_RECT, cv2.GC_INIT_WITH_MASK or cv2.GC_EVAL

The mask uses four values:

Constant Value Meaning
cv2.GC_BGD 0 Sure background
cv2.GC_FGD 1 Sure foreground
cv2.GC_PR_BGD 2 Probably background
cv2.GC_PR_FGD 3 Probably foreground

After the run, the mask contains these four values for every pixel. To get a binary mask, count the two "foreground" values as foreground.

Start from a rectangle

Python
import cv2
import numpy as np

img = cv2.imread("flower.jpg")
mask = np.zeros(img.shape[:2], np.uint8)
bgd_model = np.zeros((1, 65), np.float64)
fgd_model = np.zeros((1, 65), np.float64)

rect = (50, 40, 400, 300)                       # (x, y, width, height) around the object

cv2.grabCut(img, mask, rect, bgd_model, fgd_model, 5, cv2.GC_INIT_WITH_RECT)

fg = np.where((mask == cv2.GC_FGD) | (mask == cv2.GC_PR_FGD), 255, 0).astype(np.uint8)
cutout = cv2.bitwise_and(img, img, mask=fg)     # background becomes black

Tips for the rectangle: keep it tight but not cutting into the object, because everything outside is treated as certain background. Include the entire object, but not much more. If the image is large, resize it first (a 4-megapixel image can take many seconds); you can scale the mask back up afterwards with INTER_NEAREST.

Refine with a mask

The first result is rarely perfect. Parts of the background are kept, or parts of the object are lost. The fix is to tell the algorithm what it got wrong: mark pixels you are sure about in the mask and run again in mask mode, reusing the models.

Python
# Suppose `fix_fg` and `fix_bg` are binary images where you painted strokes (for example with the mouse).
mask[fix_bg == 255] = cv2.GC_BGD               # sure background
mask[fix_fg == 255] = cv2.GC_FGD               # sure foreground

cv2.grabCut(img, mask, None, bgd_model, fgd_model, 5, cv2.GC_INIT_WITH_MASK)

The strokes can be drawn with the mouse-callback technique from Mouse Events, Trackbars and Interactive Windows: left button paints foreground, right button paints background. A few short strokes in the wrong regions are normally enough. A good workflow is to show the result, let the user paint corrections, rerun, and repeat.

Using the result

The binary mask can be used in several ways:

  • Cutout on black: cv2.bitwise_and(img, img, mask=fg).
  • Transparent PNG: add the mask as an alpha channel: rgba = cv2.cvtColor(img, cv2.COLOR_BGR2BGRA); rgba[:, :, 3] = fg, then cv2.imwrite("cutout.png", rgba).
  • New background: result = np.where(fg[..., None] == 255, img, new_bg), and soften the edge first with a small Gaussian blur of the mask for a natural look.
  • Bounding box or crop: x, y, w, h = cv2.boundingRect(fg).
  • Contours of the object: pass fg to cv2.findContours (Finding and Drawing Contours).

Limits and alternatives

  • Similar colors. If the object and background share colors (a green frog on leaves), the color models cannot separate them, and GrabCut needs many strokes.
  • Speed. It is iterative and graph-based, so it is slow for big images.
  • Thin structures such as hair and fences are hard; the boundary is a hard cut with no partial transparency. Matting methods or neural segmentation models handle these better.
  • Automation. GrabCut needs a rectangle. In a pipeline the rectangle may come from a detector (a face detector, or a YOLO box from the dnn module).

A different classical segmenter is the watershed: you mark one marker region per object (and one for the background) in an integer array, and cv2.watershed(img, markers) floods from the markers like water filling basins, labeling each pixel and writing -1 along the boundaries. It is the standard tool for separating touching objects such as coins, using distance-transform peaks as markers.

Try it: iterate color models like GrabCut

This model keeps the heart of GrabCut and drops the graph cut. A small image has a reddish object on a bluish background. We start with a rectangle (outside = background), fit one Gaussian color model per class, relabel every pixel in the rectangle by which model fits better, and repeat.

Python Two color models, relabelled in a loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loose rectangle starts with 72 pixels labeled foreground, but only 42 really belong to the object. The first iteration moves 29 of the wrong ones to background as soon as the two color models exist, and after three iterations the labels stop changing at exactly the 42 object pixels. Real GrabCut adds the graph-cut step so neighbors tend to agree, which removes isolated misclassified pixels.

Try it: mask values to cutout

This second model shows how to turn the four GrabCut labels into a binary mask, an alpha channel and a composite on a new background, without needing OpenCV.

Python From GrabCut labels to a cutout

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What is real: the grabCut signature, the four mask values, the model arrays, the rectangle and mask modes and the use of the result match OpenCV. What is the stand-in: the first model keeps only the "fit color models and relabel" part of GrabCut with one Gaussian per class and no graph cut; the second is NumPy indexing on tiny arrays.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. When you start GrabCut with a rectangle, how are the pixels outside the rectangle treated?

    Pixels outside the rectangle are labeled GC_BGD and stay background. Inside the rectangle, they start as probably foreground.
  2. 2. What are the four values in a GrabCut mask?

    GC_BGD=0, GC_FGD=1, GC_PR_BGD=2 and GC_PR_FGD=3. To get a binary mask, count 1 and 3 as foreground.
  3. 3. What are bgdModel and fgdModel?

    They hold the GMMs. Create them with np.zeros((1, 65), np.float64) and reuse them when refining.
  4. 4. How do you correct a GrabCut result?

    Strokes in the mask give the algorithm certain labels, and the next run uses them to improve the models and the cut.
  5. 5. Which situation is hardest for GrabCut?

    GrabCut relies on color models, so similar colors blur the difference, and its hard boundary cannot capture semi-transparent hair.

Summary

  • cv2.grabCut separates an object from its background using two color models (GMMs) and a graph min-cut, starting from a rectangle (GC_INIT_WITH_RECT) or a mask (GC_INIT_WITH_MASK).
  • Mask values: 0 sure background, 1 sure foreground, 2 probably background, 3 probably foreground. Binary mask = values 1 and 3.
  • Create bgdModel and fgdModel as np.zeros((1, 65), np.float64), run about 5 iterations, and refine by painting known pixels into the mask.
  • Apply the result with bitwise_and, an alpha channel or a composite on a new background. For touching objects use the watershed with markers instead.

Next: learn how to find and describe distinctive points in Corners and Keypoints, or put segmentation into practice in Project: Color Object Detector.