Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 38 ยท Flags and Engines

Sticky Matching and Match Indices

Use the sticky flag and lastIndex to build tokenizers, and the d flag to get exact start and end positions of every capture group. Includes a tiny lexer you can run.

Most regex tasks ask "where is the next match?" Two flags answer different questions. The sticky flag y asks "does a match start exactly here?", which is what a tokenizer needs. The indices flag d asks "where exactly did each group match?", which is what an editor or a highlighter needs.

Both build on lastIndex, the property that makes g and y regexes stateful. Understanding it removes most "why does my loop never end?" bugs.

How lastIndex works

For a regex with g or y, exec() and test() start searching at lastIndex. On success, lastIndex becomes the end of the match. On failure it resets to 0.

Node.js Watching lastIndex move

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Two warnings. First, an empty match does not advance lastIndex, so a loop like while (re.exec(s)) with a pattern that can match nothing (/a*/g) runs forever; matchAll and replace handle this for you, but a hand-written loop must bump lastIndex itself. Second, lastIndex is ignored without g or y.

The sticky flag

A regex without y scans ahead to find a match anywhere at or after lastIndex. With y it must match at lastIndex or fail. It acts like an implicit \G anchor from other engines.

Node.js Global versus sticky

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Note that ^ in a sticky regex still means "start of input" (or line with m), not "start at lastIndex".

Building a tokenizer

A lexer reads source text and cuts it into tokens: numbers, names, operators. With sticky regexes, each rule is tried at the current position; the first one that matches consumes the text, and nothing can be skipped silently. If no rule matches, you know the input is invalid, and you know exactly where.

flowchart TD
  S["pos = 0"] --> T{"pos < length?"}
  T -->|"no"| Done["Done"]
  T -->|"yes"| R["Try each rule with lastIndex = pos"]
  R -->|"a rule matches"| E["Emit token, pos = lastIndex"] --> T
  R -->|"none match"| X["Error at pos"]

A sticky tokenizer never skips characters: every position is consumed by a rule or reported.

Node.js A tiny expression lexer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

You will use the same technique again in the template engine project, where it parses {{ name }} tags.

The d flag: match indices

Normally a match object tells you m.index for the whole match but nothing about each group. With d it gains m.indices: an array of [start, end] pairs (end is exclusive) for the whole match and every group, and indices.groups for named groups.

Node.js Group positions with the d flag

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

A group that did not participate in the match has undefined in indices, just like undefined in the match array.

Practical advice

  • Use matchAll or replace with a callback instead of manual exec loops when possible.
  • Reset lastIndex = 0 before reusing a global or sticky regex, or create it fresh.
  • Use y for tokenizers and parsers; use d when you need character offsets rather than just strings.
  • The d flag costs extra memory and time, so enable it only when you use the positions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the y flag require?

    A sticky regex does not scan ahead; it tries to match only at lastIndex.
  2. 2. What happens to lastIndex when exec fails on a global regex?

    On failure lastIndex is reset to zero, which is why a while loop around exec ends cleanly.
  3. 3. What do the numbers in m.indices[1] represent?

    With the d flag each entry is a [start, end] pair where end is exclusive, so text.slice(start, end) returns the group text.
  4. 4. Why is a sticky regex a good fit for a tokenizer?

    Each rule must consume text starting exactly at the current position, so an unknown character is detected instead of being jumped over.

Summary

  • g and y regexes keep state in lastIndex; exec and test update it, failure resets it.
  • A sticky regex matches only at lastIndex, which makes it a natural lexer building block.
  • Empty matches do not advance lastIndex, so manual loops need care.
  • The d flag adds indices with [start, end] pairs for every group, including named groups.
  • Prefer matchAll and replace over hand-written loops when you can.

Next: Validating Input with Regex