Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 38 ยท Patterns for Text Processing

Splitting and Tokenizing Text

Use regular expressions to split strings, keep delimiters with capture groups, tokenize text into words and numbers, parse CSV lines with quotes, and break paragraphs into sentences.

Splitting cuts a string at the places where a separator matches. Tokenizing is the mirror image: you describe the pieces you want to keep. Choosing the right direction often decides whether a pattern is simple or painful. "Split on commas" is easy. "Split on commas except inside quotes" is awkward as a separator but trivial as a list of pieces.

Splitting with a regex

str.split(regex) cuts at every match. If the regex contains a capture group, the captured text is included in the result between the pieces.

Node.js split with patterns, groups and limits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Two pitfalls. First, empty strings appear when the separator is at the start or end, or when two separators touch, so filter them (.filter(Boolean)) or trim first. Second, a separator that can match the empty string (like (?=[A-Z])) splits between characters, which is useful for camelCase but needs care.

Tokenizing: describe the pieces

When the pieces have a clear shape, match with g or matchAll is more direct than split. A word tokenizer, for example:

Node.js Words, numbers and punctuation as tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Order matters in alternation: the engine takes the first alternative that matches at a position, not the longest. Put more specific alternatives (like numbers with decimals) before general ones.

CSV lines with quotes

A naive line.split(",") breaks on "Smith, John". The pieces-first view fixes this: a field is either a quoted string (with "" as an escaped quote) or a run of non-commas.

Node.js Parsing a CSV line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

This handles one line. Real CSV allows newlines inside quotes, different delimiters and encodings, so for production work use a CSV library; a regex version is useful for quick scripts and for understanding the grammar. The sticky flag is explained in sticky matching.

Sentences and paragraphs

Splitting prose into sentences is deceptively hard because a period can also end an abbreviation (Dr.) or sit inside a number. A reasonable heuristic splits after ., ! or ? followed by whitespace and a capital letter or quote.

Node.js A heuristic sentence splitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

For linguistic accuracy, use Intl.Segmenter with granularity: "sentence". Regexes are a fast approximation.

Choosing split or match

Situation Use
Separator is simple and always the same shape split
You also need the separators split with a capture group, or matchAll
The pieces have a clear shape but separators are messy match / matchAll
Position matters (parser, lexer) Sticky regex loop

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does "a1b2c".split(/(\d)/) return?

    A capture group in the separator pattern puts the captured separators in the result between the pieces.
  2. 2. Why does " x ".split(/\s+/) contain empty strings?

    A match at position 0 or at the end leaves an empty piece before or after it. Trim first or filter the result.
  3. 3. Why is a plain split(",") unsuitable for CSV?

    A comma inside quotes belongs to the field, so the parser has to understand the quotes.
  4. 4. In an alternation like \d+|\d+.\d+, what is the problem?

    Alternation takes the first branch that matches, so list the more specific decimal form first.

Summary

  • split removes the separator; a capture group in the separator keeps it.
  • Leading, trailing and doubled separators create empty strings; trim or filter.
  • When the pieces are easier to describe than the separators, use match or matchAll.
  • Alternation is ordered: put specific alternatives before general ones.
  • CSV, sentences and other real-world formats have edge cases; know when to use a proper parser or Intl.Segmenter.

Next: How Backtracking Works