Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 61 ยท Lexical Analysis

Lexical Errors and Recovery

Handle lexical errors well: invalid characters, unterminated strings and comments, error tokens, caret diagnostics and panic-mode recovery, in Python.

Real programmers make mistakes all the time: a stray # pasted from another language, a forgotten closing quote, a number glued to a word. A compiler that reacts to the first mistake with a one-line crash is annoying. A compiler that reports the problem precisely, shows the offending line and then keeps reading so it can report the next problem in the same run is a pleasure to use. The difference is error handling, and it starts in the lexer.

The lexer we wrote raises an exception at the first problem. In this lesson we change its behavior in three ways. It will record each error and continue, it will emit error tokens so later phases know something was wrong, and it will recover by skipping just enough text that one mistake does not produce a flood of follow-up errors. We will also print diagnostics in the friendly style of modern compilers, with the source line and a caret under the problem.

What can go wrong at the lexical level

A lexer only sees characters, so it can only complain about things that cannot even form a token. The usual suspects:

  • Invalid character: @, #, a stray backtick, or a Unicode symbol the language does not use.
  • Unterminated string: an opening " with no closing quote before the end of the line.
  • Unterminated comment: /* without a matching */.
  • Bad escape sequence: "a\qb", where \q means nothing.
  • Malformed number: 12abc, 0x with no digits, or 1e in languages where that is not allowed.
  • Invalid encoding: bytes that are not valid text. Real compilers must handle this; we skip it here.

Notice what is not on the list: let = = 5 or a missing semicolon. Every token there is valid. Whether the tokens form a sensible program is a question for the parser, covered in Syntax Errors and Error Recovery.

Goals of error handling

A good error handler aims for four things, in this order of importance:

  1. Report the error clearly, with the position and a message that says what went wrong in the user's terms.
  2. Report each mistake once. One bad character should not become ten messages.
  3. Keep going, so that independent mistakes elsewhere in the file are reported in the same run.
  4. Never crash or loop forever, whatever the input. Every recovery step must consume at least one character.

Error tokens

When the lexer finds something it cannot turn into a valid token, it has two options: stop (raise an exception), or produce a special token of kind ERROR and move on. The second option has an important advantage. The error token travels into the token list, so the parser knows exactly where the lexer had trouble. The parser can then suppress its own follow-up complaints about that spot, which avoids a second, confusing message about the same mistake.

Alongside the token, the lexer appends a diagnostic to a separate list: a message, a line, a column and a length (the length lets us underline the whole bad lexeme). After lexing, the compiler checks whether the diagnostic list is empty. If not, it prints the messages and stops before code generation, but only after the parser has had a chance to find more problems.

flowchart LR
    A["Source text"] --> B["Lexer"]
    B --> C["Tokens<br/>(including ERROR tokens)"]
    B --> D["Diagnostics list"]
    C --> E["Parser"]
    D --> F["Printed together,<br/>sorted by position"]
    E --> F

The lexer sends good tokens to the parser and complaints to a separate list; nothing is thrown away.

Panic-mode recovery

Panic mode is the simplest recovery strategy, and the one used by almost all lexers. On an error, the lexer discards characters until it reaches a place where normal scanning can safely resume. The art lies in choosing how much to throw away:

  • Skip one character. Simple, but @@@ produces three errors.
  • Skip a run of invalid characters: all the consecutive characters that cannot start any token. @@@ produces one error.
  • Skip to the next whitespace. Even coarser, and it can swallow good tokens stuck to the bad ones, such as the y in @@@y.

For unterminated strings, a standard choice is to skip to the end of the line, because strings in most languages cannot span lines. For an unterminated block comment, everything up to the end of the file is part of the comment, so recovery is trivial (and the diagnostic should point at the start of the comment).

The next example compares the three strategies on the same text. The bad characters are $$ and @@@.

Python Comparing panic-mode strategies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The middle strategy wins: it reports one error per cluster and keeps every valid token. The first one floods the user with five messages, and the last one silently loses the identifier y. Skipping too little causes cascading errors; skipping too much hides real errors in the part you skipped.

Every strategy above consumes at least one character per error. That is the guarantee that the loop terminates, and it is the first thing to check when you write a recovery routine.

Showing the error: the caret display

A message like error at 3:13 forces the user to open the file and count columns. Compilers such as GCC, Clang and rustc show the line itself with a marker underneath. It takes only a few lines of code:

  1. Print file:line:col: error: message.
  2. Print the source line, with a line-number gutter.
  3. Print spaces up to the column, a ^ under the first bad character, and ~ for the rest of the lexeme.

Everything depends on the line, col and length stored in the diagnostic, which is the reason we track positions so carefully from the very first token.

A lexer that never gives up

Here is a complete recovering lexer for a slice of the toy language. It uses one master regular expression with named groups (the technique from Regular Expressions for Lexers). The trick for errors is to include error patterns in the same expression, after the good ones. An opening quote with no closing quote matches BADSTRING, an unclosed /* matches BADCOMMENT, and anything that matches nothing at all falls into the panic-mode branch. A number directly followed by a letter is caught as a malformed number, so 12abc is not silently lexed as 12 and abc.

Python A recovering lexer with caret diagnostics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four mistakes, four messages, and each one points at the right place. Look at what the recovery decided:

  • 12abc became one error token covering the whole lexeme, not a number followed by an identifier.
  • The unterminated string swallowed the rest of line 2 (the ; included), then lexing resumed cleanly on line 3.
  • The # was skipped alone, and 2 and ; after it were lexed normally.
  • The unterminated comment starts at line 4, column 1, so the caret points at the /*, where the user needs to look. Everything after it, including let ok = 1;, was inside the comment, so no tokens were produced for it.

When you design error messages, describe what the lexer saw and, where possible, what it expected: "unterminated string" is good; "unterminated string, expected a closing quote before the end of the line" is better. Quote the offending text with repr() so invisible characters such as tabs show up.

Limiting the damage

Two safeguards complete the picture:

  • An error limit. After, say, 20 errors, stop and print "too many errors". A file in the wrong language can produce thousands of messages, and nobody reads them.
  • Suppressing cascades downstream. Because error tokens exist, the parser can treat them as "poison" that matches anything, instead of reporting "unexpected token" for something the lexer has already explained.

Some tools go further and offer suggestions: "did you mean ==?" when they see = inside a condition (that one belongs to the parser), or "keyword while, not While". These are polish. Get the basics right first: precise positions, one report per mistake, never loop forever.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which of these is a lexical error, as opposed to a syntax error?

    The lexer fails when characters cannot form any valid token. The other three involve valid tokens in a wrong arrangement or meaning, which are parser or semantic problems.
  2. 2. What is the purpose of an ERROR token in the token stream?

    The ERROR token keeps the position of the problem inside the token list, which lets the parser stay quiet about a spot the lexer has already reported.
  3. 3. What does panic-mode recovery in a lexer do?

    The lexer skips a chosen amount of text (one character, a run of bad characters, or to the end of a line) and then resumes normal scanning.
  4. 4. Why must every recovery step consume at least one character?

    If a recovery step consumed nothing, the same bad position would be examined again and again.
  5. 5. Where should the diagnostic for an unterminated block comment point?

    The opening marker is where the user must look to fix the problem; the end of the file is only where the lexer found out.

Summary

  • Lexical errors are characters that cannot form valid tokens: invalid characters, unterminated strings and comments, bad escapes and malformed numbers.
  • A good lexer records a diagnostic (message, line, column, length), emits an ERROR token and keeps scanning.
  • Panic mode skips just enough text: a run of bad characters, the rest of the line for a string, the rest of the file for a comment.
  • Skipping too little causes cascading messages; skipping too much hides real errors.
  • Caret diagnostics show the source line and underline the bad lexeme, so the user does not have to count columns.
  • Every recovery step must consume input, and an error limit protects users from floods of messages.

Next: Context-Free Grammars