Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 61 ยท Lexical Analysis

Writing a Flex Program (Lex Scanner Generator)

Learn the structure of a Flex (Lex) program: definitions, rules and user code, longest match, start conditions, and run a Python stand-in of the same rules.

In Writing a Lexer you built a scanner by hand. In practice many compilers, from small tools to large language front ends, generate theirs. Lex was the original Unix scanner generator, and Flex ("fast lexical analyzer generator") is its modern, widely used replacement. You give it a list of patterns with actions, and it writes a C file containing a function yylex() that finds the next token. Under the hood it does what you learned in regex to NFA to DFA: it combines all your patterns into one automaton.

This lesson explains the layout of a Flex file, the rules that decide which pattern wins, how to build and run a scanner, and how start conditions handle things like comments. Flex itself is a command-line tool that cannot run on this site, so the Flex source is shown read-only, and the runnable examples use a small Python stand-in that follows the same matching rules.

The three sections of a Flex file

A Flex file (by convention ending in .l) has three parts separated by lines that contain only %%:

TEXT
definitions
%%
rules
%%
user code

Here is a complete scanner for a tiny C-like language:

TEXT
%{
/* Section 1: definitions. Code between %{ and %} is copied to the output. */
#include <stdio.h>
int count_ids = 0, count_nums = 0;
%}

%option noyywrap yylineno

DIGIT    [0-9]
LETTER   [A-Za-z_]
ID       {LETTER}({LETTER}|{DIGIT})*

%%
[ \t\r\n]+                  ;                      /* skip white space */
"//".*                      ;                      /* line comment */
"int"|"if"|"else"|"while"   printf("KEYWORD(%s)\n", yytext);
{ID}                        { count_ids++;  printf("ID(%s)\n", yytext); }
{DIGIT}+("."{DIGIT}+)?      { count_nums++; printf("NUM(%s)\n", yytext); }
"=="|"<="|">="|"!="         printf("OP(%s)\n", yytext);
[-+*/=<>]                   printf("OP(%s)\n", yytext);
[;(){},]                    printf("PUNCT(%s)\n", yytext);
.                           printf("error: bad character '%s' on line %d\n", yytext, yylineno);
%%

int main(void) {
    yylex();
    printf("%d identifiers, %d numbers\n", count_ids, count_nums);
    return 0;
}

Definitions. Between %{ and %} you write plain C that Flex copies unchanged (headers, counters). Below that you give names to patterns, such as DIGIT or ID, and reuse them in braces: +. Lines starting with %option change Flex's behavior; noyywrap says "there is only one input file, stop at its end" (otherwise you must supply a function yywrap), and yylineno makes Flex keep a line counter for you.

Rules. Each rule is a pattern followed by an action, written as C code. When the pattern matches, the matched text is available in yytext (a string) and its length in yyleng. An action of just ; does nothing, which is how you skip white space and comments. An action that executes return TOKEN_NUMBER; makes yylex() give that token code to its caller (a parser, usually) and resume from there on the next call.

User code. Everything after the second %% is copied verbatim. A standalone scanner puts main here. When a parser such as Bison calls the scanner, this section usually stays empty.

Pattern syntax in Flex

Flex patterns are close to the regular expressions for lexers you already know, with a few details of their own:

Pattern Meaning
abc the literal characters (letters and digits stand for themselves)
"if" or \+ quotes or a backslash make characters literal; use them for operators
[a-z_], [^"] a character class, or its complement
. any one character except newline
x*, x+, x? zero or more, one or more, optional
x{2,4} between 2 and 4 repetitions of x
x|y x or y
(x) grouping
a named definition from section 1
x/y x, but only if followed by y (the y is not part of the lexeme)
^x, x$ x at the start or the end of a line
<<EOF>> the end of the input

There are no back-references and no lazy quantifiers, because a generator needs patterns that translate to a finite automaton.

How Flex chooses a rule

When more than one pattern could match at the current position, Flex follows two rules, which together make scanners predictable:

  1. Longest match. It takes the pattern that matches the most characters.
  2. First rule on a tie. If two patterns match the same number of characters, the one listed first in the file wins.

The first rule is why iffy is an identifier (the ID pattern matches four characters, the keyword only two) and why <= is one operator rather than < followed by =. The second rule is why keywords must be listed before the identifier rule: for the text if, both match two characters, and the earlier rule wins. If a character matches no rule, Flex's built-in default rule echoes it to the output, which is why a catch-all . rule is good practice (it lets you report the error).

A Python stand-in you can run

The class below behaves like the scanner Flex generates from rules: it tries every rule in the current start condition, takes the longest match, and breaks ties by listing order. Rules are registered with a decorator, so the layout looks like a .l file. It also keeps the line number the way %option yylineno does. (It uses Python's re for each individual pattern, which is fine here because each pattern on its own is simple.)

Python A mini Flex: longest match, first rule wins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at three details in the output. First, iffy is an ID while if is a KEYWORD: both patterns match if, but the keyword rule is earlier, and for iffy the identifier rule matches longer. Second, <= is a single OP. Third, the comment spanning two lines produced no tokens but still advanced the line counter, so if is reported on line 4. The unexpected @ hit the catch-all rule and was reported as an ERROR, while scanning continued.

What happens if you swap two rules

Because ties go to the earlier rule, the order of the rules is part of the program. The snippet below runs the same input with the keyword rule first and with the identifier rule first.

Python Rule order matters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the identifier rule first, even if, while and else become identifiers, because every tie goes to the earlier rule. This is one of the most common beginner mistakes in Flex files, and the fix is simply to move the keyword rule up.

Start conditions

Some constructs need a scanner to behave differently depending on where it is, such as inside a /* ... */ comment or a string literal. Flex calls this a start condition. You declare one in the definitions section and switch with BEGIN:

TEXT
%x COMMENT

%%
"/*"              BEGIN(COMMENT);
<COMMENT>"*/"     BEGIN(INITIAL);
<COMMENT>[^*\n]+  ;                  /* eat the comment body */
<COMMENT>"*"      ;
<COMMENT>\n       ;                  /* yylineno still counts it */
<COMMENT><<EOF>>  { fprintf(stderr, "unterminated comment\n"); return 0; }

%x declares an exclusive condition: while it is active, only the rules prefixed with <COMMENT> are considered. (With %s, a non-exclusive condition, the unprefixed rules stay active too.) The Python stand-in above implements exactly this with its states argument, which is how its comment handling works. The <<EOF>> rule is how you catch a comment that is never closed, which a plain pattern cannot see.

Building, running, and connecting to a parser

Flex is a free program available in the package managers of most systems (for example sudo apt-get install flex on Debian and Ubuntu, brew install flex on macOS, or inside MSYS2/WSL on Windows). Because it generates plain C, the scanner it writes is portable: the same lex.yy.c compiles with any C compiler on any platform. On a machine where Flex is installed, the workflow has three steps:

TEXT
flex scanner.l                 # writes lex.yy.c
cc lex.yy.c -o scanner         # compile (add -lfl if you did not use %option noyywrap)
./scanner < program.txt        # run on a file

When the scanner feeds a parser, the actions stop printing and start returning token codes. The parser, generated by Bison or Yacc as described in parser generators, calls yylex() each time it needs a token. The scanner can pass the value of a token (the number, the name) through a global variable called yylval. The skeleton looks like this:

TEXT
{DIGIT}+     { yylval = atoi(yytext); return NUMBER; }
"+"          { return PLUS; }
{ID}         { yylval_name = strdup(yytext); return IDENT; }

Keep in mind how this differs from the hand-written lexer: a generated scanner is fast and compact, and easy to change (edit one line), but error recovery, nested comments and unusual rules sometimes need extra code or start conditions. Many production compilers still hand-write their scanner for exactly that control.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What are the three sections of a Flex file, in order?

    Definitions come first, then the rules, then user code, separated by lines containing only %%.
  2. 2. The input is 'if' and the file lists the pattern 'if' (keyword) before [a-z]+ (identifier). Which rule fires?

    Flex takes the longest match, and on a tie it prefers the pattern that appears earlier in the file.
  3. 3. The input is 'iffy'. Which rule fires if the keyword rule 'if' is listed first?

    Longest match is applied before the file order is consulted. Order only breaks ties.
  4. 4. What does an exclusive start condition (declared with %x) do?

    In an exclusive condition such as COMMENT, rules without the <COMMENT> prefix are inactive. That is why it is ideal for comments and strings.
  5. 5. What does yytext hold inside an action?

    yytext is the lexeme just matched, and yyleng is its length.

Summary

  • A Flex file has definitions, rules (a pattern plus a C action) and user code, separated by %%; Flex generates a C function yylex().
  • Matching follows two rules: the longest match wins, and on a tie the earlier rule wins. Put keywords before the identifier rule.
  • yytext, yyleng and yylineno give the lexeme, its length and the current line; a catch-all . rule is the place to report bad characters.
  • Start conditions (%x, BEGIN) let one scanner switch behavior for comments, strings and similar regions; <<EOF>> rules catch unterminated ones.
  • Build with flex scanner.l then a C compiler; when connected to a parser, actions return token codes and pass values through yylval.

Next: Lexer Errors and Recovery