Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 38 ยท Projects and Practice

Project: A Markdown-to-HTML Converter with AWK and sed

Write a small Markdown-to-HTML converter: headings, paragraphs, lists, code blocks, bold, italics, links and HTML escaping, using a line-by-line AWK state machine plus sed.

Static site generators, README previews and wikis all turn Markdown into HTML. A complete Markdown parser is a big program, but a useful subset fits in about 80 lines of AWK. Building it teaches the most important AWK technique for structured text: the state machine. The program reads line by line, remembers what kind of block it is inside (paragraph, list, code block), and decides what to print when the block starts or ends.

This is a learning project, not a replacement for a real Markdown library. It supports: headings (# to ###), paragraphs, bullet lists, fenced code blocks, **bold**, *italic*, `code`, and [text](url) links, and it escapes <, > and &.

The state machine

stateDiagram-v2
  [*] --> Idle
  Idle --> Paragraph: text line
  Paragraph --> Idle: blank line
  Idle --> List: "- item"
  List --> Idle: other line
  Idle --> Code: fence line
  Code --> Idle: fence line

Which block we are in decides how the next line is handled and which closing tag is needed.

Three small variables hold the state: para (are we in an open <p>?), list (in a <ul>?) and code (in a fenced block?). Rather than a separate state variable, the program uses a helper close_blocks() that ends whichever block is open before starting another.

Step 1: escaping and inline formatting

AWK has no regex back-references in sub, so inline spans are converted with a match loop: find the span, cut out the inner text, rebuild the string with tags around it.

Bash Step 1: inline formatting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

Two details worth studying. In escape, the replacement "\\&amp;" needs the double backslash, because a single & in a replacement means "the matched text". And span receives its regex as a string ("\\*[^*]+\\*", backslashes doubled): a bare /regex/ passed as an argument would be evaluated as $0 ~ /regex/ and hand the function a 0 or 1. The links function builds its result piece by piece in a local variable.

Step 2: blocks

Now the main rules. Notice that fenced code is handled first, because inside a code block nothing else may be interpreted.

Bash Step 2: the complete converter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The output is HTML you could save as a file and open in a browser. Look at the three small decisions that make it work:

  1. Paragraph lines are joined. A paragraph's <p> is opened with printf (no newline), each following line adds a space and its text, and a blank line or a block change closes it with </p>.
  2. Every block start closes the previous block. close_blocks() is called before headings, code fences and blank lines, so tags never overlap.
  3. END closes whatever is still open. Without it, a file that does not end in a blank line would leave <p> unclosed.

Step 3: wrap it in a page with sed

A full HTML page needs a header and footer around the body. Instead of editing the AWK code, wrap the output in a pipeline. sed can inject the pieces, and cat can supply the boilerplate.

Bash Step 3: a complete HTML page

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with read or cat), it gets the next line from this box.

The title comes from the first heading: sed -n 's/^# *//p;T;q' prints the first line that starts with # (with the marker removed) and quits (T branches to the end of the script when no substitution happened, so non-matching lines are skipped; q ends after the first match). This is a GNU feature; with a BSD sed use awk '/^# /{print substr($0,3); exit}' instead.

Known limits (and ideas)

The converter is intentionally small. It does not handle nested lists, ordered lists, block quotes, tables, images, hard line breaks, or underscores for emphasis. Inline code spans containing * get emphasis applied inside them. Ideas for extending it:

  • Add 1. item lists with a second state variable (olist) and <ol>.
  • Support > quote lines with <blockquote>.
  • Add ![alt](src) images before links, with the same match loop.
  • Generate an automatic table of contents: collect headings in an array during the main pass and print the list in BEGIN-time by processing the file twice (awk -f toc.awk file file) using the NR == FNR idiom.
  • Write a test file with one example per feature and compare against expected output with diff.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is the fenced-code rule placed before the heading and list rules?

    AWK runs the rules in order, and next in the code rule stops the others from touching code lines.
  2. 2. Why does the escape function use "\&" as the replacement?

    In sub/gsub replacements, & means the matched text. Writing \& yields a literal ampersand.
  3. 3. What is the purpose of END { close_blocks() }?

    A file that does not end with a blank line would otherwise leave an unclosed <p> or <ul>.
  4. 4. What is a state machine in this program?

    The state variables remember the current block so the program knows which tags to close and open.

Summary

  • A line-by-line state machine (para, list, code) handles block structure.
  • Inline spans are converted with a match loop because AWK's sub has no back-references.
  • Escape &, < and > first, using \\& for a literal ampersand in the replacement.
  • next makes rules exclusive; close_blocks() and an END block keep tags balanced.
  • Wrap the output with a header and footer in a shell pipeline; extend the converter one feature at a time.

Next: Project: A Template Engine with AWK and sed