Skip to content
Tutorials Center
My learning

Topics

Lesson 140 of 147 ยท Projects

Project: A Small Markdown-to-HTML Converter

Build a Markdown-to-HTML converter in Go in four stages: blocks, lists and code, inline formatting with regexp, then table tests and safe links.

Every page of this website was written in Markdown and turned into HTML by a program. Markdown looks simple, but converting it exposes a whole list of things programmers need to get right: reading text line by line, deciding what each line means, escaping characters so that user text can never become markup, and testing with many small cases. That makes it a very good Go project: it uses strings, regexp, slices and a table of tests, and nothing else.

You will build a converter for a useful subset of Markdown in four stages, and each stage is a complete program. Stage 1 handles headings, paragraphs and horizontal rules. Stage 2 reads the document into a list of blocks (lists, quotes, fenced code) and renders them. Stage 3 adds inline formatting: code spans, bold, italic, links and images. Stage 4 hardens it with a table of tests and a check that refuses dangerous links.

This is a teaching converter, not a full CommonMark implementation. It supports ATX headings (#), paragraphs, horizontal rules, bulleted and numbered lists, block quotes, fenced code blocks, `code`, **bold**, *italic*, links and images. Real projects should use a library such as goldmark. Building your own is the best way to understand what such a library does. Each program converts a sample document that is written in the program, so you can edit the lines and press Run.

The plan

Markdown converters do two separate jobs, and keeping them separate is the main design decision:

  1. Parse blocks. Look at the lines of the document and group them into blocks: a heading, a paragraph, a list, a quote, a code block. This is line-based and uses strings and a few small regular expressions.
  2. Render blocks. Turn each block into HTML. Inside the text of a block, apply the inline rules (bold, links, and so on).

There is one rule that matters more than all the others: escape first, or make sure nothing unescaped can reach the output. Text such as a < b or <script> written by a user must come out as a &lt; b and &lt;script&gt;. The html.EscapeString function does that, and every stage uses it for every piece of text.

Stage 1: headings, paragraphs and rules

Start with the simplest useful converter. It reads a document line by line, collects consecutive text lines into a paragraph, and closes the paragraph when it meets a blank line, a heading or a rule.

A heading line is one to six # characters followed by a space and the text. Seven hashes are not a heading, and neither is #tag without a space. A horizontal rule is a line made only of three or more -, * or _ characters (spaces allowed between them). Go's regular expressions (RE2) have no back-references, so the check "all the same character" is a small function instead of a pattern. See Regular Expressions for what RE2 can and cannot do.

Go Stage 1: headings, paragraphs and rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

A few things to check in the output. The two-line paragraph becomes one <p> and &, < and > are escaped. The seven-hash line and the #nospace line land in a paragraph, because parseHeading rejected them. The first --- and the last * * * both become <hr>. The flush closure is a small example of why closures are handy: it captures out and para so every branch of the loop can end the current paragraph with one call.

Tip: strings.Builder with fmt.Fprintf is the standard way to assemble text in a loop. Building the result with out += ... copies the whole string every time. See Working with Strings.

Stage 2: blocks, lists, quotes and code

Lists, quotes and fenced code need more than a single flag. A list is a group of lines, a quote contains other blocks (a quote can hold a heading or a list), and inside a code fence nothing has any meaning at all, not even # or -.

So stage 2 splits the work. parseBlocks turns lines into a slice of block values, and render turns blocks into HTML. A block is a struct with a kind and the data that kind needs:

  • Fenced code starts with a line that begins with three backticks (and an optional language name) and runs to the next fence, or to the end of the document if it is never closed. The lines are kept exactly as written and only escaped at the end.
  • Lists are consecutive lines that match the same pattern: -, * or + followed by a space, or digits followed by . or ). Two small regular expressions capture the item text. A bullet line followed by a numbered line starts a new list, which is what most renderers do.
  • Quotes collect consecutive lines that begin with >, remove the marker and call parseBlocks on what is left. That recursion is all it takes to support quotes inside quotes and lists inside quotes.
  • A paragraph gathers lines until a blank line or until another block would start, so a heading directly below a text line is still a heading.
Go Stage 2: a block parser and a renderer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Run it and compare with the source lines. The * item and the - items end up in the same list because both match bulletRe. The 1. and 2) lines form an ordered list. The > block becomes a <blockquote> that contains a paragraph and a nested <ul>: the recursion in action. The code block keeps its < and && as escaped text, and the # and - inside it did not start a heading or list because the fence case is checked first.

Order matters in the switch. The fence case must come before the heading, rule and list cases, or a # comment inside a code block would become a heading. Whenever a parser has several rules, list the most "absorbing" one first. The startsBlock helper repeats the same priorities so a paragraph stops at the right place.

Stage 3: inline formatting with regexp

Block structure is done. Inside each block we still have raw text like use **bold** and code or a [link](https://go.dev). The inline function converts those with regular expressions. It has three traps.

1. Code spans protect their contents. In `a*b*c` the asterisks are literal text. So code spans must be found first and set aside. The technique is stash and restore: replace each finished piece of HTML with a placeholder (a NUL byte, a number, a NUL byte), run the remaining rules on the text, and put the pieces back at the end. Nothing the later rules do can touch the stashed HTML.

2. Escape before the emphasis rules, but never double-escape. Everything that was not stashed is passed through html.EscapeString. Placeholders survive because they contain no special characters. Link text is formatted by calling the same function recursively on the raw text, which is escaped once.

3. Underscores appear inside words. snake_case_name must stay as typed. Asterisk emphasis works anywhere (un*frigging*believable), but underscore emphasis requires a word boundary (\b) on the outside. Both kinds also demand that the emphasized text does not start or end with a space, so 2 * 3 * 4 is plain arithmetic.

The order of the emphasis rules also matters: ***text*** first, then **text**, then *text*. If the single-star rule ran first, it would grab pieces of the double-star markers.

Go Stage 3: bold, italic, code, links and images

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Look at the second paragraph's output: the code span keeps *stars* and escapes & and the tags, while the **bold** next to it is converted. The link text contains **bold** and is formatted because format calls itself on the link text. my_snake_case_name is unchanged, and 2 * 3 * 4 too. The raw <b>HTML</b> in the last paragraph is escaped and shown as text.

Tip: ReplaceAllStringFunc receives only the whole match, so the function runs the pattern again with FindStringSubmatch to get the groups. That double work is fine for text this short. For hot paths, use FindAllStringSubmatchIndex and build the result yourself, as explained in Regular Expressions.

Two things remain. The first is security. A Markdown link such as [click](javascript:alert(1)) becomes <a href="javascript:alert(1)">, and a click runs script. Escaping HTML does not prevent that, because the dangerous part is the URL scheme. The converter must check the scheme and refuse anything outside a short allow-list: http, https, mailto, or no scheme at all (relative links such as /go/introduction). Use an allow-list, never a block-list; there are too many ways to write a bad scheme. The check uses net/url. A URL that does not even parse (for example with control characters hidden in it) is rejected too. Unsafe links are rendered as their text only, and unsafe image sources are dropped.

The second is tests. A converter has dozens of small rules that interact, so the right test is a table of {name, input, want} rows, as in Table-Driven Tests. Every bug you fix later gets a new row, and the table never forgets it. Since the editor has no go test runner, main runs the same table by hand and prints the failures. The read-only file below shows how the same table would look in a real markdown_test.go:

Go
package main

import "testing"

func TestConvert(t *testing.T) {
	for _, tc := range cases { // the same table as in the program below
		t.Run(tc.name, func(t *testing.T) {
			if got := convert(tc.in); got != tc.want {
				t.Errorf("convert(%q)\n got: %q\nwant: %q", tc.in, got, tc.want)
			}
		})
	}
}

The final program has all the code and 24 test cases. The main function prints a summary line and the converted sample. A good test table also documents the edge cases you decided on, such as 2 * 3 * 4, my_var_name and the unclosed code fence.

Go Stage 4: safe links and 24 table tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Break something on purpose: change a want string or delete the javascript check in safeURL, then run again and read the failure report. The %q verb shows invisible things such as the trailing newline, which is why it is the right verb for comparing text in tests.

Design lessons from this project

  • Separate parsing from rendering. The block slice is a small intermediate representation. You can render it to HTML today and to plain text tomorrow without touching the parser, and you can test the two halves separately.
  • Stash and restore is the general trick for "apply rules to everything except these parts". Compilers, template engines and syntax highlighters use the same idea.
  • Escape by default. Every code path ends in html.EscapeString. A converter that escapes in one place and "forgets" another has a cross-site scripting hole, and URLs need an extra scheme check.
  • Order is part of the grammar. The fence is checked before the heading, the triple star before the double star, code spans before links. When two rules can match the same text, the order you test them in decides the answer.
  • RE2 is linear-time but small. There are no back-references and no look-ahead, so some checks (isRule) are plain Go code. In return, no input can make a pattern run for minutes.
  • Tables of tests grow with the code. Add a row for every bug.

Ideas to extend it

  • Setext headings (Title underlined with ===) and closing # sequences.
  • Nested lists using indentation, and list items that span several lines.
  • Tables in the GitHub style (pipes and a separator row).
  • Hard line breaks (two trailing spaces) and backslash escapes such as \*.
  • Heading ids and a generated table of contents, like the one on this site.
  • A command-line tool: read a file given as an argument with os.ReadFile, or standard input with bufio.Scanner, and write the HTML to standard output. See Command-Line Programs.
  • Fuzz the converter with go test -fuzz and assert that its output never contains <script. See Benchmarks and Fuzzing.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the converter find code spans before it applies bold and italic rules?

    In `a*b*c` the stars are literal text. Stashing the finished <code> HTML behind a placeholder protects it from every later rule.
  2. 2. Why is isRule written as a function instead of a pattern like ^([-*_])\1\1+$?

    Go uses RE2, which guarantees linear-time matching and therefore leaves out back-references and look-around. Plain Go code handles the 'all the same character' test.
  3. 3. What does convert("[x](javascript:alert)") produce in stage 4?

    safeURL allows only http, https, mailto and scheme-less URLs. For anything else the link is replaced by its text, so no clickable script URL is produced.
  4. 4. In parseBlocks, why is the code-fence case listed before the heading case?

    Once inside a fence the loop copies lines without interpreting them. That only works if the fence is recognized first and consumes its lines in an inner loop.
  5. 5. Why does the test runner print results with %q?

    %q prints a quoted string with escapes, which makes trailing newlines and stray spaces visible. Those are exactly the details that make text-comparison tests fail.

Summary

  • A Markdown converter has two stages: parse lines into blocks, then render blocks to HTML. Quotes call the block parser again on their contents.
  • Use strings.Builder to assemble output, strings.Cut, TrimSpace, HasPrefix and small regular expressions for line classification.
  • Escape every piece of text with html.EscapeString; protect code spans first with stash and restore.
  • Rule order is part of the grammar: absorbing rules (code fences) first, *** before ** before *.
  • Allow only known-safe URL schemes (http, https, mailto, relative); never try to block-list bad ones.
  • A table of {name, input, want} rows documents behavior and catches regressions; print failures with %q.

Next: Go Cheat Sheet