Skip to content
Tutorials Center
My learning

Topics

Lesson 81 of 147 ยท Standard Library

CSV, XML and Other Encodings

Read and write CSV with encoding/csv, XML with encoding/xml, Go-only data with gob, and convert bytes with base64, hex and encoding/binary. Includes text versus binary format trade-offs.

JSON is not the only format a Go program meets. Spreadsheets export CSV, older systems and many feeds speak XML, byte-level protocols pack numbers into exact byte layouts, and tokens and hashes travel as Base64 or hex text. The standard library has a package for each, and they all follow the same style: small functions, io.Reader and io.Writer based APIs, and struct tags where a mapping is needed.

This lesson tours the packages under encoding/: csv, xml, gob, base64, hex and binary, and finishes with advice on choosing between text and binary formats.

CSV with encoding/csv

CSV looks trivial until a field contains a comma, a quote or a line break. Never split lines on "," yourself; use csv.Reader, which handles quoting ("say ""hi"", then leave") and embedded newlines correctly.

Go Reading CSV records

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Read returns one record as a []string. Every field is a string, so converting to numbers is your job (with strconv, checking the error). ReadAll returns all records at once and is fine for small files; for big ones loop with Read, which holds only one record in memory.

Reader settings

The Reader struct has fields you set before the first Read:

  • Comma: the field separator (a rune). Use ';' for European exports or '\t' for TSV.
  • Comment: lines starting with this rune are skipped.
  • FieldsPerRecord: 0 (the default) means every record must have as many fields as the first; a negative value disables the check.
  • TrimLeadingSpace: ignore spaces after a separator.
  • LazyQuotes: tolerate stray quotes in sloppy files.
  • ReuseRecord: reuse the returned slice for speed (copy it if you keep it).
Go Semicolons, comments and ragged rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

When a row has the wrong number of fields, the error is a *csv.ParseError holding the Line and Column; errors.Is(err, csv.ErrFieldCount) identifies this case. Note that Read still returns the record together with the error, so you can keep the data and decide what to do.

Writing CSV

csv.Writer quotes fields only when needed. It buffers output, so call Flush and check Error at the end, or the last rows may never be written:

Go Writing CSV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Fields with commas, quotes or newlines are wrapped in quotes, and inner quotes are doubled. Set w.Comma for another separator and w.UseCRLF = true for Windows line endings.

CSV files that start with =, +, - or @ can be interpreted as formulas when opened in a spreadsheet. If you export user-provided text, prefix such cells with a single quote or tab to avoid CSV injection.

XML with encoding/xml

encoding/xml mirrors encoding/json: you describe the document with struct tags and call Marshal or Unmarshal. Tag options choose between elements, attributes and text:

  • xml:"name": a child element called name.
  • xml:"id,attr": an attribute.
  • xml:",chardata": the text inside the element.
  • xml:"items>item": nest elements, here <items><item>.
  • xml:",omitempty", xml:"-" and xml:",comment" work as you'd expect.

The special field XMLName xml.Name sets and receives the element name.

Go Marshal and Unmarshal XML

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Marshal escapes special characters (& becomes &amp;), and xml.Header is the standard <?xml ...?> declaration line, which Marshal does not add on its own.

Streaming large XML

xml.Unmarshal loads a whole document. To process a huge file one element at a time, use xml.NewDecoder and walk the tokens, calling DecodeElement on the elements you care about:

Go Token-by-token decoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

encoding/xml is a pragmatic subset of XML: it does not validate against schemas and has limited namespace support. It is fine for feeds, SOAP-less APIs and config files. For heavy XML work, such as XPath or XSD, programs usually reach for third-party packages.

Base64 and hex: bytes as text

Binary data (hashes, keys, small images) often has to travel through text-only channels: JSON, URLs, headers. Base64 uses 64 characters and expands data by about a third; hex uses two characters per byte. Both are encodings, not encryption: anyone can reverse them.

Go base64 and hex

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

Pick StdEncoding for general use, URLEncoding or RawURLEncoding for URLs, filenames and JWT-style tokens. For streams there are base64.NewEncoder and NewDecoder, which wrap a writer or reader. hex.Dump prints the familiar offset, bytes and ASCII layout that is great for debugging.

Binary data with encoding/binary

Network protocols and file formats specify exactly how many bytes a number occupies and in which byte order (endianness). encoding/binary gives you binary.BigEndian and binary.LittleEndian for single values, and binary.Read/binary.Write for whole fixed-size structs:

Go Byte order and fixed-size structs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

binary.Write and Read use reflection and only accept fixed-size types: no int (its size varies), strings, slices without a known length or maps. Use int32, uint16 and arrays instead. For hot paths, the PutUint32/Uint32 and Append* functions are much faster because they avoid reflection.

gob: Go-to-Go binary encoding

encoding/gob serializes almost any Go value (including maps, slices, nested structs and interfaces you register) into a compact, self-describing binary stream. It is designed for Go programs talking to Go programs, such as the standard net/rpc package. Only exported fields are sent, and the receiver needs a compatible type, not an identical one.

Go A gob round trip

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with bufio.Scanner or fmt.Scan), it gets the next line from this box.

For a single value gob is not smaller than JSON, because the stream carries type information first. The savings appear when one encoder sends many values of the same type, since the description is sent only once. Do not use gob as a public or long-term storage format: other languages can't read it, and it isn't designed to be hardened against hostile input.

Text or binary: choosing a format

Format Readable by humans Cross-language Typical use
JSON yes yes web APIs, config, logs
CSV yes yes tables, spreadsheet import and export
XML yes yes feeds, legacy and document formats
gob no no (Go only) Go-to-Go RPC, caches
encoding/binary no yes protocols and file headers with a fixed layout

Text formats are easy to debug and share, but larger and slower to parse. Binary formats are compact and fast but need a spec (or a schema tool such as Protocol Buffers, which lives outside the standard library) to stay compatible over time. A good default: JSON at the edges of your system, something else only when measurements or a protocol demand it.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A CSV field contains a comma: Ada,"London, UK",98. How many fields does csv.Reader return?

    The quotes protect the comma, so the record is [Ada, "London, UK", 98]: three fields.
  2. 2. What must you do after the last Write call on a csv.Writer?

    The writer buffers its output. Without Flush the last records may never reach the destination.
  3. 3. Which struct tag makes a field an XML attribute instead of a child element?

    The attr option writes the field as an attribute. chardata stores the element's text content.
  4. 4. Is Base64 a way to protect secrets?

    Base64 and hex only change how bytes are represented as text. Use encryption to keep data secret.
  5. 5. Why does binary.Write reject a struct with an int field?

    Use int32, uint16 and similar fixed-size types, so every machine writes the same number of bytes.

Summary

  • Use encoding/csv instead of splitting on commas: it handles quotes and newlines; set Comma, Comment and FieldsPerRecord as needed, and always Flush a csv.Writer.
  • CSV values are strings; convert with strconv and check errors.
  • encoding/xml maps structs with tags (attr, chardata, a>b); use xml.Decoder tokens for big documents.
  • base64 and hex turn bytes into text; they are encodings, not encryption. Choose URLEncoding or RawURLEncoding for URLs.
  • encoding/binary reads and writes fixed-size values in an explicit byte order.
  • gob is a compact Go-only format; prefer JSON or protocol schemas for anything shared.

Next: Regular Expressions shows how to search and transform text with patterns.