Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 119 ยท Data Formats

CSV, Properties and Other Formats

Read and write CSV in Java without the classic bugs, handle quotes, commas and newlines, master Properties files, meet YAML and binary formats, and pick the right one.

CSV looks like the simplest format there is: values separated by commas, one record per line. That impression is exactly why so much CSV code is broken. Real files contain commas inside values, quotes inside quotes, line breaks inside a single field, semicolons instead of commas, and an invisible byte at the very start. A parser that calls line.split(",") works in the demo and corrupts data in production.

This lesson shows those traps on real examples, builds a small parser that handles them correctly, and points you to the libraries you should use in production. Then it covers .properties files (Java's own configuration format and its own surprises), YAML through Jackson, a quick tour of binary formats, and a practical guide to choosing a format. JSON and XML were covered in JSON with Jackson and XML Processing.

The rules of CSV

There is no single CSV standard, but RFC 4180 describes what most tools produce and expect:

  1. Each record is one line, ending in a line break (officially CRLF, \r\n; plain \n is common).
  2. Fields are separated by commas. The first line may be a header with column names.
  3. A field that contains a comma, a double quote or a line break must be enclosed in double quotes.
  4. A double quote inside a quoted field is written twice: "She said ""hi""" means She said "hi".
  5. Every record should have the same number of fields.

So this is a valid file with three records and three columns:

TEXT
id,name,comment
1,"Smith, Anna","Ordered ""the blue one"""
2,Ben Okafor,"Line one
line two"

Notice record 2: its comment spans two physical lines. Any code that reads the file line by line has already lost before it splits anything.

Why split(",") fails

Here is the naive approach, run on that file. It reads line by line and splits on commas.

Java The naive CSV parser breaks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Every line except the header is wrong:

  • Record 1 has 4 fields, because the comma in Smith, Anna was treated as a separator, and the quotes are still part of the values.
  • Record 2 became two records, with half a comment each.
  • Record 3 has 2 fields instead of 3: String.split silently drops trailing empty strings. The last two lines of output show the fix for that one (split(",", -1)), but no amount of split tricks handles quotes and embedded line breaks.

(The \" in the text block is only there because a text block can't contain three quotes in a row otherwise; see Text Blocks and Escape Sequences.)

A correct parser in 40 lines

A real CSV parser reads character by character and remembers one thing: am I inside quotes? Inside quotes, commas and line breaks are ordinary characters, and "" means one literal quote. Outside quotes, a comma ends a field and a line break ends a record.

The example below implements that state machine, plus the matching writer, which quotes a field only when it has to. It also guards against CSV injection: spreadsheet programs treat a cell starting with =, +, - or @ as a formula, so an attacker who sets their name to =HYPERLINK(...) can run formulas on the computer of whoever opens your export. Prefixing such values with a ' defuses them.

Java An RFC 4180 parser and writer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

All three records now have exactly three fields, the quotes are gone, the doubled quotes became single ones, and the line break inside record 2 survived. Writing the data back produces a file that reads back identically. The attacker's formula was written as '=HYPERLINK(...), which spreadsheets show as plain text (it contains quotes and a comma, so it is also quoted).

This parser is fine for learning and small internal tools. For production data, use a library: they handle more edge cases, are faster, and stream large files.

CSV in the real world

Even a correct parser meets files that bend the rules:

  • Different separators. Excel in many European locales writes ; because , is the decimal separator there. Tabs (TSV) and | are common too. Good libraries let you configure the delimiter.
  • Encoding and the BOM. Excel's "CSV UTF-8" adds a byte order mark (U+FEFF) at the start, so your first column header becomes \uFEFFid and header.equals("id") fails. Strip it when you read, and always specify the charset explicitly (see Character Sets and Encoding).
  • Lost formatting. Spreadsheets "helpfully" turn 00742 into 742, long numbers into 7.42E+11 and 3-4 into a date. If a CSV went through Excel, check IDs and codes.
  • Ragged rows. Some rows have fewer or more fields. Decide whether to reject them or pad them, and report the line number.
  • Huge files. Read with a BufferedReader and process record by record instead of loading everything into memory.

CSV libraries (read-only)

Library Notes
Apache Commons CSV Small, well-known, supports many dialects (CSVFormat.EXCEL, TDF, RFC4180).
OpenCSV Popular; can bind rows to beans with annotations.
FastCSV Very fast, no dependencies, RFC 4180 compliant.
Jackson jackson-dataformat-csv CsvMapper reads CSV into records with the same API you know from ObjectMapper.

A taste of Apache Commons CSV (not available on our server):

Java
CSVFormat format = CSVFormat.RFC4180.builder()
    .setHeader()                    // first line is the header
    .setSkipHeaderRecord(true)
    .build();

try (Reader in = Files.newBufferedReader(Path.of("customers.csv"), StandardCharsets.UTF_8);
     CSVParser parser = format.parse(in)) {
    for (CSVRecord row : parser) {
        String name = row.get("name");          // access by column name
        long line = parser.getCurrentLineNumber();
    }
}

Properties files

.properties files are Java's original configuration format, and you still meet them everywhere: application.properties in Spring Boot, message bundles for translations, build settings. java.util.Properties reads and writes them. The basics are covered in Properties Files and Configuration; here are the parsing rules that surprise experienced developers:

  • The separator can be =, : or plain whitespace. server port 8080 sets server to port 8080.
  • Whitespace around the key and before the value is trimmed, but trailing whitespace in the value is kept. A stray space after true makes "true ".equals(...) fail.
  • Lines starting with # or ! are comments. A # in the middle of a line is part of the value.
  • A backslash at the end of a line continues the value on the next line.
  • If a key appears twice, the last one wins, silently.
  • load(InputStream) reads ISO-8859-1, so UTF-8 text like Zรผrich arrives garbled. Use load(Reader) with an explicit UTF-8 reader. (Java 9+ reads ResourceBundle properties as UTF-8, but Properties.load(InputStream) still uses ISO-8859-1.)
Java Properties parsing surprises

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Work through the output carefully, because every line is a lesson:

  • db.pool size 10 became the key db.pool with the value size 10.
  • feature.enabled is true with a trailing space (the \s escape in the text block keeps it visible in the source), so Boolean.parseBoolean returns false. Exactly the kind of bug that takes an afternoon to find.
  • greeting includes # this is NOT a comment.
  • The continued message line was joined, with the leading spaces of the second line removed.
  • timeout is 45, the last definition.
  • city was written with a รผ escape, which Properties decodes into รผ.
  • Read as UTF-8 bytes through load(InputStream), Zรผrich became 7 garbled characters instead of 6.

YAML via Jackson

YAML is the other common configuration format (Spring Boot's application.yml, Kubernetes, Docker Compose, CI pipelines). It is a superset of JSON that uses indentation instead of braces:

YAML
server:
  port: 8080
datasource:
  url: jdbc:postgresql://db:5432/shop
  pool:
    max-size: 10
features: [search, reviews]

In Java, the easiest way to read it is Jackson's YAML module, jackson-dataformat-yaml (built on SnakeYAML). Its YAMLMapper is an ObjectMapper, so everything from the Jackson lessons works the same (read-only; the module isn't on our server):

Java
import com.fasterxml.jackson.dataformat.yaml.YAMLMapper;

record Pool(@JsonProperty("max-size") int maxSize) {}
record DataSourceConfig(String url, Pool pool) {}
record AppConfig(Map<String, Object> server, DataSourceConfig datasource, List<String> features) {}

AppConfig config = new YAMLMapper().readValue(Path.of("app.yml").toFile(), AppConfig.class);
System.out.println(config.datasource().pool().maxSize());   // 10

YAML has traps of its own:

  • Indentation is syntax, and tabs are not allowed for it. One misaligned space moves a key to a different parent.
  • Unquoted values guess their type. Under the older YAML 1.1 rules, which SnakeYAML follows, country: NO becomes the boolean false (the famous "Norway problem"), version: 1.10 becomes the number 1.1, and on/off/yes are booleans. Quote strings that could be misread.
  • Untrusted YAML is dangerous with libraries that create arbitrary Java classes from tags like !!java.lang.ProcessBuilder. Jackson's YAML module maps into your own types and doesn't do this, but older direct uses of SnakeYAML did; SnakeYAML 2.0 made safe loading the default.

Binary and other formats

Text formats are easy to read and debug, but verbose and slow to parse. For high-volume internal traffic and big data, binary formats are common:

  • Protocol Buffers (Google) and Apache Avro: schema-based, compact, fast, with code generation and rules for evolving the schema. Protobuf is the format behind gRPC; Avro is common with Kafka.
  • MessagePack and CBOR: "binary JSON", same data model, smaller and faster. Jackson has modules for both, as well as for Smile, Ion and others.
  • Apache Parquet and ORC: column-oriented files for analytics (Spark, data lakes).
  • TOML and INI: simple configuration formats, less common in Java.
  • Java serialization (ObjectOutputStream): avoid it for data exchange; it is Java-only, fragile and a serious security risk with untrusted input (see Serialization).

Choosing a format

The example below writes the same three products as CSV, JSON and properties, and compares their sizes. It is a small illustration of the trade-off between compactness and structure.

Java One dataset, three formats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

CSV is the smallest, but it only knows strings: the tag list had to be packed into one field with a made-up | separator, and the reader must know the price is a decimal. JSON is larger but self-describing and keeps the list as a real array. Properties repeats every key and needs naming conventions to express structure; it is meant for configuration, not datasets. Note that Properties.store also writes a timestamp comment as its first line, which the example skips so the output stays the same between runs.

A rough guide:

You need to... Use
Exchange tabular data with spreadsheets, analysts, legacy systems CSV (with a real parser, explicit charset and delimiter)
Build web APIs, store documents, send events between services JSON
Integrate with SOAP, standards bodies, publishing, office files XML
Write configuration for Java apps properties or YAML
Move lots of data fast between your own services Protobuf, Avro or another binary format
Store large datasets for analytics Parquet

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does "x,,y,,".split(",").length return?

    split keeps the empty field between the two commas in the middle, but drops trailing empty strings, so the result is [x, , y]. split(",", -1) would return all 5.
  2. 2. How is the value Say "no", please written as a single CSV field?

    The field contains quotes and a comma, so it is wrapped in double quotes and each inner quote is doubled.
  3. 3. A properties file contains debug=true (with a trailing space). What does Boolean.parseBoolean(props.getProperty("debug")) return?

    Properties keeps trailing whitespace in values, and parseBoolean only accepts exactly "true" (ignoring case), so "true " gives false.
  4. 4. Why does an exported customer named =SUM(A1:A9) need special handling in a CSV export?

    Excel and similar programs treat cells starting with =, +, - or @ as formulas. Prefix such values (for example with an apostrophe) in exports meant for spreadsheets.
  5. 5. Under YAML 1.1 rules (as in SnakeYAML), what is the value of country in country: NO?

    YAML 1.1 treats yes/no/on/off as booleans, so NO becomes false. Quote it as "NO" to keep the string.

Summary

  • CSV fields that contain commas, quotes or line breaks are quoted, and quotes inside are doubled. Line-by-line reading and split(",") can't handle that; split also drops trailing empty fields.
  • A correct parser tracks whether it is inside quotes; for production, use a library such as Apache Commons CSV, FastCSV, OpenCSV or Jackson's CsvMapper.
  • Watch for semicolon delimiters, the UTF-8 BOM, spreadsheet reformatting and CSV injection in exports.
  • Properties files accept =, : or whitespace as separators, keep trailing spaces, let the last duplicate win, and load(InputStream) assumes ISO-8859-1; use a UTF-8 Reader.
  • YAML is read in Java with Jackson's YAMLMapper; beware indentation and unquoted values like NO or 1.10.
  • Binary formats (Protobuf, Avro, MessagePack, CBOR, Parquet) trade readability for size and speed. Pick the format by who reads the data and how much of it there is.

Next: Unit Testing with JUnit 6 starts the Testing chapter, so you can prove that parsers like the one in this lesson keep working.