Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 39 ยท Parsing, Security and Tooling

Command-Line Data Tools

Learn the command-line tools for working with JSON, YAML, XML and CSV - jq, yq, xmllint, Miller, csvkit and linters - plus how queries and schema validation work, with runnable Node stand-ins.

Data files are everywhere, and you will often need to peek into one, reshape it, or check it without writing a program. The command line has a mature toolbox for that. Learning a few of these tools pays off every week: inspecting an API response, editing a Kubernetes manifest in a script, cleaning a CSV export, or failing a CI build when a config is invalid. This lesson tours the main tools and then shows, with runnable Node stand-ins, how their two core ideas work: path queries and schema validation.

The tools below (jq, yq, xmllint, ...) are not installed in this site's sandbox, so the command examples are read-only: install the tool locally to run them. The TryIt blocks run real JavaScript that imitates the concepts.

jq: the JSON processor

jq reads JSON, applies a small functional language (a filter), and prints the result. It is the standard tool for JSON on the command line.

Terminal
# Pretty-print
curl -s https://api.example.com/users | jq .

# Pick fields
jq '.users[] | {name, email}' data.json

# Filter and count
jq '[.users[] | select(.active)] | length' data.json

# Raw strings (no quotes), one per line
jq -r '.users[].name' data.json

# Build new JSON with variables from the shell
jq --arg who "ada" '.users[] | select(.name == $who)' data.json

The pieces: . is the current value, .a.b walks keys, .[0] indexes, .[] iterates, | pipes the output of one filter into the next, select(cond) keeps values where the condition holds, and map, sort_by, group_by, to_entries and friends transform lists. jq -c prints compact one-line JSON, which pairs perfectly with JSON Lines.

yq: YAML (and more)

yq (the Go version by Mike Farah) applies jq-like expressions to YAML, JSON, XML, TOML and CSV, and can edit in place while preserving comments.

Terminal
yq '.services.web.image' compose.yaml
yq -i '.version = "2.0"' config.yaml           # edit in place
yq -o=json . config.yaml                        # YAML to JSON
yq -p=json -o=yaml . data.json                  # JSON to YAML
yq eval-all '. as $item ireduce ({}; . * $item)' a.yaml b.yaml   # deep merge files

Another tool called yq (the Python one, a jq wrapper) exists; check which one you have with yq --version.

XML tools: xmllint and xmlstarlet

xmllint ships with libxml2 and is the standard XML checker. It checks well-formedness, validates against schemas, evaluates XPath and pretty-prints.

Terminal
xmllint --noout file.xml                          # well-formed?
xmllint --format file.xml                         # pretty-print
xmllint --xpath '//book[price>20]/title/text()' catalog.xml
xmllint --schema catalog.xsd --noout catalog.xml  # validate against an XSD
xsltproc style.xsl input.xml                      # apply an XSLT stylesheet

When parsing XML from untrusted sources, pass --nonet and avoid --noent so that network access and entity substitution stay off (see Data Format Security).

CSV and tabular tools

  • csvkit (csvcut, csvgrep, csvstat, csvjson, csvsql) treats CSV like a database.
  • Miller (mlr) is "awk, sed and sort for CSV, TSV and JSON": mlr --icsv --ojson cat data.csv converts, mlr --icsv --opprint sort -nr price then head -n 5 data.csv ranks.
  • xsv / qsv are fast Rust tools for huge files, column -s, -t is a quick aligned view, and DuckDB can run SQL straight on CSV and JSON files: duckdb -c "SELECT user, count(*) FROM 'events.jsonl' GROUP BY 1".

Validators and linters

Format Tools
JSON jq empty file.json (syntax check), ajv (schema), jsonlint
YAML yamllint, kubeconform/kubeval (Kubernetes), ajv with a YAML loader
XML xmllint --noout, --schema, --dtdvalid
TOML taplo, toml-test
Markdown markdownlint-cli2, lychee or markdown-link-check (links), vale (prose)

Put these in your pre-commit hooks and CI pipeline: a failing check on a malformed file is far cheaper than a failed deploy.

How path queries work

Behind jq, JSONPath and XPath there is the same idea: a query is a list of steps; each step maps one value to zero or more values.

flowchart LR
  A["data"] --> B[".users"]
  B --> C["[]  (each item)"]
  C --> D["select(.active)"]
  D --> E[".name"]
  E --> F["results"]

A pipeline of steps; each step may fan out (one value becomes many) or filter (some disappear).

Node.js A tiny jq-style query evaluator (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The evaluator runs each step on every current value (flatMap), which is exactly why .users[] can feed a filter that sees one user at a time. The same model explains XPath (//book[price>20]/title) in the XML chapter.

How schema validation works

A schema is a machine-readable description of what a valid document looks like. Validators walk the data and the schema together. Here is a small validator for the core JSON Schema keywords (type, required, properties, enum, minimum, items):

Node.js A minimal JSON Schema validator (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The same pattern appears in JSON Schema, XML Schema (XSD) and Kubernetes validation: data plus schema in, a list of precise errors out. Collecting all errors with paths, rather than stopping at the first, is what makes validators useful.

A practical toolbox workflow

  1. Inspect: jq . or yq . to pretty-print; head and wc -l for size.
  2. Explore: jq 'keys', jq '.[0]', mlr summary to learn the shape.
  3. Validate: lint syntax first, then validate against a schema.
  4. Transform: write the jq/yq/mlr filter, test it on a small sample, then run on the real file.
  5. Automate: save as a script, add to CI or a pre-commit hook.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the jq filter .users[] | .name do?

    The [] iterates the array, and the pipe passes each user to .name.
  2. 2. Which tool checks an XML file against an XSD schema?

    xmllint can validate against XML Schema, DTD and RELAX NG.
  3. 3. Why should a validator report all errors with their paths?

    Complete, located errors are far easier to act on than a single failure message.
  4. 4. What is the advantage of yq -i over editing a YAML file with a script that loads and re-dumps it?

    Loading and re-dumping YAML usually loses comments and formatting; yq edits in place and keeps them.

Summary

  • jq for JSON, yq for YAML (and conversion), xmllint for XML, Miller/csvkit/DuckDB for tables.
  • Queries are pipelines of steps that fan out and filter; schema validation walks data and schema together.
  • Lint syntax first, validate against a schema second, and automate both in CI.
  • The commands on this page are read-only here; the Node evaluators are teaching stand-ins.

Next: Project: A Layered Configuration Loader