Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท Templates and Data Bags

Data Bags: Shared Data for Many Nodes

Learn what Chef data bags are, how data bag items are stored as JSON, how recipes load and search them, and when to use them instead of attributes.

Attributes belong to a node or a cookbook. But some data belongs to the whole organization: the list of users who may log in, the virtual hosts of a company's websites, the IP addresses of the monitoring servers. Putting that data into every cookbook would be messy. Chef gives it a separate home called a data bag.

A data bag is a named collection on the Chef Infra Server. Each entry in it is a data bag item, a JSON document with an id. Recipes read items with data_bag_item and look through many of them with search. In this lesson you will see how data bags are created, how recipes use them, and how to decide between a data bag, an attribute and a role.

There is no Chef Server on this site. The knife commands and recipe code below are real and read-only. The runnable examples build a small in-memory data bag in Ruby so you can practice the logic that recipes use: load an item, loop over items, and filter.

Anatomy of a data bag

Think of data bags as folders and items as files:

flowchart LR
  S["Chef Infra Server"] --> B1["data bag: users"]
  S --> B2["data bag: sites"]
  B1 --> I1["item: alice"]
  B1 --> I2["item: bob"]
  B2 --> I3["item: shop"]

Each bag holds items; each item is JSON with a unique id.

An item is plain JSON. The only required key is id, which must match the file name when you keep items in a repository:

JSON
{
  "id": "alice",
  "shell": "/bin/bash",
  "groups": ["dev", "docker"],
  "ssh_keys": ["ssh-ed25519 AAAA... alice@laptop"]
}

Typical repository layout and the knife commands that upload it:

TEXT
data_bags/
  users/
    alice.json
    bob.json
Terminal
knife data bag create users
knife data bag from file users data_bags/users/alice.json
knife data bag show users alice
knife data bag list

Using data bags in a recipe

data_bag_item fetches one item, data_bag lists the item names, and search finds items matching a query:

Ruby
alice = data_bag_item('users', 'alice')

user alice['id'] do
  shell alice['shell']
  action :create
end

# Every item in the bag:
data_bag('users').each do |name|
  entry = data_bag_item('users', name)
  # ...
end

# Only items whose JSON matches a query:
search(:users, 'groups:docker').each do |entry|
  group 'docker' do
    members [entry['id']]
    append true
  end
end

Items behave like hashes, so entry['shell'] works. Note one difference from node attributes: data bag items are not merged by precedence. You get exactly what is stored.

A stand-in data bag you can run

The following Ruby code mimics the same three operations on a hash. It is not Chef, but the logic (load one item, loop, filter by a field) is exactly what recipes do:

Ruby Stand-in: data_bag_item, data_bag and search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

The last call shows a real behavior: asking for an item that does not exist raises an error and stops the Chef run, so recipes often guard with data_bag('users').include?('alice').

Turning items into resources

The usual pattern is "for each item, declare a resource". Because resources are idempotent, you can add a user to the bag and the next run creates only that user:

Ruby Stand-in: loop over items and declare resources

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Data bag, attribute or role?

Situation Best place
A default value for one cookbook (port, package name) Cookbook attribute
A setting that differs per environment Environment attribute
A list of things that many nodes iterate over (users, sites, certificates) Data bag
A secret such as a password or key Encrypted data bag or a secrets manager
Which recipes a kind of server runs Role or Policyfile

Rule of thumb: if the data is a list of records, or if non-programmers need to edit it without touching a cookbook, use a data bag.

Limits and good habits

  • Item size is limited by the server (about 1 MB by default), so do not store large files.
  • Keep data bags in version control (data_bags/ in the repository) and upload them in your pipeline, so the bag is reviewed like code.
  • Search needs the server and is slower than reading a hash, and with a new Chef Server search results can lag a few seconds behind a change.
  • In Chef Solo or local mode (chef-client -z), data bags are read from the local data_bags/ directory, which makes tests easy.
  • Never put secrets in a plain data bag. The next lesson shows encryption.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which key must every data bag item contain?

    Each item is a JSON document with a required id that is unique inside the bag.
  2. 2. Which call fetches one item named alice from the users bag?

    data_bag_item takes the bag name and item id. data_bag lists the item names of a bag.
  3. 3. When is a data bag a better choice than a cookbook attribute?

    Data bags suit lists of records that many nodes loop over and that people want to edit without changing a cookbook.
  4. 4. Are data bag items merged with attribute precedence rules?

    Data bags are plain data. There is no precedence merge.

Summary

  • A data bag is a named collection on the Chef Server; each item is a JSON document with an id.
  • Recipes use data_bag, data_bag_item and search; local mode reads data_bags/ from disk.
  • Loop over items and declare resources; idempotence makes later runs cheap.
  • Use data bags for shared lists of records, attributes for defaults, and encrypted bags or a secrets manager for secrets.
  • Keep bags in version control and mind item size and search lag.

Next: Encrypted Data Bags and Secrets