Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: A Multi-Environment Fleet

Combine roles, environments, data bags, search and Policyfile ideas in one project: manage dev, staging and production servers with a shared cookbook, in a Ruby model.

Real infrastructure has more than one server and more than one stage. In this project you design the configuration for a small company: web servers behind a load balancer, a database, three environments (dev, staging, production), a shared list of administrators, and a rollout process. You will use almost everything from the second half of this course: roles, environments, attribute precedence, data bags, search and promotion.

The real Chef files are shown read-only. A Ruby model at the end combines them to compute each node's final settings and its load balancer configuration, so you can check your design before touching a server.

Requirements

  1. Three environments: dev, staging, production.
  2. Roles: base (every node), web, lb (load balancer), db.
  3. Different settings per stage: fewer workers in dev, strict log level in production.
  4. Admin users come from one data bag and exist on every node; only production admins get sudo rights on production.
  5. The load balancer builds its backend list by searching for web nodes in its own environment.
  6. Cookbook versions are exact in production, loose in dev.
  7. Changes are promoted dev, then staging, then production.

Design

flowchart LR
  subgraph prod["production environment"]
    LB1["lb01 role[base], role[lb]"] --> W1["web01 role[base], role[web]"]
    LB1 --> W2["web02 role[base], role[web]"]
    W1 --> D1["db01 role[base], role[db]"]
  end
  subgraph stg["staging environment"]
    LB2["lb01 (staging)"] --> W3["web01 (staging)"]
  end

Same roles in every environment. The environment decides which nodes belong together and which versions and settings apply.

Roles

Ruby
# roles/base.rb
name 'base'
run_list 'recipe[apt]', 'recipe[users]', 'recipe[ntp]', 'recipe[monitoring::agent]'

# roles/web.rb
name 'web'
run_list 'recipe[nginx]', 'recipe[myapp::web]'
default_attributes('myapp' => { 'workers' => 4 })

# roles/lb.rb
name 'lb'
run_list 'recipe[haproxy]', 'recipe[myapp::lb]'

Environments

Ruby
# environments/dev.rb
name 'dev'
cookbook_versions('myapp' => '>= 0.1.0')
default_attributes('myapp' => { 'workers' => 1, 'log_level' => 'debug' },
                   'users'  => { 'sudo_groups' => %w[dev ops] })

# environments/production.rb
name 'production'
cookbook_versions('myapp' => '= 2.5.0', 'nginx' => '= 12.1.3', 'users' => '= 1.8.2')
default_attributes('myapp' => { 'log_level' => 'warn' },
                   'users'  => { 'sudo_groups' => %w[ops] })
override_attributes('myapp' => { 'debug_endpoints' => false })

Data bag of administrators

JSON
{ "id": "alice", "groups": ["dev", "ops"], "shell": "/bin/bash" }
{ "id": "bob",   "groups": ["dev"],        "shell": "/bin/zsh" }

The users recipe creates every user from the bag and grants sudo to those who are members of node['users']['sudo_groups']. In production only ops may use sudo, so Bob exists everywhere but can use sudo only in dev and staging if staging lists dev.

The load balancer recipe

Ruby
# cookbooks/myapp/recipes/lb.rb
backends = search(:node, "role:web AND chef_environment:#{node.chef_environment}",
                  filter_result: { 'name' => ['name'], 'ip' => ['ipaddress'] })
                  .sort_by { |n| n['name'] }

template '/etc/haproxy/haproxy.cfg' do
  source 'haproxy.cfg.erb'
  variables(backends: backends, stats: node.chef_environment != 'production')
  notifies :reload, 'service[haproxy]', :delayed
end

The two details that make this safe: the chef_environment restriction and the sort, which keeps the file the same between runs (otherwise search order changes cause pointless reloads).

Model the whole fleet

The model below holds nodes, roles, environments and data bag items. For each node it computes the merged attributes (cookbook default < environment default < role default, and environment override last), the sudo users, and the load balancer backends.

Ruby Stand-in: compute settings for every node

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with gets), it gets the next line from this box.

Check the output against the requirements. Web nodes get 4 workers from the role in staging and production, but the dev environment's workers: 1 default is overridden by the role default (role beats environment at the default level), so dev web nodes also get 4. That surprises many people. If you want dev to be smaller, either set override_attributes in the environment or drop the role default and put the numbers in the environments. Production alone hides debug endpoints through its override, and only alice has sudo there. Each load balancer lists only the web nodes of its own environment.

Rollout process

  1. A developer changes the myapp cookbook and bumps the version to 2.6.0; CI runs Cookstyle, ChefSpec and Test Kitchen.
  2. The pipeline uploads it and sets dev to >= 0.1.0, so dev picks it up automatically.
  3. After checks in dev, change the staging environment to = 2.6.0.
  4. After staging looks good, change production to = 2.6.0 in a reviewed pull request.
  5. If something goes wrong, change the constraint back to = 2.5.0; the next runs restore the old version.

With Policyfiles you would push the same locked policy to the dev, staging and production groups in turn, which gives the same flow with stronger guarantees.

Things to improve

  • Add chef_environment checks in the recipes so a node cannot accidentally join the wrong group.
  • Put database passwords into an encrypted data bag per environment (or a secrets manager) and mark templates sensitive.
  • Use splay so 200 web nodes do not hit the server at once.
  • Alert on nodes that have not checked in for three intervals.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the load balancer search include chef_environment?

    Without the environment filter every node with the web role would be listed, in every stage.
  2. 2. A role sets workers to 4 as a default, and the dev environment sets workers to 1 as a default. What value do dev web nodes get?

    At the default level the role ranks above the environment, so the role value wins.
  3. 3. Why is the backend list sorted in the recipe?

    A stable order keeps the rendered file identical when nothing has changed.
  4. 4. How do you roll back a bad cookbook release with environments?

    Changing the constraint makes the next runs use the previous version.

Summary

  • Combine roles (what a server is), environments (which stage, versions and stage settings) and data bags (shared records) for a multi-stage fleet.
  • Search with an environment filter builds dynamic configuration such as load balancer backends.
  • Know the precedence: a role default beats an environment default, an environment override beats both.
  • Promote changes dev, staging, production, pin production exactly, and keep rollback a one-line change.

Next: Chef Practice Set and Interview Questions