Exploring "Infrastructure as code" with Opscode Chef Part 3 (the first cookbook)

What is a cookbook?

Cookbooks serve as the fundamental unit of configuration and policy details that Chef uses to bring a node into a specific state.  A cookbook is a folder that stores all policies & configurations related to clients. Cookbooks are created on the workstation and then uploaded to a Chef server. From there, recipes and policies described within the cookbook can be assigned to nodes as part of the node's "run-list". A run-list is a sequential list of recipes and roles that are run on a node by chef-client in order to bring the node into compliance with the policy you set for it.

Creating a cookbook:

To create a new cookbook use the 'knife cookbook create' command followed by the cookbook name:

[sahil@cwork chef-repo]$ knife cookbook create apache

WARN: This command is being deprecated in favor of `chef generate cookbook` and will soon return an error.

Please use `chef generate cookbook` instead of this command.

 at /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.15.19/lib/chef/knife.rb:429:in `block in run_with_pretty_exceptions'

** Creating cookbook apache in /home/sahil/chef-repo/cookbooks

** Creating README for cookbook: apache

** Creating CHANGELOG for cookbook: apache

** Creating metadata for cookbook: apache

This creates a cookbook called apache & creates the entire cookbook directory structure for this cookbook under /home/sahil/chef-repo/cookbooks/apache.

Let's see what chef cooked up in this folder:

[sahil@cwork apache]$ pwd

/home/sahil/chef-repo/cookbooks/apache

[sahil@cwork apache]$ ls -l

total 12

drwxrwxr-x. 2 sahil sahil    6 Nov  5 10:40 attributes

-rw-rw-r--. 1 sahil sahil  433 Nov  5 10:40 CHANGELOG.md

drwxrwxr-x. 2 sahil sahil    6 Nov  5 10:40 definitions

drwxrwxr-x. 3 sahil sahil   20 Nov  5 10:40 files

drwxrwxr-x. 2 sahil sahil    6 Nov  5 10:40 libraries

-rw-rw-r--. 1 sahil sahil  276 Nov  5 10:40 metadata.rb

drwxrwxr-x. 2 sahil sahil    6 Nov  5 10:40 providers

-rw-rw-r--. 1 sahil sahil 1455 Nov  5 10:40 README.md

drwxrwxr-x. 2 sahil sahil   23 Nov  5 10:41 recipes

drwxrwxr-x. 2 sahil sahil    6 Nov  5 10:40 resources

drwxrwxr-x. 3 sahil sahil   20 Nov  5 10:40 templates

[sahil@cwork apache]$

Now let's describe what the different components listed above mean.

Anatomy of a cookbook:

Recipes:
The recipes directory is the place where most of the work gets done. A recipe is a ruby file contained in a cookbook. It contains all configuration settings required to bring a node to desired state. A cookbook can contain more than one recipe or depend on other outside recipes. By default, the recipes directory will contain a default.rb file which we'll use to describe the configuration we want chef to apply to the client node. Recipes contain a collection of resources & should be assigned to a run list in a node.

Resource:
A resource may represent a user, file, service etc & are written using a ruby DSL (Domain Specific Language). A resource inside a recipe is a ruby code block which defines a step that needs to be implemented by chef on the node to achieve a desired configuration state. Basically, this means that resources are declarative i.e. we say what we want to happen & not how. Chef uses the 'platform' the node is running on to determine the correct provider for a resource.

Attributes:
Attributes are used to define the desired state of the resource. An attribute can also provide details abut a specific component on the node. Attributes can be used to install packages, create users etc.

Templates:

Cookbook also contains templates for creating configuration files on the node.Templates are used to create a configuration file on the node, by adding the relevant values that are applicable to that particular node during chef agent run.

The syntax of the recipe named default defined in default.rb file is as follows:

type "name" do

    attribute "value"

    action :type_of_action

end

Here is the default.rb file from the apache cookbook we just created:

[sahil@cwork recipes]$ cat default.rb

#

# Cookbook Name:: apache

# Recipe:: default

#

# Copyright 2016, YOUR_COMPANY_NAME

#

# All rights reserved - Do Not Redistribute

#

package 'httpd' do

  action :install

end

[sahil@cwork recipes]$

So, in the above recipe we are telling chef to use 'package' resource & install httpd.

Providers:

This location will contain your custom actions and the actual code that will execute a particular action. Let's consider you have created a custom resource of your own, with few custom actions of your own. You then need to actually define the things that needs to happen during that action.

You define your actions in this directory.

Definition:

Definitions allow you can actually create a newly named resource, by using the existing list of resources. A good use case of using definitions is when you have a pattern of resources repeating over and over again in your cookbook.

Definition in itself is not a resource as such, but its a collection of resources. 

Files:

This directory contains all the files that you need to copy to the servers as part of your chef cookbook

Now, that we have created the cookbook & gone through the different aspects of a cookbook it's time to apply the cookbook.

Upload the cookbook:

We need to upload the cookbook we created to the central chef server.

[sahil@cwork recipes]$ sudo knife cookbook upload apache

Uploading apache       [0.1.0]

Uploaded 1 cookbook.

[sahil@cwork recipes]$

Add cookbook to the node's run list:

Open the web interface & click on the nodes tab. From there select the node & click on 'edit run list'

Drag & drop the recipe apache from available recipes section to current run list section.

To execute the cookbook go to the chef client node & run chef-client command.

[sahil@cclient1 ~]$ sudo chef-client

Starting Chef Client, version 12.15.19

resolving cookbooks for run list: ["apache"]

Synchronizing Cookbooks:

  - apache (0.1.0)

Installing Cookbook Gems:

Compiling Cookbooks...

Converging 1 resources

Recipe: apache::default

  * yum_package[httpd] action install

    - install version 2.4.6-40.el7.centos.4 of package httpd

Running handlers:

Running handlers complete

Chef Client finished, 1/1 resources updated in 03 minutes 16 seconds

[sahil@cclient1 ~]$

Verify that apache did get installed:

[sahil@cclient1 ~]$ rpmquery httpd

httpd-2.4.6-40.el7.centos.4.x86_64

[sahil@cclient1 ~]$

When a chef client executes a cookbook from it's run list, this process is known as 'chef run'. Let's describe a typical chef run:

Whenever the chef-client command is run, OHAI is executed & automatic attributes are collected.

These attributes are used to build node objects. Info returned by OHAI is in JSON format.

The node authenticates itself with the chef server using /etc/chef/client.pem key file.

The chef server processes the node objects which contain run lists associated with the nodes.

The run list in turn has info about roles & recipes.

Once this information is identified the chef server sends the configuration that is to be applied to the clients.

This info is shared in the form of node objects in JSON format.

The chef client then downloads the cookbooks containing the recipes which are to be applied to the node.

The recipes are stored in /var/chef/cache directory on the node.

The client would then start applying the recipes:

The chef client will first load the required libraries as they would be required by Ruby & other language extensions.

The recipes are then loaded & appropriate actions are performed to bring the node to a desired state.

If the recipes get applied successfully a message is sent to the queue stating that the run list executed successfully.

Notification handlers:
They are the final part of a chef client run & are installed as modules with the chef client. After the chef client run has been completed, we can write a notification handler to take an appropriate action in case the chef run fails.
For example, a notification handler to email the stack trace of a failed chef run to the chef admin.
Notification handlers can also be used for reporting purposes since it contains all the data corresponding to a chef client run.

Summarizing the chef run:

1. When chef client runs, first it executes OHAI which gathers all the data about the node (attributes)
2. The client is authenticated.
3. The cookbooks are synchronized & loaded.
4. Then the cookbooks are converged (recipes are applied) on policies.
5. Success/failure status is checked.
6. The result is communicated to chef server server.

Chef node authentication:

When knife bootstraps a node it transfers the organization validation key (validation.pem) file to the node. The node then connects to the chef server & using the validation.pem file it requests a API client from the server. The server in turn provides the client.pem file which the node uses in future to communicate with the node & sign requests. In a scenario where the client.pem file is not available for some reason the above mentioned process is repeated. If both client.pem & validation.pem are not available then the node is not able to authenticate with the chef server & we get a 401 error.

Breaking down the converge phase:

In the policy converge phase the chef client checks whether it's conforming to the policies as laid out to it by the chef server & the cookbooks it just downloaded. If it conforms then the idempotence property comes into play & no action is taken. Otherwise the recipes are applied appropriately.
The converge phase has two parts:

Compile phase: The cookbooks are loaded & each recipe is read & the resources from the recipes are collected in a resource collection list.

Execute phase: In this phase the actions defined in the recipes are executed for the corresponding resources.

Add cookbook to the node's run list without using web interface:

Earlier I described how we could add a cookbook to a node's run list from the GUI interface. Lets' now do it without a GUI.

I edited the apache cookbook created & added steps to start apache after install:

[sahil@cwork recipes]$ cat default.rb

#

# Cookbook Name:: apache

# Recipe:: default

#

# Copyright 2016, YOUR_COMPANY_NAME

#

# All rights reserved - Do Not Redistribute

#

package 'httpd' do

  action :install

end

service 'httpd' do

  action :start

end

I then uploaded the cookbook to the server again.

[sahil@cwork recipes]$ sudo knife cookbook upload apache

Uploading apache         [0.1.0]

Uploaded 1 cookbook.

Then I updated the node's run list using the knife utility from within the chef workstation:

[sahil@cwork recipes]$ sudo  knife node run_list add mychefnode "recipe[apache]"

mychefnode:

  run_list: recipe[apache]

Then I logged into the client to apply the cookbook via chef-client:

[sahil@cclient1 ~]$ sudo chef-client

Starting Chef Client, version 12.15.19

resolving cookbooks for run list: ["apache"]

Synchronizing Cookbooks:

  - apache (0.1.0)

Installing Cookbook Gems:

Compiling Cookbooks...

Converging 1 resources

Recipe: apache::default

  * yum_package[httpd] action install (up to date)

Running handlers:

Running handlers complete

Chef Client finished, 0/1 resources updated in 19 seconds

[sahil@cclient1 ~]$ sudo chef-client

Starting Chef Client, version 12.15.19

resolving cookbooks for run list: ["apache"]

Synchronizing Cookbooks:

  - apache (0.1.0)

Installing Cookbook Gems:

Compiling Cookbooks...

Converging 2 resources

Recipe: apache::default

  * yum_package[httpd] action install (up to date)

  * service[httpd] action start

    - start service service[httpd]

Running handlers:

Running handlers complete

Chef Client finished, 1/2 resources updated in 02 seconds

Note: while adding the recipe if we use the name of the cookbook, the default.rb file is used. 

Adding another recipe from a cookbook to a node's run list:

So, under the apache cookbook, I created another recipe called addscreen:

[sahil@cwork recipes]$ cat addscreen.rb

#

# Cookbook Name:: apache

# Recipe:: addscreen

#

# Copyright 2016, YOUR_COMPANY_NAME

#

# All rights reserved - Do Not Redistribute

#

package 'screen' do

  action :install

end

Now we need to upload the cookbook to the server again:

[sahil@cwork recipes]$ sudo knife cookbook upload apache

Uploading apache         [0.1.0]

Uploaded 1 cookbook.

Add the cookbook to to the node's run list:

[sahil@cwork recipes]$ sudo  knife node run_list add mychefnode "recipe[apache::addscreen]"

mychefnode:

  run_list:

    recipe[apache]

    recipe[apache::addscreen]

To view the current run list & some more information about the client node, run:

[sahil@cwork chef-repo]$ sudo knife node show mychefnode
Node Name:   mychefnode
Environment: _default
FQDN:        cclient1
IP:          192.168.44.102
Run List:    recipe[apache], recipe[apache::addscreen]
Roles:
Recipes:     apache, apache::default, apache::addscreen
Platform:    centos 7.0.1406
Tags:
[sahil@cwork chef-repo]$

To apply the recipe login to the node & type chef-client:

[sahil@cclient1 ~]$ sudo chef-client

Starting Chef Client, version 12.15.19

resolving cookbooks for run list: ["apache", "apache::addscreen"]

Synchronizing Cookbooks:

  - apache (0.1.0)

Installing Cookbook Gems:

Compiling Cookbooks...

Converging 3 resources

Recipe: apache::default

  * yum_package[httpd] action install (up to date)

  * service[httpd] action start (up to date)

Recipe: apache::addscreen

  * yum_package[screen] action install

    - install version 4.1.0-0.23.20120314git3c2946.el7_2 of package screen

Running handlers:

Running handlers complete

Chef Client finished, 1/3 resources updated in 26 seconds

Delete a recipe from a node's run list:

[sahil@cwork recipes]$ sudo  knife node run_list remove  mychefnode "recipe[addscreen]"

mychefnode:

  run_list:

    recipe[apache]

    recipe[apache::addscreen]

Changing the order of recipes in a run list:

Run the chef node show command to view the order of the chef run:

[sahil@cwork recipes]$ sudo knife node show mychefnode
Node Name:   mychefnode
Environment: _default
FQDN:        cclient1
IP:          192.168.44.102
Run List:    recipe[apache], recipe[apache::addscreen]
Roles:
Recipes:     apache, apache::default, apache::addscreen
Platform:    centos 7.0.1406
Tags:

Now to set/change the order use the knife run_list set command as shown below:

[sahil@cwork recipes]$ sudo  knife node run_list set mychefnode "recipe[apache::addscreen]","recipe[apache]"

mychefnode:

  run_list:

    recipe[apache::addscreen]

    recipe[apache]

Run knife node show again to check the current chef run order:

[sahil@cwork recipes]$ sudo knife node show mychefnode

Node Name:   mychefnode

Environment: _default

FQDN:        cclient1

IP:          192.168.44.102

Run List:    recipe[apache::addscreen], recipe[apache]

Roles:

Recipes:     apache, apache::default, apache::addscreen

Platform:    centos 7.0.1406

Tags:

I bascally told chef client to apply the addscreen recipe before the default recipe.