Exploring "Infrastructure as code" with Opscode Chef Part 2 (lab setup)

In the lab setup I've used chef version 12 & Cent OS 7 for the server, workstation & client components. I've tried to configure everything following the path of least resistance i.e the easiest way I could think of.

The system details:

192.168.44.100  cserver  => central chef server

192.168.44.101  cwork => chef workstation

192.168.44.102  cclient1 => chef client node.

Chef admin user "sahil" is created on all machines with sudo access. This is the user we'll be using to manage chef.

Installing chef server:

Download & install the chef server core rpm:

[root@cserver ~]# rpm -ivh chef-server-core-12.10.0-1.el7.x86_64.rpm

warning: chef-server-core-12.10.0-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:chef-server-core-12.10.0-1.el7   ################################# [100%]

Next run "chef-server-ctl reconfigure" to configure chef server related services ie: Nginx, RabbitMQ, Postgresql, Solr & much more.

[root@cserver ~]# chef-server-ctl reconfigure

----------------

    - /var/opt/opscode/local-mode-cache/cookbooks/private-chef/recipes/redis_lb.

  Cloning resource attributes for directory[/var/opt/opscode/nginx/etc/addon.d]

Previous directory[/var/opt/opscode/nginx/etc/addon.d]: /var/opt/opscode/local-m

Current  directory[/var/opt/opscode/nginx/etc/addon.d]: /var/opt/opscode/local-m

    - /var/opt/opscode/local-mode-cache/cookbooks/private-chef/recipes/nginx.rb:

Chef Client finished, 394/468 resources updated in 04 minutes 14 seconds

Chef Server Reconfigured!

This will take a while to complete as it does a lot of work.

Once finished, we can check the status of the services:

[root@cserver ~]# chef-server-ctl status

run: bookshelf: (pid 5907) 317s; run: log: (pid 5938) 316s

run: nginx: (pid 5790) 321s; run: log: (pid 6123) 310s

run: oc_bifrost: (pid 5728) 323s; run: log: (pid 5763) 322s

run: oc_id: (pid 5778) 321s; run: log: (pid 5783) 321s

run: opscode-erchef: (pid 6005) 314s; run: log: (pid 5982) 315s

run: opscode-expander: (pid 5867) 317s; run: log: (pid 5882) 317s

run: opscode-solr4: (pid 5807) 319s; run: log: (pid 5814) 319s

run: postgresql: (pid 5698) 324s; run: log: (pid 5711) 323s

run: rabbitmq: (pid 5611) 325s; run: log: (pid 5604) 325s

run: redis_lb: (pid 5223) 451s; run: log: (pid 6119) 311s

[root@cserver ~]#

Next, we move to installing the web interface (chef manage)

[root@cserver ~]# chef-server-ctl install chef-manage

OR

[root@cserver ~]# rpm -ivh chef-manage-2.4.4-1.el7.x86_64.rpm

warning: chef-manage-2.4.4-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:chef-manage-2.4.4-1.el7          ################################# [100%]

Thank you for installing the Chef Management Console add-on!

Next, we need to let chef configure the web interface provided by chef-manage:

[root@cserver ~]# chef-manage-ctl reconfigure

To use this software, you must agree to the terms of the software license agreement.

Press any key to continue.

Type 'yes' to accept the software license agreement, or anything else to cancel.

yes

--------------------------------------------------------------------------

Running handlers:

Running handlers complete

Chef Client finished, 88/188 resources updated in 63.818536296 seconds

chef-manage Reconfigured!

[root@cserver ~]#

This step involves accepting the license agreement. As I mentioned earlier the web interface is free for up to 25 nodes only & you would incur a subscription free for more than 25 nodes.

Now we create the user & organization:

 sudo chef-server-ctl user-create sahil sahil suri sahil@example.com 123456

In the above command:

sahil: user name

sahil suri: full name of the user

sahil@example.com: the user's email address

123456: the user's password.

sudo chef-server-ctl org-create testorg "TestOrg.Ltd" --association_user sahil

In the above command:

testorg: organization short name.

TestOrg.Ltd: organization long name.

sahil: the admin user for chef

you can optionally create an organization from the web interface as well.

Now that the chef server components & users are setup, let's open up a browser to https://<chef server IP> which in this case is https://192.168.44.100

Installing & configuring chef workstation:

We'll download & install the chef development kit rpm.

[sahil@cwork ~]$ sudo rpm -ivh chefdk-0.19.6-1.el7.x86_64.rpm

warning: chefdk-0.19.6-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY

Preparing...                          ################################# [100%]

Updating / installing...

   1:chefdk-0.19.6-1.el7              ################################# [100%]

Thank you for installing Chef Development Kit!

The development kit installs a suite of software required by the chef workstation most importantly the chef knife utility.

We can run 'chef verify' to confirm that required components have been successfully installed.

[sahil@cwork ~]$ sudo chef verify

Running verification for component 'berkshelf'

Running verification for component 'test-kitchen'

Running verification for component 'tk-policyfile-provisioner'

Running verification for component 'chef-client'

Running verification for component 'chef-dk'

Running verification for component 'chef-provisioning'

Running verification for component 'chefspec'

----------------------------------------------------------

If your workstation will primarily be used to manage Chef for your infrastructure, you will likely want to default to the version of Ruby installed with Chef. You can do this by modifying your .bash_profile so that Chef's Ruby takes precedence:

echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile

Next we go the chef server web interface to get the 'chef starter kit'.

To do this, open up the web interface & go to administration tab. From there under the organization section click on starter kit. Then just click on download starter kit & copy it to the chef workstation.

Chef starter kit is an archive that contains the directory structure for the chef-repo repository as well as knife utility & cookbook settings. 

I've copied the chef starter kit to the home directory of the admin user i.e sahil. Now let's extract it.

[sahil@cwork ~]$ unzip chef-starter.zip

Archive:  chef-starter.zip

  inflating: chef-repo/README.md

   creating: chef-repo/cookbooks/

  inflating: chef-repo/cookbooks/chefignore

   creating: chef-repo/cookbooks/starter/

   creating: chef-repo/cookbooks/starter/attributes/

  inflating: chef-repo/cookbooks/starter/attributes/default.rb

   creating: chef-repo/cookbooks/starter/files/

   creating: chef-repo/cookbooks/starter/files/default/

  inflating: chef-repo/cookbooks/starter/files/default/sample.txt

  inflating: chef-repo/cookbooks/starter/metadata.rb

   creating: chef-repo/cookbooks/starter/recipes/

  inflating: chef-repo/cookbooks/starter/recipes/default.rb

   creating: chef-repo/cookbooks/starter/templates/

   creating: chef-repo/cookbooks/starter/templates/default/

  inflating: chef-repo/cookbooks/starter/templates/default/sample.erb

  inflating: chef-repo/.gitignore

   creating: chef-repo/.chef/

   creating: chef-repo/roles/

  inflating: chef-repo/.chef/knife.rb

  inflating: chef-repo/roles/starter.rb

  inflating: chef-repo/.chef/sahil.pem

Downloading authentication keys:

The chef workstation requires 2 authentication keys to interact with the chef server. The first is the adminuser.pem file which we got from the chef starter kit & is located as /home/sahil/chef-repo/.chef/sahil.pem. The second key is the organization validation key.

To get this, open up the web interface & go to administration tab. From there under the organization section click on 'reset validation key'. This will bring up a new key. Click on download to download the key.

Copy the downloaded key testorg-validator.pem to the chef workstation under /home/sahil/chef-repo/.chef/

Configuring knife:

The working on knife command can be modified by editing the knife.rb file found in ~/chef-repo/.chef/knife.rb

[sahil@cwork .chef]$ cat knife.rb

# See https://docs.getchef.com/config_rb_knife.html for more information on knife configuration options

current_dir = File.dirname(__FILE__)

log_level                :info

log_location             STDOUT

node_name                "sahil"

client_key               "#{current_dir}/sahil.pem"

validation_client_name   "testorg-validator"

validation_key           "#{current_dir}/testorg-validator.pem"

chef_server_url          "https://cserver/organizations/testorg"

cookbook_path            ["#{current_dir}/../cookbooks"]

[sahil@cwork .chef]$

Here is a description of the different parameters:

·  node_name: This specifies the name that knife will use to connect to your Chef server. This should match your user name.

·  client_key: This should be the name and path to the user key that you copied over from the Chef server. We can use the #{current_dir} snippet to fill in the path if the key is in the same directory as the knife.rb file.

·   validation_client_name: This is the name of the validation client that knife will use to bootstrap new nodes. This will take the form of your organization short name, followed by -validator.

·  validation_key: Like the client_key, this includes the name and path to the validation key you copied from the Chef server. Again, you can use the #{current_dir} Ruby snippet to specify the current directory if the validation key is in the same directory as the knife.rb file.

·  chef_server_url: This is the URL where the Chef server can be reached. It should begin with https://, followed by your Chef server's domain name or IP address. Afterwards, the path to your organization should be specified by appending /organizations/your_organization_name.

h Now we need to obtain the chef servers' SSL certificates. to do this type:

[sahil@cwork .chef]$ sudo knife ssl fetch

WARNING: Certificates from cserver will be fetched and placed in your trusted_cert

directory (/home/sahil/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should

verify the authenticity of these certificates after downloading.

Adding certificate for cserver in /home/sahil/chef-repo/.chef/trusted_certs/cserver.crt

[sahil@cwork .chef]$

We can use the following command to check if our chef workstation has been configured correctly:

[sahil@cwork .chef]$ sudo knife client list

testorg-validator

[sahil@cwork .chef]$

 Bootstraping a client:

  

      Bootstrapping is the process of installing & configuring a chef agent on a client node so that it can be managed by chef. We do a bootstrap using knife which will communicate with the chef server API, create a logical client instance for the node.This client instance will authenticate the client node with the chef server via ssh keys & is responsible for communicating with the chef client API on the node

      Chef knife will then connect to client node via ssh & pass to the node the chef server URL, validation client name & the validation key. This information is maintained in knife.rb file. Once this payload has been sent to the client node it then downloads & install the required chef client version & will register the node with the chef server & finally run chef client.

      Type the following command to bootstrap a client node:

      knife bootstrap <nodename> -x <user name> -P <password> -N "Node" --sudo 

[sahil@cwork .chef]$ sudo knife bootstrap cclient1 -N mychefnode -x sahil -P 123456 --sudo

Doing old-style registration with the validation key at /home/sahil/chef-repo/.chef/testorg-validator.pem...

Delete your validation key in order to use your user credentials instead

Connecting to cclient1

cclient1 -----> Installing Chef Omnibus (-v 12)

cclient1 downloading https://omnitruck-direct.chef.io/chef/install.sh

cclient1   to file /tmp/install.sh.24640/install.sh

cclient1 trying wget...

cclient1 el 7 x86_64

cclient1 Getting information for chef stable 12 for el...

cclient1 downloading https://omnitruck-direct.chef.io/stable/chef/metadata?v=12&p=el&pv=7&m=x86_64

cclient1   to file /tmp/install.sh.24645/metadata.txt

cclient1 trying wget...

cclient1 sha1   cd53d4f6431e912d0ddafce3cabd113fec3dda05

cclient1 sha256 66a514699bddd3f507a3b80b687cd71079a09e2aa01fa8111bee7f06a95c44c7

cclient1 url    https://packages.chef.io/files/stable/chef/12.15.19/el/7/chef-12.15.19-1.el7.x86_64.rpm

cclient1 version        12.15.19

cclient1 downloaded metadata file looks valid...

cclient1 downloading https://packages.chef.io/files/stable/chef/12.15.19/el/7/chef-12.15.19-1.el7.x86_64.rpm

cclient1   to file /tmp/install.sh.24645/chef-12.15.19-1.el7.x86_64.rpm

cclient1 trying wget...

cclient1 trying curl...

cclient1 Comparing checksum with sha256sum...

cclient1 Installing chef 12

cclient1 installing with rpm...

cclient1 warning: /tmp/install.sh.24645/chef-12.15.19-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY

cclient1 Preparing...                          ################################# [100%]

cclient1 Updating / installing...

cclient1    1:chef-12.15.19-1.el7              ################################# [100%]

cclient1 Thank you for installing Chef!

cclient1 Starting the first Chef Client run...

cclient1 Starting Chef Client, version 12.15.19

cclient1 Creating a new client identity for mychefnode using the validator key.

cclient1 resolving cookbooks for run list: []

cclient1 Synchronizing Cookbooks:

cclient1 Installing Cookbook Gems:

cclient1 Compiling Cookbooks...

cclient1 [2016-11-05T10:24:21-04:00] WARN: Node mychefnode has an empty run list.

cclient1 Converging 0 resources

cclient1

cclient1 Running handlers:

cclient1 Running handlers complete

cclient1 Chef Client finished, 0/0 resources updated in 06 seconds

To verify that client has been bootstraped successfully:

[sahil@cwork .chef]$ sudo knife node list

mychefnode

[sahil@cwork .chef]$ sudo knife client list

mychefnode

testorg-validator

[sahil@cwork .chef]$

On the client, we can verify that chef client has been installed:

[root@cclient1 ~]# rpmquery chef

chef-12.15.19-1.el7.x86_64

[root@cclient1 ~]#

With the client now part of our chef managed infrastructure, we'll write our first cookbook in the next article.