week1 – Overview of Chef.pdf
Tag: Chef Guide
Logging in Chef Explained
Chef Server
All logs generated by the Chef server can be found in /var/log/opscode. Each service enabled on the system also has a sub-directory in which service-specific logs are located, typically found in /var/log/opscode/service_name.
The Chef server has built-in support for easily tailing the logs that are generated. To view all the logs being generated on the Chef server, enter the following command:
> chef-server-ctl tail
To view logs for a specific service:
> chef-server-ctl tail SERVICENAME
where SERVICENAME should be replaced with name of the service for which log files will be viewed. SERVICENAME represents the name of any service that is listed after running the “> chef-server-ctl service-list” subcommand.
Another way to view log files is to use the system utility tail:
> tail -50f /var/log/chef-server/opscode-chef/current
> tail -50f /var/log/opscode/opscode-chef/current
Supervisor Logs
Supervisor logs are created and managed directly by the service supervisor, and are automatically rotated when a the current log file reaches 1,000,000 bytes. 10 log files are kept. The latest supervisor log is always located in /var/log/chef-server/service_name/current and rotated logs have a filename starting with @ followed by a precise tai64n timestamp based on when the file was rotated.
Supervisor logs are available for the following services:
- bifrost
- bookshelf
- nginx
- opscode-erchef
- opscode-expander
- opscode-expander-reindexer
- opscode-solr4
- postgresql
- rabbitmq
- redis
nginx
The nginx service creates both supervisor and administrator logs. The administrator logs contain both access and error logs for each virtual host utilized by the Chef server. Each of the following logs require external log rotation.
| Logs | Description |
|---|---|
| /var/log/opscode/nginx/access.log | The Web UI and API HTTP access logs. |
| /var/log/opscode/nginx/error.log | The Web UI and API HTTP error logs. |
| /var/log/opscode/nginx/internal-account.access.log | The opscode-account internal load-balancer access logs. |
| /var/log/opscode/nginx/internal-account.error.log | The opscode-account internal load-balancer error logs. |
| /var/log/opscode/nginx/internal-authz.access.log | The opscode-authz internal load-balancer access logs. |
| /var/log/opscode/nginx/internal-authz.error.log | The opscode-authz internal load-balancer error logs. |
| /var/log/opscode/nginx/internal-chef.access.log | The opscode-chef and opscode-erchef internal load-balancer access logs. |
| /var/log/opscode/nginx/internal-chef.error.log | The opscode-chef and opscode-erchef internal load-balancer error logs. |
| /var/log/opscode/nginx/nagios.access.log | The nagios access logs. |
| /var/log/opscode/nginx/nagios.error.log | The nagios error logs. |
| /var/log/opscode/nginx/rewrite-port-80.log | The rewrite logs for traffic that uses HTTP instead of HTTPS. |
To follow the logs for the service:
$ chef-server-ctl tail nginx
Chef Client
Client.rb3 file might help you. Default value of log_location is STDOUT. You can give /path/to/log_location in place of this. You can locate this client.rb file in C:\chef\client.rb or /etc/chef/client.rb directories.
Use the verbose logging that is built into the chef-client:
-l LEVEL, –log_level LEVEL
The level of logging to be stored in a log file.
-L LOGLOCATION, –logfile c
The location of the log file. This is recommended when starting any executable as a daemon. Default value: STDOUT.
Knife
Use the verbose logging that is built into knife:
-V, –verbose
Set for more verbose outputs. Use -VV for maximum verbosity.
chef-solo
-l LEVEL, –log_level LEVEL
The level of logging to be stored in a log file.
The Chef file and folder locations are different on Linux and Windows machines. This article explains the purpose of each file and the location.
Summary
| Linux | Windows | |
|---|---|---|
| Cookbook location | /var/chef/cache/cookbooks | C:\chef\cache\cookbooks |
| Chef Client run log | /var/log/chef.log | First run only C:\chef\chef-client.log |
| Subsequent Chef client runs C:\chef\log\client.log |
||
| Error log | /var/chef/cache/chef-stacktrace.out | C:\chef\cache\chef-stacktrace.out |
| Ohai output | /var/chef/cache/failed-run-data.json | C:\chef\cache\failed-run-data.json |
| Recommended location for custom log files | /tmp/cheflog.log | C:\Logs\Chef\cheflog.log |
| Chef Client configuration | /etc/chef/client.rb | C:\chef\client.rb |
When you test your cookbook in Test Kitchen
The .kitchen.yml file contains the username to execute the Chef cookbook. It is specified under platforms:, transport:, username:
Use that value in place of USER-NAME-FROM-KITCHEN-YML below.
| Linux | Windows | |
|---|---|---|
| Cookbook location | /tmp/kitchen/cookbooks /tmp/kitchen/cache/cookbooks |
C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cookbooks |
| Error log | /tmp/kitchen/cache/chef-stacktrace.out | C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cache\chef-stacktrace.out |
| Ohai output | /tmp/kitchen/cache/failed-run-data.json | C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\cache\failed-run-data.json |
| Data bags | /tmp/kitchen/data_bags | C:\Users\USER-NAME-FROM-KITCHEN-YML\AppData\Local\Temp\kitchen\data_bags |
Cookbook location
When the Chef recipes are executed, all cookbooks are stored on the node. You can examine the code to make sure your latest changes are reflected on the machine.
The log of the Chef client run
The output of the Chef cookbook execution is in the chef.log or chef-client.log file
On Windows
The log of the first Chef Client run and subsequent runs are stored in different log files. After the initial Chef Client run, the rest of the log entries are collected in the second file.
Stacktrace
Chef saves information on the hard drive when scripts are executed. If there is a failure, the stack trace of the last error is saved in the chef-stacktrace.out file.
Ohai output
All the information that Ohai collects on the instance, is saved in the failed-run-data.jsonfile, even if there is no error. It is a great resource to get the server specific values.
Reference
https://docs.chef.io/server_logs.html
https://docs.chef.io/debug.html
Chef notifies and subscribes explained with examples
Chef notifies and subscribes explained with examples
A notification is a property on a resource that listens to other resources in the resource collection and then takes actions based on the notification type (notifies or subscribes).
Timers
A timer specifies the point during the chef-client run at which a notification is run. The following timers are available:
:before
Specifies that the action on a notified resource should be run before processing the resource block in which the notification is located.
:delayed
Default. Specifies that a notification should be queued up, and then executed at the very end of the chef-client run.
:immediate, :immediately
Specifies that a notification should be run immediately, per resource notified.
Notifies
A resource may notify another resource to take action when its state changes. Specify a ‘resource[name]’, the :action that resource should take, and then the :timer for that action. A resource may notify more than one resource; use a notifies statement for each resource to be notified.
The syntax for notifies is:
notifies :action, ‘resource[name]’, :timer
Example
The following examples show how to use the notifies notification in a recipe.
Delay notifications
template ‘/etc/nagios3/configures-nagios.conf’ do
# other parameters
notifies :run, ‘execute[test-nagios-config]’, :delayed
end
Notify immediately
By default, notifications are :delayed, that is they are queued up as they are triggered, and then executed at the very end of a chef-client run. To run an action immediately, use :immediately:
template ‘/etc/nagios3/configures-nagios.conf’ do
# other parameters
notifies :run, ‘execute[test-nagios-config]’, :immediately
end
and then the chef-client would immediately run the following:
execute ‘test-nagios-config’ do
command ‘nagios3 –verify-config’
action :nothing
end
Subscribes
A resource may listen to another resource, and then take action if the state of the resource being listened to changes. Specify a ‘resource[name]’, the :action to be taken, and then the :timer for that action.
Note that subscribes does not apply the specified action to the resource that it listens to – for example:
file ‘/etc/nginx/ssl/example.crt’ do
mode ‘0600’
owner ‘root’
end
service ‘nginx’ do
subscribes :reload, ‘file[/etc/nginx/ssl/example.crt]’, :immediately
end
In this case the subscribes property reloads the nginx service whenever its certificate file, located under /etc/nginx/ssl/example.crt, is updated. subscribes does not make any changes to the certificate file itself, it merely listens for a change to the file, and executes the :reload action for its resource (in this example nginx) when a change is detected.
The syntax for subscribes is:
subscribes :action, ‘resource[name]’, :timer
Examples
The following examples show how to use the subscribes notification in a recipe.
Prevent restart and reconfigure if configuration is broken
Use the :nothing action (common to all resources) to prevent the test from starting automatically, and then use the subscribes notification to run a configuration test when a change to the template is detected:
execute ‘test-nagios-config’ do
command ‘nagios3 –verify-config’
action :nothing
subscribes :run, ‘template[/etc/nagios3/configures-nagios.conf]’, :immediately
end
Reference
Example
What is the significance of the default directory under chef cookbook /templates?
What is the significance of the default directory under chef cookbook /templates?
A cookbook is frequently designed to work across many platforms and is often required to distribute a specific template to a specific platform. This is a New in Chef Client 12.0. A cookbook can be designed to support the distribution of templates across platforms, while ensuring that the correct template ends up on each system.
The pattern for template specificity depends on two things: the lookup path and the source. The first pattern that matches is used:
/host-$fqdn/$source
/$platform-$platform_version/$source
/$platform/$source
/default/$source
/$source
Use an array with the source property to define an explicit lookup path. For example:
template ‘/test’ do
source [“#{node.chef_environment}.erb”, ‘default.erb’]
end
The following example emulates the entire file specificity pattern by defining it as an explicit path:
template ‘/test’ do
source %W{
host-#{node[‘fqdn’]}/test.erb
#{node[‘platform’]}-#{node[‘platform_version’]}/test.erb
#{node[‘platform’]}/test.erb
default/test.erb
}
end
A cookbook may have a /templates directory structure like this:
/templates/
windows-6.2
windows-6.1
windows-6.0
windows
default
and a resource that looks something like the following:
template ‘C:\path\to\file\text_file.txt’ do
source ‘text_file.txt’
mode ‘0755’
owner ‘root’
group ‘root’
end
This resource would be matched in the same order as the /templates directory structure. For a node named host-node-desktop that is running Windows 7, the second item would be the matching item and the location:
/templates
windows-6.2/text_file.txt
windows-6.1/text_file.txt
windows-6.0/text_file.txt
windows/text_file.txt
default/text_file.txt
How to Implement Chef roles using Chef server ?
What is Role?
A role is a way to define certain patterns and processes that exist across nodes in an organization as belonging to a single job function. Each role consists of zero (or more) attributes and a run-list. Each node can have zero (or more) roles assigned to it. When a role is run against a node, the configuration details of that node are compared against the attributes of the role, and then the contents of that role’s run-list are applied to the node’s configuration details. When a chef-client runs, it merges its own attributes and run-lists with those contained within each assigned role.
How to use Roles in Chef?
- Create a Role and add the cookbooks into it.
- Assign the role into each node or bootstrap new nodes using roles
- The the run list
How to create Role?
Method 1: In Chef Server directly
> knife role create client1&
Add the run list e.g. “recipe[nginx]” under “run_list”
Save & exit
The role will be created in Chef Server.
Example
{ "name": "client1", "description": "", "json_class": "Chef::Role", "default_attributes": { }, "override_attributes": { }, "chef_type": "role", "run_list": [ "recipe[nginx]", "recipe[phpapp::web]" ], "env_run_lists": { } } Let’s download the role from the Chef server so we have it locally in a Chef repository.
> knife role show client1 -d -Fjson > roles/client1.json
Now, Lets bootstrap the node using knife with roles
> knife bootstrap --run-list "role[client1]" --sudo hostname
How to edit the roles in chef Server?
> knife role edit client1
Method 2: In local repo under chef-repo folder
> vi webserver.rbexample –
name "web_server" description "Role for web servers" run_list( "role[base]", "recipe{web_server]" ) & Then upload to chef server using following commands.
> knife role from file path/to/role/file
How Assigning Roles to Nodes?
> knife node list> knife node edit node_nameThis will bring up the node’s definition file, which will allow us to add a role to its run_list:
{ "name": "client1", "chef_environment": "_default", "normal": { "tags": [ ] }, "run_list": [ "recipe[nginx]" ] } For instance, we can replace our recipe with our role in this file:
{ "name": "client1", "chef_environment": "_default", "normal": { "tags": [ ] }, "run_list": [ "role[web_server]" ] } How to bootstrap node using role?
> knife bootstrap {{address}} --ssh-user {{user}} --ssh-password '{{password}}' --sudo --use-sudo-password --node-name node1 --run-list 'role[production]'> knife bootstrap --run-list "role[phpapp-web]" --sudo hostnameHow to run roles against nodes?
You can run chef-client on multiple nodes via knife ssh command like, To query for all nodes that have the webserver role and then use SSH to run the command sudo chef-client, enter:
> knife ssh "role:webserver" "sudo chef-client"To find the uptime of all of web servers running Ubuntu on the Amazon EC2 platform, enter:
> knife ssh "role:web" "uptime" -x ubuntu -a ec2.public_hostnameReference
http://docs.chef.io/roles.html
https://docs.chef.io/knife_ssh.html
https://docs.chef.io/knife_role.html
Chef – Documenting Cookbooks automatically
Problems Area –
Our infrastructure has many cookbooks that aim to be reusable, primarily through encapsulating behaviour in LWRPs. This led to an explosion of LWRPs and sometimes the documentation didn’t keep up or did just not exist.
Solution 1: Follow Best Practices
This command will create a README.rdoc by default, and I prefer Markdown, so I specify the -r md option.
> knife cookbook create smartmontools -r md
Solution 2: Automate using plugins
README.md which should be generated automatcially must contain information about the recipes, attributes, platform compatibility and cookbook requirements (i.e. depends, recommends, suggests etc).
Mathias Lafeldt wrote a knife plugin that generates an initial README.md from the metadata.rb file in a a cookbook.
Link –
Extension of Mathias Lafeldt was done here with regenerate README.md from the cookbook source code
Please refer these useful blogs as well which is referenced in order to write this article…
Know about metadata.rb and metadata.json in Chef
What is Metadata in chef?
In order to understand cookbooks without evaluating them every files, we generate metadata about cookbooks
What is metadata.rb?
Before unerstanding about metadata.json, we need to understand about metadata.rb.
Every cookbook requires a small amount of metadata. Metadata is stored in a file called metadata.rb that lives at the top of each cookbook’s directory. The contents of the metadata.rb file provides hints to the Chef server so that cookbooks are deployed to each node correctly.
A metadata.rb file is never interpreted directly. Rather, the metadata.rb file provides a simple location to store and edit data that is then compiled by the Chef server and stored as JSON data.
Metadata is compiled whenever ….
1. a cookbook is uploaded to the Chef server or
2. when the knife cookbook metadata subcommand is run. knife creates a metadata.rb file automatically whenever the knife cookbook create subcommand is run.
metadata.json
metadata.json is generated from metadata.rb. to generate or regenerate the Chef cookbook metadata, which creates a new ‘metadata.json’ file from the ‘metadata.rb’ file.
From the command line of your Chef development server, enter the following command:
# knife cookbook metadata my-cookbook -o <COOKBOOK_PATH>
Example: # knife cookbook metadata my-cookbook-name -o /opt/development/cookbooks/
A metadata.json file can be edited directly, should temporary changes be required. Any subsequent upload or action that generates metadata will cause the existing metadata.json file to be overwritten with the newly generated metadata. Therefore, any permanent change to metadata that is required should only be made in the metadata.rb file.
What if metadata.rb and metadata.json both are present in cookbooks?
If a cookbook contains both a metadata.rb and metadata.json at cookbook load time, the metadata.rb will be preferred. This should be swapped around to ensure that dynamically built metadata which has been compiled is read properly without error.
knife cookbook upload reads metadata.rb and ignores metadata.json. It generates the data from the .rb file, and uploads that. It does not create a metadata.json file, on disk or in the cookbook itself–this is just metadata in the cookbook object.