Chef Validatorless Bootstrap Process Explanation With Source

I believe there can be hundreds or thousands of articles over the internet explaining the bootstrap process of chef, but after reading them, I still have doubt about that, so I decide to look into it in the code level.

From what I learned from the source, basically there are three basic steps involving the chef bootstrap:

  • Create client and node on the chef server

  • Render bootstrap template with the configuration you specify

  • Run the boostrap script/template on the node via ssh

If we are looking from the source code, that is run method of Chef::Knife::Bootstrap class

# use ellipsis to skip line out of scope for simplicity
def run

    # ......

    # Step 1
    client_builder.run

    # .....

    # Step 2
    bootstrap_context.client_pem = client_builder.client_path
    # ......

  begin
    # Step 3
    knife_ssh.run
  rescue Net::SSH::AuthenticationFailed
    # ......
  end
end

Let’s break down a little bit

STEP 1: ClientBuilder

Basically, ClientBuilder is the class to create client/node object for our machine on the chef server side

And, chef_config is the config from you knife configuration file, like #{Dir.home}/.chef/knife.rb, knife_config is the config from the command line input

@client_builder = Chef::Knife::Bootstrap::ClientBuilder.new(
          chef_config: Chef::Config,
          knife_config: config,
          ui: ui
        )

If we look in the run method of ClientBuilder, it’s basically doing two works: create_client! and create_node!

def run
  sanity_check

  ui.info("Creating new client for #{node_name}")

  @client = create_client!

  ui.info("Creating new node for #{node_name}")

  create_node!
end

create_client!

Let’s look at what create_client! do. It simply run Chef::ApiClient::Registration class, as:

# Create the client object and save it to the Chef API
def create_client!
  Chef::ApiClient::Registration.new(node_name, client_path, http_api: rest).run
end

Here, it’s the comment from the author of the source code

# ==Chef::ApiClient::Registration
# Manages the process of creating or updating a Chef::ApiClient on the
# server and writing the resulting private key to disk. Registration uses
# the validator credentials for its API calls. This allows it to bootstrap
# a new client/node identity by borrowing the validator client identity
# when creating a new client.

To check out more closer:

def run
  # to make sure the resulting private key destination path is writable
  # the path is temp dir, something like, @tmpdir = Dir.mktmpdir
  # File.join(@tmpdir, "#{node_name}.pem")
  # One example is "/var/folders/2t/3qdzts6n75z981qp15dvzqc40000gn/T/d20181209-1663-7yn4es/one_node.pem"
  assert_destination_writable!
	# ....
  begin
    client = api_client(create_or_update)
  rescue Net::HTTPFatalError => e
		# .....
  end
  # writing down the private key to destination path as above
  write_key
  # return client object
  client
end

According to the source below, create_or_update method will try to create the client firstly, if that fails, then it will try to update the client.

def create_or_update
  create
rescue Net::HTTPServerException => e
  # If create fails because the client exists, attempt to update. This
  # requires admin privileges.
  raise unless e.response.code == "409"
  update
end

Let’s take client create for example, it’s lengthy to get into the details to about create_client, it would be easier to understand what’s happening behind the scenes.

To create a client, knife bootstrap command will post json data to the chef server, for instance https://localhost/clients:

{"name":"one_node","admin":false,"public_key":"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsMGJmQfMI04E2U1RDJgB\n0mb5zlDDXAOwxza9r4FvHy/0/enY9soea6MmDTaTRDawpuLumqlZefYWLrIc3u45\ns6d+DFPnl0bYuQMUsANoGWgQsx7uqoqRI1dT/62YFbrLLLX+N8HXHHI2cElZOWwW\nL6xP2+RCLq2XuL/w06TXbxXEq2oYrwZo3kBf12glS/CcDwmjpkQfAWXPp0vSTx+b\nvotSfiy2NOiZexCIGTzvTYAWSH55XR94HMRpO9aGcnuTuAmidzgHjeXGgQCzA+FE\nAsesyEGxVw9orPJKyPjg87gYW+tjafO9EuB9w6skSoPWZDhMEmqFx5Mz0BwZms1F\nuwIDAQAB\n-----END PUBLIC KEY-----\n"}

the public key of post_data is generated by OpenSSL::PKey::RSA.generate(2048).public_key.to_pem

def post_data
  post_data = { name: name, admin: false }
  post_data[:public_key] = generated_public_key if self_generate_keys?
  post_data
end
def generated_private_key
  @generated_key ||= OpenSSL::PKey::RSA.generate(2048)
end

def generated_public_key
  generated_private_key.public_key.to_pem
end

The response from chef server:

{
  "uri": "https://localhost:443/clients/one_node",
  "chef_key": {
    "public_key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsMGJmQfMI04E2U1RDJgB\n0mb5zlDDXAOwxza9r4FvHy/0/enY9soea6MmDTaTRDawpuLumqlZefYWLrIc3u45\ns6d+DFPnl0bYuQMUsANoGWgQsx7uqoqRI1dT/62YFbrLLLX+N8HXHHI2cElZOWwW\nL6xP2+RCLq2XuL/w06TXbxXEq2oYrwZo3kBf12glS/CcDwmjpkQfAWXPp0vSTx+b\nvotSfiy2NOiZexCIGTzvTYAWSH55XR94HMRpO9aGcnuTuAmidzgHjeXGgQCzA+FE\nAsesyEGxVw9orPJKyPjg87gYW+tjafO9EuB9w6skSoPWZDhMEmqFx5Mz0BwZms1F\nuwIDAQAB\n-----END PUBLIC KEY-----\n",
    "uri": "https://localhost:443/clients/one_node/keys/default",
    "name": "default",
    "expiration_date": "infinity"
  }
}

Then, private key paired with the public key above, will be written to the destination path, for example, "/var/folders/2t/3qdzts6n75z981qp15dvzqc40000gn/T/d20181209-1663-7yn4es/one_node.pem"

def private_key
  if self_generate_keys?
    generated_private_key.to_pem
  else
    @server_generated_private_key
  end
end
def write_key
  ::File.open(destination, file_flags, 0600) do |f|
    f.print(private_key)
  end
rescue IOError => e
  raise Chef::Exceptions::CannotWritePrivateKey, "Error writing private key to #{destination}: #{e}"
end

And the client object will be return as instance of Chef::ApiClient

def api_client(response)
  return response if response.is_a?(Chef::ApiClient)

  client = Chef::ApiClient.new
  client.name(name)
  client.public_key(api_client_key(response, "public_key"))
  client.private_key(api_client_key(response, "private_key"))
  client
end

Example:

#=> api_client(response)
=> Chef::ApiClient name:'one_node' admin:'false' validator:'false' public_key:'-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApVcU4uXuGbHx8+eTNjN0
hdLiQ5kJ37A3lcmmV3X4madReIJNi45+2GbzokE5T5FmX70EvaDOqjffkb0sLj5K
XtLHm3JzbUv63jDyzZTEvCdJUXneRW9kem30DxvO/Hl/PI0CG5SXRPTX1+nMwf2p
DA9aFGYTEC5PHuhOw5ILWF/oFdN32hHL0IWVjw5Eno5fduZ+jkNNjTG5irKaz7S1
pYWwfi4OIM6F/jeCVzIbAmi2smEDbFJOcMMnZKBvKYnzPB8bGvb0+UebRuq/KyT3
/Ck0ckq5pJ+aYtGvcoGuSvkuVXVUxaqSBgn2As1C4SOg3r7zZnomEg0IW1rsn+8t
pwIDAQAB
-----END PUBLIC KEY-----
' private_key:''

create_node!

Node creation looks relatively more simple

# Create the node object (via the lazy accessor) and save it to the Chef API
def create_node!
  node.save
end

node object is created and populated from Chef::Config or command line input, and post to chef server with the freshly created client and client_key.

# Create a new Chef::Node.  Supports creating the node with its name, run_list, attributes
# and environment.  This injects a rest object into the Chef::Node which uses the client key
# for authentication so that the client creates the node and therefore we get the acls setup
# correctly.
#
# @return [Chef::Node] new chef node to create
def node
  @node ||=
    begin
      node = Chef::Node.new(chef_server_rest: client_rest)
      node.name(node_name)
      node.run_list(normalized_run_list)
      node.normal_attrs = first_boot_attributes if first_boot_attributes
      node.environment(environment) if environment
      node.policy_name = policy_name if policy_name
      node.policy_group = policy_group if policy_group
      (knife_config[:tags] || []).each do |tag|
        node.tags << tag
      end
      node
    end
end

# @return [Chef::ServerAPI] REST client using the client credentials
def client_rest
  @client_rest ||= Chef::ServerAPI.new(chef_server_url, client_name: node_name, signing_key_filename: client_path)
end

Under the hood, it tries to put the json data below to update the one_node on chef server, or post to create the node.

Very simple node data for simplicity

{"name":"one_node","chef_environment":"_default","json_class":"Chef::Node","automatic":{},"normal":{},"chef_type":"node","default":{},"override":{},"run_list":[]}

And, server will response with uri of the node

{
  "uri": "https://localhost:443/nodes/one_node"
}

STEP 2: BootstrapContext

The purpose of BootstrapContext is to render bootstrap template which will run on remote node.

The template is general located the chefdk gem path: "/opt/chefdk/embedded/lib/ruby/gems/<ruby version>/gems/chef-<chef version>/lib/chef/knife/bootstrap/templates/chef-full.erb"

The template looks like

# ......
do_download() {
# ......
# this method is to download chef-client package with all possible way, python/wget/per/curl...
}

# ......
# try to download Chef Omnibus Package if no /usr/bin/chef-client is detected

  install_sh="<%= knife_config[:bootstrap_url] ? knife_config[:bootstrap_url] : "https://omnitruck-direct.chef.io/chef/install.sh" %>"
  if test -f /usr/bin/chef-client; then
    echo "-----> Existing Chef installation detected"
  else
    echo "-----> Installing Chef Omnibus (<%= latest_current_chef_version_string %>)"
    do_download ${install_sh} $tmp_dir/install.sh
    sh $tmp_dir/install.sh -P chef <%= latest_current_chef_version_string %>
  fi
# .....

# lay down the certificate generated by create_node! just now
<% if client_pem -%>
cat > /etc/chef/client.pem <<'EOP'
<%= ::File.read(::File.expand_path(client_pem)) %>
EOP
chmod 0600 /etc/chef/client.pem
<% end -%>

# .....
# generate /etc/chef/client.rb based on Chef::Config or command line input
cat > /etc/chef/client.rb <<'EOP'
<%= config_content %>
EOP

# .....
echo "Starting the first Chef Client run..."
# actually start `/usr/bin/chef-client` for first time
<%= start_chef %>

So, where is client_pem,start_chef,config_content defined?. The key is this line, Erubis::Eruby.new(template).evaluate(bootstrap_context), this code lines apply bootstrap_context of Chef::Knife::Bootstrap to the template, so that client_pem,start_chef,config_content will be evulated.

def render_template
  @config[:first_boot_attributes] = first_boot_attributes
  template_file = find_template
  template = IO.read(template_file).chomp
  Erubis::Eruby.new(template).evaluate(bootstrap_context)
end

One example:

#=> k.bootstrap_context.start_chef
=> "chef-client -j /etc/chef/first-boot.json --no-color"

Final bootstrap bash script to be run on the target node

#=> puts k.render_template
#....
  install_sh="https://omnitruck-direct.chef.io/chef/install.sh"
  if test -f /usr/bin/chef-client; then
    echo "-----> Existing Chef installation detected"
  else
    echo "-----> Installing Chef Omnibus (-v 13)"
    do_download ${install_sh} $tmp_dir/install.sh
    sh $tmp_dir/install.sh -P chef -v 13
  fi
# .....
mkdir -p /etc/chef

cat > /etc/chef/client.pem <<'EOP'
-----BEGIN RSA PRIVATE KEY-----
.............
cUBiob2YrYYBgav5PT1Ubyf4m3kKnj73u67r6e5bFcAnM2PSB4E=
-----END RSA PRIVATE KEY-----

EOP
chmod 0600 /etc/chef/client.pem
#....
cat > /etc/chef/client.rb <<'EOP'
chef_server_url  "https://localhost:443"
validation_client_name "chef-validator"
log_location   STDOUT
node_name "one_node"
ssl_verify_mode :verify_none

EOP

cat > /etc/chef/first-boot.json <<'EOP'
{"run_list":[]}
EOP


echo "Starting the first Chef Client run..."

chef-client -j /etc/chef/first-boot.json --no-color'

STEP 3: Chef::Knife::Ssh

Ok, the next step is to run the bootstrap script on the remote node, there is no doubt we need to rely on ssh, that is also chef trying to do.

knife_ssh.run is instance of Chef::Knife::Ssh with proper settings, no matter combination of ssh_user/ssh_password or ssh_user/ssh_identity_file. server_name is the name of target node, ssh_command is simply render_template

# setup a Chef::Knife::Ssh object using the passed config options
#
# @return Chef::Knife::Ssh
def knife_ssh
  ssh = Chef::Knife::Ssh.new
  ssh.ui = ui
  ssh.name_args = [ server_name, ssh_command ]
  ssh.config[:ssh_user] = user_name || config[:ssh_user]
  ssh.config[:ssh_password] = config[:ssh_password]
  # ....
  ssh.config[:ssh_identity_file] = config[:ssh_identity_file] || config[:identity_file]
	# ...
  ssh
end

If there is no other accident happening to the run of boostrap script, you will be happy to see the node is showed on chef server. From below, Run List is empty, since we don’t specify one yet

#=> knife node show one_node  -s https://localhost:443
Node Name:   one_node
Environment: _default
FQDN:        
IP:          
Run List:    
Roles:       
Recipes:     
Platform:     
Tags: