Consul-Template to Automate Certificate Management for HashiCorp Vault PKI

Learn how to automate certificate management for the HashiCorp Vault PKI secrets engine.

Video

Below is a video explanation and demo.

Video Chapters

You can skip to the relevant chapters below:

00:00 – Introduction
00:55 – Diagram of Workflow
03:50 – Demo Steps
05:47 – Demo Config Walk-through
10:00 – Run the Demo
17:27 – Distributing Consul-Template
18:35 – Conclusion

Overview

In this blog post, we talk about how to use consul-template to automate certificate management for the HashiCorp Vault PKI secrets engine.

We previously discussed how the HashiCorp Vault PKI secrets engine works to create certificates. However, we didn’t discuss how to automate the creation and renewal of certificates. This is what we cover here.

We have the same setup as before with Vault running a root and an intermediate CA. Then we configure consul-template to automatically:
1. Authenticate to Vault
2. Renew the Vault authentication token
3. Create a new certificate in Vault
4. Place the newly created certificate in the proper folder for Grafana
5. Reload the Grafana Docker container so that Grafana could apply the newly generated certificate

The distribution and installation of consul-template itself could be done by one of the two following methods:
1. HashiCorp Packer to embed consul-template in the images of VMs created
2. A configuration management tool such as Ansible

Code

Pre-requisites

The following is required to follow along:

An Installation of HashiCorp Vault
An Installation of Grafana on a Linux VM or in a Docker container with a Linux image

Workflow

Below is a diagram showing the workflow between the consul-template and Vault.

The consul-template agent does the following:
1. Authenticates into Vault
2. Renews/Creates the Grafana certificate in Vault
3. Renders the certificate and private key in a Grafana folder based on a template file
4. Restarts the Grafana process or container

Automated Certificate Renewal Steps

Now let’s examine the steps to make consul-template work.

Create a Vault PKI policy and a Token

Create a policy that has the update capability.

path "pki-int-ca/issue/server-cert-for-home" {
  capabilities = ["update"]
}

Then write the policy in Vault.

vault policy write pki pki.hcl

Finally, generate a Vault token based on the policy you just created.

vault token create -policy="pki" -period=24h -orphan

Consul-Template Configuration

You can find the configuration file and the template files for consul-template in the consul-template-grafana folder in GitHub.

Below you’ll find the consul-template configuration file

consul-template.hcl:

vault {
  address      = "https://vault.tekanaid.com"
  # I'm using the environment variable VAULT_TOKEN instead.
  # token        = "s.xxxxxx"
  # grace        = "1s"
  unwrap_token = false
  renew_token  = true
}
syslog {
  enabled  = true
  facility = "LOCAL5"
}
template {
  source      = "/home/sam/automation/grafana/config/certs/grafana_cert.tpl"
  destination = "/home/sam/automation/grafana/config/certs/grafana_cert.pem"
  perms       = 0755
  command     = "docker restart automation_grafana_1"
}
template {
  source      = "/home/sam/automation/grafana/config/certs/grafana_key.tpl"
  destination = "/home/sam/automation/grafana/config/certs/grafana_key.pem"
  perms       = 0755
  command     = "docker restart automation_grafana_1"
}

Notes on the Consul-Template Configuration File

We used the VAULT_TOKEN environment variable with the token generated before instead of defining the token in this file. This way it’s more secure. To do that you need to export the environment variable like this:

export VAULT_TOKEN=s.xxxxx

You can use an unwrap_token to securely introduce the token into the environment. You can check our blog post on the Secret Zero Problem and Secure Introduction here.
There are 2 template blocks, one is for the certificate and the other is for the key.
The template block has a source, which is the template file and a destination, which is the certificate or the key files rendered.
You can specify the permissions for the certificate and key files.
Finally, consul-template can run a command for you after it renders the file. In our case, the command is to restart the docker container so that Grafana can use the certificate.

Template Files

These template files use Golang’s templating engine. As you see below, consul-template uses them to create/renew and retrieve the certificate and private key from Vault. You need to specify the following:
– The path of the PKI secrets engine
– Time-to-Live (TTL), which determines the expiry time of the certificate
– The common name of the certificate
– Any IP_SANS you want to include

grafana_cert.tpl:

{{ with secret "pki-int-ca/issue/server-cert-for-home" "ttl=30s" "common_name=docker01.home" "ip_sans=192.168.1.80" }}
{{ .Data.certificate }}
{{ end }}

grafana_key.tpl:

{{ with secret "pki-int-ca/issue/server-cert-for-home" "ttl=30s" "common_name=docker01.home" "ip_sans=192.168.1.80" }}
{{ .Data.private_key }}
{{ end }}

Start Consul-Template

Now that we have all our files ready, we can start the consul-template agent. Use the command below to start it in the foreground to see the logs. However, you should use Systemd to run this in production.

consul-template -config consul-template.hcl

Navigate to the Grafana page using https and check that the certificate is valid for 30 seconds. Then wait a minute to see the new Valid from and Valid to dates change automatically. This shows you that consul-template is renewing the certificate for you in the background.

If you stop consul-template and wait for a minute, you’ll find that the certificate is now invalid. You may need to open an incognito window to see the effect. You can check this section of the video for a demo.

In practice, you would want to increase the TTL. We like to put it at 90 days. You can then check the certificate in your browser to verify. Below is a screenshot from Chrome.

Conclusion

Automating certificate management for the Vault PKI secrets engine can be done using the consul-template tool from HashiCorp. Consul-template takes care of the creation and renewal of certificates for system administrators. It runs as a daemon in the background and automatically renews certificates by authenticating with Vault and retrieving a new certificate. This is done before the old certificate expires.

If you missed our previous blog post on HashiCorp Vault PKI Secrets Engine for Certificate Management you can check it out here.