HashiCorp Vault PKI Secrets Engine Demo for Certificate Management

Have you ever wondered if there is an easier way to manage certificates? In this post, you will learn how to use HashiCorp Vault’s PKI secrets engine to do so.

Video

Below is a video explanation and demo.

Video Chapters

You can skip to the relevant chapters below:

00:00 – Introduction
03:43 – Terraform to create the Root and Intermediate CAs
07:13 – Generate a leaf certificate for Grafana
10:13 – Add the cert to Grafana
11:20 – Chrome doesn’t trust the cert
12:30 – Add Root & Intermediate CAs to Windows Cert Store
16:30 – Chrome now trusts the Grafana cert
20:30 – Revoke the cert
22:20 – Clear the CRL Cache in Windows 10
23:25 – Chrome shows the cert is revoked
24:44 – Walkthrough of the Terraform Code
36:00 – Conclusion

Overview

Certificate management is not an easy task. Most system administrators dread the day they have to work on renewing a certificate. This is because of a couple of reasons:

It’s a manual and complex process done every few years, so it’s hard to remember everything. Therefore, you’ll find that system administrators have a method of procedure stored somewhere for when they need to go through this process again.
It’s risky because it may incur an outage during the process.

HashiCorp Vault’s Public Key Infrastructure (PKI) secrets engine can streamline distributing TLS certificates and allows users to create PKI certificates with a single command. Vault PKI reduces the overhead around the usual manual process of:
– Generating a private key and CSR
– Submitting to a CA
– Waiting for a verification and signing process to complete

This is all done while additionally providing an authentication and authorization mechanism to validate as well.

It’s quite simple to issue certificates with Vault and we’ll show you how in this blog post. You can now create short-lived certificates, which is considered best practice from a security point of view.

We use the Vault Terraform provider to build everything. You can also opt to use the CLI, API, or Web UI to run commands to configure Vault for PKI using this guide. If you’ve been following some of my blog posts, you’ll see that I’m a big fan of using Terraform wherever possible to define our infrastructure as code.

Credit

This work is inspired by Steve Dillon’s medium blog here.

Basic Definitions

Before jumping into the demo and the code, let’s make sure we understand a few definitions.

What is Certificate Management?

Certificate management is a collection of processes involved in securely creating, issuing, renewing, revoking, and archiving digital certificates. A digital certificate is an electronic document that contains information about an entity (e.g., organization or individual) that has been verified by a trusted third party (e.g., a certificate authority). Certificates are typically used to establish secure connections over networks such as the Internet or to digitally sign documents.

A Certificate Authority (CA)

A CA service provides an essential component of the online security infrastructure. CA services enable organizations to generate, issue and manage digital certificates which are used to securely identify and authenticate server communications over the internet. By issuing trusted certificates, CAs help ensure that data sent over the internet is secure by verifying the hosting entity’s identity and the integrity of the transmitted data. This helps to protect sensitive information such as financial transactions, passwords, and medical records from malicious attackers. The CA service is an indispensable tool for organizations looking to securely manage their digital identities while maintaining trust in their online communication. With a Certificate Authority service, organizations can be sure that all of their data is secure and protected. By using a CA service, organizations can create trustworthy digital identities for themselves, enabling them to confidently connect with customers and partners. A certificate authority service is a powerful tool for organizations looking to take advantage of secure online communication with confidence.

A Certificate Signing Request (CSR)

A CSR is a message sent from an applicant to a certificate authority in order to apply for a digital certificate. The CSR contains information about the applicant, such as the public key and identity information, which is used by the CA to create the certificate. The CSR also contains a signature which is used to authenticate the request and ensure it is from a legitimate source. The certificate authority verifies the information provided in the CSR before signing and issuing the digital certificate, making sure it meets all applicable requirements. This helps to ensure that only trusted entities can issue certificates, keeping your data secure. With a CSR, organizations are able to obtain digital certificates quickly and securely, allowing them to establish secure connections with customers and partners.

A Vault Server

The Hashicorp Vault server is an open-source solution for securely storing and managing secrets, such as cryptographic keys, passwords, and other sensitive information. It enables organizations to manage their own private key infrastructure in a secure manner by providing an encrypted storage system for certificates and other confidential data. The Vault server provides an easy-to-use interface for securely managing secrets, as well as APIs for automating certificate and key management. The Vault server also provides granular access control so that organizations can limit who can access their confidential data. With its robust security features, the Hashicorp Vault server is an essential tool for protecting sensitive information and maintaining a secure environment. It makes it easy to keep all of an organization’s secrets safe and secure. It also makes it easier for organizations to manage their digital certificates and other confidential data, giving them the peace of mind that comes with knowing their sensitive data is protected.

A Vault Client

This could be a human or an application accessing the Vault server. The communication between the client and server is typically encrypted in transient. Any secrets on the server are stored encrypted at rest.

The Vault Certificate Authority (Vault CA)

The Vault Certificate Authority (Vault CA) is an open-source tool used to securely manage Public Key Infrastructure (PKI) certificates. It enables users to generate, issue, and revoke certificates based on their own trust policies and requirements. The Vault CA also provides a centralized platform for issuing and managing certificates across environments, making it easy to securely distribute and revoke certificates quickly in response to changes. With the Vault CA, users can ensure secure communication between servers while also reducing manual tasks associated with certificate management. This makes it a powerful tool for organizations looking to securely manage their PKI infrastructure. By using Vault CA, organizations can maintain trust and security within their network without needing to manually manage certificates.

The Vault PKI API

The Vault PKI API provides an interface for interacting with the Vault CA, enabling organizations to easily and programmatically manage their certificate inventory. The PKI API is designed to be user-friendly and straightforward, allowing developers to quickly write scripts that interact with the Vault CA backend. It also allows users to automate certificate management such as issuing, renewing and revoking certificates, as well as managing certificate stores. With the PKI API, organizations can integrate and manage their certificates within their existing applications with ease while still maintaining a secure infrastructure. The Vault PKI API makes it easy and efficient to securely manage all of your organization’s certificates. This helps ensure that your sensitive data is always safe and secure. The PKI API is an essential component of the Vault CA, giving organizations the peace of mind and security that comes with having a secure certificate management platform.

Code

Pre-requisites

The following is required to follow along:

An Installation of HashiCorp Vault
Terraform
An Installation of Grafana on a Linux VM or in a Docker container with a Linux image

Create the Root and Intermediate CAs in Vault

First, we need to run the env-vars.sh script to export the VAULT_ADDR and VAULT_TOKEN environment variables. We used the root token for Vault for this demo, but you should not do this in production. You would need to authenticate into Vault using your preferred authentication method.

env-vars.sh:

#!/bin/bash
export VAULT_ADDR=https://<url>
export VAULT_TOKEN=s.xxxxxxxxx

Next, we run terraform apply to create the root and intermediate CAs along with roles in Vault. For a walkthrough of the Terraform code, check this section of the video.

Create the Leaf Certificate with Vault

To create leaf certificates: Run the following command:

./create-server-certs.sh <cert_name> <common_name> <ip_sans> <TTL_in_seconds>

An example would look like this:

./create-server-certs.sh grafana docker01.home 192.168.1.80 31556952

Below is the bash script that we ran. Notice the certificate and private key are created inside of an output folder. Make sure not to push this output folder into GitHub.

create-server-certs.sh:

#!/bin/bash
# pass the name of the cert as the first argument to the script, common name as second argument, ip_sans as third, and TTL in seconds as fourth. Example: `./create-server-certs.sh grafana docker01.home 192.168.1.80 31556952`
set -eu
echo Creating cert for $1
echo Common name: $2
echo IP Sans: $3
echo TTL: $4
source ./env-vars.sh
mkdir -p output/$1
vault write pki-int-ca/issue/server-cert-for-home ttl=$4 common_name="$2" ip_sans="$3" -format=json > output/$1.json
cat output/$1.json | jq -r '.data.certificate' > output/$1/$1_cert.pem
cat output/$1.json | jq -r '.data.private_key' > output/$1/$1_key.pem
cat output/$1.json | jq -r '.data.issuing_ca' > output/$1/ca.pem
cat output/$1.json | jq -r '.data.ca_chain[]' > output/$1/ca_chain.pem
# Dump the certificates in text mode
openssl x509 -noout -text -in output/$1/ca.pem > output/$1/ca.pem.txt 
openssl x509 -noout -text -in output/$1/$1_cert.pem > output/$1/$1_cert.pem.txt

Moving the Certificate and the Private Key

Now that we have our certificate and key, we need to move them to the server where Grafana is running. Below is an scp command to do that.

# Example command to scp the folder containing the cert and the key from local to remote machine:
scp -r /mnt/c/Users/Sam/Deployments/HashiCorp/Vault/vault-ca-demo/output/grafana [email protected]:/home/sam/automation/grafana/config/certs

You then need to reload Grafana so it can pick up the certificate. Now use your browser to navigate to the Grafana site using https. You’ll notice that your browser gives you a warning message. Below is what it looks like in Chrome:

This message is basically saying that the browser doesn’t trust this certificate because it doesn’t know nor trust the CA that issued the certificate. This is because our Vault’s root and intermediate CAs are not part of the certificate store of Windows. We’ll see how to add these two CAs to the Windows certificate store in the next section.

Add the Root & Intermediate CAs to the Windows Certificate Store

The Windows certificate store doesn’t understand the .pem format so we need to convert to the .crt format using the script below:

convert-pem.sh:

#!/bin/bash
# Convert the root and int certs from .pem to .crt to be used in the Windows Certificate Store
openssl x509 -outform der -in output/root_ca/ca_cert.pem -out output/root_ca/ca_cert.crt
openssl x509 -outform der -in output/int_ca/int_cert.pem -out output/int_ca/int_cert.crt

In the Microsoft Management Console (mmc):
– You will need to import the root ca certificate into the Trusted Root Certification Authorities -> Certificates folder
– Also, import the intermediate ca certificate into the Intermediate Certification Authorities -> Certificates folder
– You may need to reboot the computer (I tried restarting the Chrome browser, but that didn’t work)

Below you can see where we’ve imported both the Vault root and intermediate CAs in the Windows certificate store.

Vault Root CA added to Windows Cert Store

Vault Intermediate CA added to Windows Cert Store

You can also watch the video section going through the process here.

Chrome Trusts the Grafana Certificate

Now you can once again navigate to the Grafana URL using https. Make sure you open a new incognito window to avoid caching. When you do, you’ll see that Chrome trusts the certificate issued as it trusts the entire certificate chain. The certificate chain is made up of the root CA and the intermediate CA.

Revoke the Certificate

You can revoke a certificate by following this example:

vault write pki-int-ca/revoke serial_number="62:d3:ac:77:93:25:34:11:e0:47:27:0f:d1:db:92:67:51:8c:30:3c"

You can also remove a revoked certificate and clean the Certificate Revocation List (CRL) by running this command:

vault write pki_int/tidy tidy_cert_store=true tidy_revoked_certs=true

Clear the CRL Cache in Windows

There is a CRL cache that needs to be flushed in Windows so that the Chrome browser reaches out to the CRL URL to check if the certificate was revoked. I found this out from this forum post.

To view the CRL Cache:

certutil -urlcache crl

To Clear it:

certutil -setreg chain\ChainCacheResyncFiletime @now

Chrome Shows the Certificate is Revoked

Now if you try navigating to the Grafana URL again, you won’t be able to proceed because the certificate is revoked.

Conclusion

In this blog post, we talked about certificate management with the Vault PKI secrets engine. We saw how easy it was to create a root and intermediate CA inside of Vault. It was also quite quick and simple to create a leaf certificate for Grafana. We didn’t need to go through the lengthy process of:
1. Generating a private key and CSR
2. Submitting to a CA
3. Waiting for a verification and signing process to complete

We also witnessed how to revoke a certificate easily.

Most organizations have their own root CA which they use to sign an intermediate CA for Vault. They then import that intermediate CA into Vault and use Vault to issue leaf certificates. The leaf certificates issued by Vault now are trusted internally in an organization because the certificate chain is trusted based on the root CA.

You may be wondering if there is a way to automate the renewal of leaf certificates. Take a look at this post that talks about Consul-Template to Automate Certificate Management for HashiCorp Vault PKI.