Webblog App Part 2 - Secrets Development Phases with Vault
- 9 minutes read- 1805 words
This is Part 2 of our Webblog app series where we discuss secrets development phases. As a reminder, my goal was to learn the different HashiCorp tools by developing a web app called the Webblog app. In Part 1, we discussed Infrastructure as Code and Policy as Code concepts. We also showed how to prepare the infrastructure for the app using Terraform. We built a Vault cluster backed by a Consul cluster for storage. This was all built on Google’s Kubernetes Engine (GKE) using Terraform providers.
In this post, we will discuss 4 secrets development phases for the Webblog app. These phases are very common among organizations as they adopt a centralized Secrets Management solution such as Vault. We’ll call these phases:
The Webblog app is a simple Python/Flask app that talks to a MongoDB datastore. We used GitLab as our Version Control System (VCS) and as a Continuous Integration/Continuous Deployment (CI/CD) system to develop the Webblog app. We used GitLab’s internal docker registry to host the docker container for the app. On every commit and push to the master branch, GitLab builds a docker image for the app and pushes it to the GitLab docker registry. For simplicity, I created a GitLab runner on my local computer. You could build GitLab runners as VMs using Terraform or you could run the GitLab runners as containers within a K8s cluster.
I used helm to deploy MongoDB ahead of the Webblog app. In a later iteration, I may build a helm chart to encapsulate both the Python/Flask app and the MongoDB datastore. The CI/CD workflow in this latter case would be a simple upgrade to the helm chart.
Here is what the Webblog app looks like when deployed:
To start using Vault, there are two steps; the first step is authentication, and the second step is the retrieval of secrets.
A. Vault Authentication
To retrieve secrets from Vault, we first need to authenticate. Vault brokers the authentication to a third party. That third party validates the request. Upon successful authentication, Vault confirms authorization to resources and provides a token that allows access to those resources.
We used Terraform to apply the necessary configuration for Vault as mentioned in Part 1 in the Vault config section. You can find the Terraform code for that here. If you followed along in Part 1 and applied the Terraform plan for the Vault configuration, then you’re all set to proceed. Otherwise, please go back and do so now.
Make sure to run the following command inside the Vault container after you run Terraform apply. This is an easier way to properly set up the K8s authentication backend since it will automatically capture the kubernetes_host, the kubernetes_ca_cert, and the token_reviewer_jwt variables from within the K8s environment.
Now let’s take a look at the 4 secrets development phases that we went through as we developed this app.
1. Secrets in a Configuration File - No Vault
Before I heard of Vault, I simply defined the MongoDB credentials in a .env file and the app pulled these credentials from that file. I then had to add the .env file to .gitignore to avoid checking it into the VCS. This was risky because it’s easy to accidentally push the creds to a public VCS repo. I have seen some organizations pushing creds to private repos. However, when you have a large organization with many repos. You don’t know who has access to what repos. Moreover, the rotation of these creds is difficult as there is no centralized place to do so.
2. Crawl - Vault Static Key/Value (KV) Secrets
This phase usually occurs when folks are first introduced to Vault. That’s what happened to me. I saw how the KV secrets engine was pretty simple to use, so I started with that.
It works as follows:
When helm installed Vault, it installed it with a Vault Agent Injector. This Vault Agent Injector creates a sidecar with a Vault agent for any pod based on K8s annotations that you specify with a deployment.
The Vault agent sidecar auto-authenticates with Vault and retrieves the KV secret.
Vault agent sidecar then automatically injects the KV secret into the app pod at the file path of: /app/secrets/.envapp.
We modified the app slightly to pick up the secret MongoDB credentials from the file path. The app is still unaware of Vault.
The app then connects to MongoDB using the credentials it obtained.
Below is the configuration section for the K8s app deployment showing the K8s annotations used
Once again, the Vault Agent Injector creates a sidecar with a Vault agent for any pod based on K8s annotations that you specify with a deployment.
The Vault agent sidecar auto-authenticates with Vault.
The Vault agent sidecar injects a Vault token into the file path /app/secrets/token inside the app’s pod.
The app reads the Vault token from the file path.
We modified the app to use the Vault API. The app now requests a dynamic secret from Vault
Vault creates username and passwords creds in MongoDB and passes these creds back to the app.
The app connects to MongoDB.
We configured the dynamic secret to have a TTL of 10 seconds, to show well in the demo.
The app has the logic to request a new secret from Vault when authentication to the Database fails.
The app is now Vault aware and required some changes in the code.
4. Run - Vault Encryption as a Service (EaaS)
Now that we’ve elegantly solved the MongoDB secret creds problem, could we do better? I heard of an EaaS offering from Vault that is a simple API call. It’s basically another secrets engine called: Vault transit engine. I decided to use it to encrypt the content of the posts in MongoDB. This will ensure that if a hacker somehow gains access to my database, he/she will only see encrypted data for the content of the blog posts. This is also useful if you don’t want your DBAs to see certain data in your database.
I liked Vault’s Encryptions as a Service because as a developer, I don’t need to worry about these things:
Finding and maintaining a library in Python that does proper encryption/decryption
Managing the life cycle of the encryption keys. This is a real headache.
The app makes an API call to Vault with the plain-text and an encryption key name that it wants to use. The encryption key should have already been configured in Vault beforehand.
Vault responds with cipher-text
The app writes the cipher-text to MongoDB
Below is what the content post data looks like when not encrypted
And here is what it looks like when encrypted
The app retrieves the post content from MongoDB as cipher-text
The app then makes an API call to Vault with the cipher-text and the same key name it used for encryption.
Vault responds with the plain-text
The app presents the content of the posts in plain-text. The user doesn’t notice anything about the encryption and decryption process.