Table of Contents
HashiCorp Vault 101 - Certified Vault Associate
Get started with HashiCorp Vault and prepare for your Vault Associate Exam
Terraform 101 - Certified Terraform Associate
Learn all you need to know to ace the Terraform Associate Exam and go beyond the certification

Webblog App Part 2 - Secrets Development Phases with Vault

Don't get overwhelmed by all the bells and whistles that HashiCorp Vault has to offer. Here you can learn about some secrets development phases for a web application. We discuss the crawl, walk, and run phases.

Created: April 22, 2020 | Updated: January 27, 2023


Below is a video explanation and demo.

Webblog App Part 2 - Secrets Development Demo Video


This is Part 2 of our Webblog app series where we discuss secrets development phases. As a reminder, my goal was to learn the different HashiCorp tools by developing a web app called the Webblog app. In Part 1, we discussed Infrastructure as Code and Policy as Code concepts. We also showed how to prepare the infrastructure for the app using Terraform. We built a Vault cluster backed by a Consul cluster for storage. This was all built on Google's Kubernetes Engine (GKE) using Terraform providers.

In this post, we will discuss 4 secrets development phases for the Webblog app. These phases are very common among organizations as they adopt a centralized Secrets Management solution such as Vault. We'll call these phases:

  1. Secrets in a Configuration File - No Vault
  2. Crawl - Vault Static Key/Value (KV) Secrets
  3. Walk - Vault Dynamic Secrets
  4. Run - Vault Encryption as a Service (EaaS)


Get FREE access to the source code by subscribing to my newsletter
You only need to subscribe once. Already subscribed? Enter your email to get instant access to the code.


The following accounts are required to follow along:

Webblog App Deployment Workflow

The Webblog app is a simple Python/Flask app that talks to a MongoDB datastore. We used GitLab as our Version Control System (VCS) and as a Continuous Integration/Continuous Deployment (CI/CD) system to develop the Webblog app. We used GitLab's internal docker registry to host the docker container for the app. On every commit and push to the master branch, GitLab builds a docker image for the app and pushes it to the GitLab docker registry. For simplicity, I created a GitLab runner on my local computer. You could build GitLab runners as VMs using Terraform or you could run the GitLab runners as containers within a K8s cluster.

I used helm to deploy MongoDB ahead of the Webblog app. In a later iteration, I may build a helm chart to encapsulate both the Python/Flask app and the MongoDB datastore. The CI/CD workflow in this latter case would be a simple upgrade to the helm chart.

Webblog App Deployment Workflow

Here is what the Webblog app looks like when deployed:

Webblog App Screenshot

Vault Configuration

To start using Vault, there are two steps; the first step is authentication, and the second step is the retrieval of secrets.

Vault Overview

A. Vault Authentication

To retrieve secrets from Vault, we first need to authenticate. Vault brokers the authentication to a third party. That third party validates the request. Upon successful authentication, Vault confirms authorization to resources and provides a token that allows access to those resources.

There are multiple authentication methods that Vault supports. We used the Kubernetes one for our app.

B. Retrieval of Vault Secrets

Once the app is authenticated to Vault, it receives a token. This token is tied to a policy that gives the app authorization to do certain things such as list/read/update secrets for a certain secrets engine. Vault has many secrets engines that you could choose from depending on your use case. In this post, we make use of the KV secrets engine, the dynamic MongoDB database secrets engine and the Transit secrets engine.

Applying the Vault Configuration via Terraform

We used Terraform to apply the necessary configuration for Vault as mentioned in Part 1 in the Vault config section. If you followed along in Part 1 and applied the Terraform plan for the Vault configuration, then you're all set to proceed. Otherwise, please go back and do so now.

Make sure to run the following command inside the Vault container after you run Terraform apply. This is an easier way to properly set up the K8s authentication backend since it will automatically capture the kubernetes_host, the kubernetes_ca_cert, and the token_reviewer_jwt variables from within the K8s environment.

vault write auth/kubernetes/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \

Secrets Development Phases

Now let's take a look at the 4 secrets development phases that we went through as we developed this app.

1. Secrets in a Configuration File - No Vault

Before I heard of Vault, I simply defined the MongoDB credentials in a .env file and the app pulled these credentials from that file. I then had to add the .env file to .gitignore to avoid checking it into the VCS. This was risky because it's easy to accidentally push the creds to a public VCS repo. I have seen some organizations pushing creds to private repos. However, when you have a large organization with many repos. You don't know who has access to what repos. Moreover, the rotation of these creds is difficult as there is no centralized place to do so.

2. Crawl - Vault Static Key/Value (KV) Secrets

This phase usually occurs when folks are first introduced to Vault. That's what happened to me. I saw how the KV secrets engine was pretty simple to use, so I started with that.

It works as follows:

KV Secret App Workflow

  1. When helm installed Vault, it installed it with a Vault Agent Injector. This Vault Agent Injector creates a sidecar with a Vault agent for any pod based on K8s annotations that you specify with a deployment.
  2. The Vault agent sidecar auto-authenticates with Vault and retrieves the KV secret.
  3. Vault agent sidecar then automatically injects the KV secret into the app pod at the file path of: /app/secrets/.envapp.
  4. We modified the app slightly to pick up the secret MongoDB credentials from the file path. The app is still unaware of Vault.
  5. The app then connects to MongoDB using the credentials it obtained.

Below is the configuration section for the K8s app deployment showing the K8s annotations used

        prometheus.io/port: "9117"
        prometheus.io/scrape: "true"
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/agent-inject-token: "true"
        vault.hashicorp.com/agent-limits-cpu: "250m"
        vault.hashicorp.com/agent-limits-mem: "64Mi"
        vault.hashicorp.com/agent-requests-cpu: "128m"
        vault.hashicorp.com/agent-requests-mem: "32Mi" 
        vault.hashicorp.com/role: "webblog"
        vault.hashicorp.com/secret-volume-path: "/app/secrets/"
        vault.hashicorp.com/agent-inject-secret-.envapp: "internal/data/webblog/mongodb"
        vault.hashicorp.com/agent-inject-template-.envapp: |
          {{- with secret "internal/data/webblog/mongodb" -}}
          DB_USER={{ .Data.data.username }}
          DB_PASSWORD={{ .Data.data.password }}
          {{- end -}}
      creationTimestamp: null

3. Walk - Vault Dynamic Secrets

Now that we got our app utilizing Vault as a centralized secrets management system, I figured we could do better. So the 2 reasons for this next phase were:

  • Auto-rotation of credentials with a time-to-live (TTL)
  • A unique set of credentials that could be traced back to the requestor app

Let's now discuss the next phase of our secrets development journey. Here is how the dynamic MongoDB database secrets engine works:

Dynamic DB Secret App Workflow

  1. Once again, the Vault Agent Injector creates a sidecar with a Vault agent for any pod based on K8s annotations that you specify with a deployment.
  2. The Vault agent sidecar auto-authenticates with Vault.
  3. The Vault agent sidecar injects a Vault token into the file path /app/secrets/token inside the app’s pod.
  4. The app reads the Vault token from the file path.
  5. We modified the app to use the Vault API. The app now requests a dynamic secret from Vault
  6. Vault creates username and passwords creds in MongoDB and passes these creds back to the app.
  7. The app connects to MongoDB.

Additional Notes:

  • We configured the dynamic secret to have a TTL of 10 seconds, to show well in the demo.
  • The app has the logic to request a new secret from Vault when authentication to the Database fails.
  • The app is now Vault aware and required some changes in the code.

4. Run - Vault Encryption as a Service (EaaS)

Now that we've elegantly solved the MongoDB secret creds problem, could we do better? I heard of an EaaS offering from Vault that is a simple API call. It's basically another secrets engine called: Vault transit engine. I decided to use it to encrypt the content of the posts in MongoDB. This will ensure that if a hacker somehow gains access to my database, he/she will only see encrypted data for the content of the blog posts. This is also useful if you don't want your DBAs to see certain data in your database.

I liked Vault's Encryptions as a Service because as a developer, I don't need to worry about these things:

  • Finding and maintaining a library in Python that does proper encryption/decryption
  • Managing the life cycle of the encryption keys. This is a real headache.

Encryption Workflow

App Encryption Workflow

  1. The app makes an API call to Vault with the plain-text and an encryption key name that it wants to use. The encryption key should have already been configured in Vault beforehand.
  2. Vault responds with cipher-text
  3. The app writes the cipher-text to MongoDB

Below is what the content post data looks like when not encrypted

Post Content Data not Encrypted in MongoDB

And here is what it looks like when encrypted

Post Content Data Encrypted in MongoDB

Decryption Workflow

App Decryption Workflow

  1. The app retrieves the post content from MongoDB as cipher-text
  2. The app then makes an API call to Vault with the cipher-text and the same key name it used for encryption.
  3. Vault responds with the plain-text
  4. The app presents the content of the posts in plain-text. The user doesn't notice anything about the encryption and decryption process.


In this second part of our Webblog App series, we discussed the 4 phases of secrets development that we went through. We've found Vault to be a security swiss army knife that has many useful functions. We explored the following 3 secrets engines: KV secrets engine, the dynamic MongoDB database secrets engine and the Transit secrets engine.

Join us in Part 3 as we move the app to use Consul Connect's, Service Mesh.

Other Posts
Terraform for Beginners - A Beginner's Guide to Automating Cloud Infrastructure
Terraform vs Ansible - Demo the Differences - Part 2
Terraform vs Ansible - Learn the Differences - Part 1
HashiCorp Vault Backup and Restore Raft Snapshots from Kubernetes to AWS S3
AWS Lambda - Terraform Configuration Example with API Gateway
Securing the Future - DevSecOps Trends for 2023
36 Top DevOps Questions to Get You Started in 2023
Terraform to Create a Ubuntu 22.04 VM in VMware vSphere ESXi
HashiCorp Packer to Build a Ubuntu 22.04 Image Template in VMware vSphere
Migrate Secrets from AWS Secrets Manager to HashiCorp Vault with Python, Docker, and GitLab
Migrate Secrets from AWS Secrets Manager to HashiCorp Vault with Terraform
env0 - A Terraform Cloud Alternative
Terraform Import Example - AWS EC2 Instance
DevOps Engineer NOT on Linux? You're MISSING OUT!
HashiCorp Vault API Tutorial and Pro Tips
HashiCorp Vault Tutorial for Beginners
Create a Pihole Docker Ad Blocker with Ansible and Terraform
Terraform vSphere Windows Example to Join an AD Domain
Build a Kubernetes k3s Cluster in vSphere with Terraform and Packer
HashiCorp Packer to Build a Ubuntu 20.04 Image Template in VMware
Consul-Template to Automate Certificate Management for HashiCorp Vault PKI
HashiCorp Vault PKI Secrets Engine Demo for Certificate Management
Jenkins, Vault, Terraform, Ansible, and Consul Delivering an End-to-End CI/CD Pipeline
Secret Zero Problem Solved for HashiCorp Vault
Hashicorp Packer, Terraform, and Ansible to Set Up Jenkins
Hashicorp Vault Azure Secrets Engine - Secure Your Azure Resources
HashiCorp Waypoint - Will it Replace Your CI/CD?
HashiCorp Boundary - Make Sure Your Human To Machine Access Is Secure
HashiCorp Packer for VMware Ubuntu Templates and Terraform for building VMs
HashiCorp Packer VMware Windows Templates and Terraform for VMs
Webblog App Part 4 – HashiStack – Nomad Consul Vault Terraform
Webblog App Part 3 - Consul Connect Service Mesh
Webblog App Part 2 - Secrets Development Phases with Vault
Webblog App Part 1 - Infrastructure as Code with Terraform
Microservices Applications'​ Life Cycle
HashiCorp Vault 201 - Vault for Apps in Kubernetes
Learn how to use HashiCorp Vault for your applications in Kubernetes