Webblog App Part 4 – HashiStack – Nomad Consul Vault Terraform
- 10 minutes read- 2007 words
This is Part 4 and the final post of our Webblog app series. In this post, we make use of the entire HashiStack. This is similar to the previous setup used in Part 3, except that we swap Kubernetes for Nomad as our orchestrator.
As a quick recap, my goal was to learn the different HashiCorp tools by developing a web app called the Webblog app.
In Part 1, we discussed Infrastructure as Code and Policy as Code concepts. We also showed how to prepare the infrastructure for the app using Terraform. We built a Vault cluster backed by a Consul cluster for storage. This was all built on Google’s Kubernetes Engine (GKE) using Terraform providers.
Below is a summary of how the entire HashiStack was used for this app. In this blog post, we will cover each of the 4 HashiCorp tools (Terraform, Vault, Nomad, and Consul) and how they work together to deliver our app.
Terraform to deploy a Nomad and a Consul cluster using Packer images
Terraform used to configure Vault and to run jobs in Nomad
Vault used to supply dynamic database secrets to the application in addition to encrypting the content of my posts
Nomad used to orchestrate and run the application
Consul used for all things networking including running a service mesh
Terraform and Packer - Infrastructure Provisioning
I forked the work done by Gruntwork in this repo into my own terraform google nomad repo. This was because I wanted to create one mono repo with both the app and the infrastructure that supports the app. Gruntwork created Packer configs to deploy images in GCP containing both Consul and Nomad clients and servers. These images are used by Terraform to create the VMs in GCP. More information on infrastructure deployment is found below.
Infrastructure Deployment Workflow
The infrastructure deployment of this Hashistack is quite simple. Below are the tools used:
Github is used for version control.
HashiCorp Packer is used for creating images in Google Cloud.
Terraform is used to build 3 VMs from the Packer created images.
Terraform Cloud is tied to Github so upon a commit and a push into Github, Terraform Cloud deploys the 3 VMs into Google Cloud. These VMs are for the following:
1 VM acts as both a Nomad and a Consul server.
2 VMs are used as both Nomad and Consul clients.
We make use of the Vault cluster that was already deployed in Part 1 to keep things simple.
Application Deployment Workflow
If you recall, the Webblog app is a simple Python/Flask app that talks to a MongoDB database. Both the Python app and MongoDB are containerized.
The application workflow is as follows:
Commit and push code to Github
Terraform uses the Nomad provider to deploy or update the Python app and MongoDB into Nomad
Consul takes care of networking between the Python app and MongoDB
The Python app requests MongoDB credentials from Vault
Vault generates dynamic credentials in MongoDB and passes them to the Python app
The Python app encrypts the content of the blog posts via the Vault Encryption as a Service API
Vault - Secrets Management
As mentioned above, we are re-using the Vault cluster that was created in Part 2 via Terraform in Google’s Kubernetes Engine. We also use the Vault Terraform provider to configure Vault to work in our new HashiStack setup. Vault is configured to use a new MongoDB database secrets engine. However, we are utilizing the existing transit engine for encryption. Below are the relevant code sections to this new set up. You can see the updated repo here.
Let’s take a look at how Nomad works in conjunction with the rest of the HashiStack.
Nomad Terraform Provider
As mentioned earlier, we use the Nomad Terraform provider to deploy Nomad jobs. We need to specify the NOMAD_TOKEN as an environment variable since we’re using Access Control Lists (ACLs) in the Nomad cluster. We define this in Terraform Cloud as a sensitive environment variable.
Below is an image of the Terraform configuration used to deploy the Nomad job. You can see the nomad job file in the Github repo.
Once Terraform deploys the Nomad job, you can see it running in the Nomad dashboard as shown below.
ACLs are used by Nomad to secure the system. There are three major components to the ACL system: tokens, policies, and capabilities. The components are illustrated in the diagram below and you can find more in this learn guide.
The config file for Nomad servers and clients needs to have the ACL stanza and some more configuration items according to the Nomad ACL Bootstrap guide. Below is a simple example for the ACL stanza in the config file for Nomad servers and clients:
enabled = true
After bootstrapping the Nomad cluster, you need to create policies. Check out how to create a policy for an app developer to deploy jobs. Below is the policy config I used in my app, however, for a production app, you will need to be more restrictive with the policy.
Similar to the CNI plugins, Docker is not installed by default in the Packer images. So make sure you have it installed so that the Nomad Docker driver works.
Nomad Vault Config
You need to add the following config on the Nomad Servers to allow connections to Vault. I used the Vault root token for this demo, however, this is definitely not recommended in production. You can read more about the Nomad Vault integration in this document.
To allow Nomad to connect to Consul, you will need the following config in the Nomad server and client config files. Notice the use of the token here since we’re using ACLs in Consul.
address = "127.0.0.1:8500"
token = "xxxxx"
Nomad Host Volumes
Nomad host volumes can manage storage for stateful workloads running inside a Nomad cluster. In my app, I used host volumes as per this guide. Nomad also supports Container Storage Interface (CSI), however, it’s currently in beta as of the time of writing this blog post.
The configuration needed on the Nomad clients for enabling a host volume is found below:
When you need to orchestrate non-containerized workloads or a combination of both containerized and non-containerized workloads. Nomad supports a variety of task drivers.
For teams that are new to the microservices and containers world and need a simple solution. K8s has a steeper learning curve.
When a simple architecture is needed. Nomad is a single binary, both for clients and servers, and requires no external services for coordination or storage. You can think of K8s as a solution with batteries included, but that sometimes adds unnecessary complexity. Nomad has the philosophy of build as you go.
Consul - Networking
Consul behaves the same way as when we used it with Kubernetes in Part 3. Let’s take a look at how it works with Nomad.
Consul Service Mesh
As we did in Part 3, here we will also use Consul Connect Service Mesh for networking. Below is a diagram showing the control plane and the data plane for Consul Connect. You can also reference the Understand Consul Service Mesh guide to learn more about Consul Connect.
You can see how Nomad is tightly integrated with Consul. A MongoDB service along with a sidecar proxy service is automatically created when Nomad creates a job. Also, the Python frontend service with a sidecar proxy is created.
When using Consul Connect with ACLs, it’s best practice to explicitly define Intentions to avoid any confusion. You can find out more here.
ACLs are used in Consul to secure access to the UI, API, CLI, service, and agent communications. It’s fine to run Consul with no ACLs in a test/dev environment, however, in production, it is a must.
Consul clients and servers need the following ACL stanza in the config file to enable ACLs. HashiCorp learn has a good guide on ACLs.
It’s been a fun and educational journey exploring the different HashiCorp products in this four-part series of blogs. I hope that, from a developer’s perspective, you learned the reason to use these tools just as I have.
Below you’ll find my key learnings from using the entire HashiStack:
All the HashiCorp tools within the HashiStack work seamlessly together
Terraform is an elegant provisioning tool for infrastructure deployments, for Nomad job deployments, and for Vault configurations. Terraform Cloud’s integration with various version control systems improves workflow tremendously.
Vault is a swiss army knife when it comes to secrets management. The database dynamic secrets engine along with the transit engine secured my application by significantly reducing the attack surface
Vault and Consul are first-class citizens in a K8s environment just as in a Nomad one
Choose Nomad over K8s when you need a simple lightweight orchestrator for containerized and non-containerized workloads
Consul Connect Service Mesh is a joy to work with and makes connectivity of microservices in both Nomad and K8s environments very simple and straightforward
Finally, I hope you’ve enjoyed this journey as much as I have and I encourage you to explore the HashiStack in more depth as I surely will.