Migrate Secrets from AWS Secrets Manager to HashiCorp Vault with Terraform

Introduction

Andrew at Money Leaves Bank finally convinced Claire, his CIO, that HashiCorp Vault should be their secrets management solution as they are becoming a multi-cloud company. Now he is faced with the challenge of migrating their secrets hosted in AWS Secrets Manager to HashiCorp Vault. In this blog post, learn why Andrew decides to use Terraform for this task and how he implements the solution.

Video

Below is a video explanation and demo.

Video Chapters

In this video, we look at a demo of how to use Terraform to migrate secrets from AWS Secrets Manager to HashiCorp Vault.

You can skip to the relevant timestamps below:

• 00:00 – Introduction
• 00:17 – Scenario
• 01:36 – Terraform Public Module Overview
• 03:49 – Terraform101 and Vault101 Announcements
• 07:07 – Demo Starts
• 11:52 – Secrets in the State File
• 12:39 – Important Closing Remarks

Overview

To accomplish his task, Andrew considers a couple of options:
1. He could use a multi-purpose language such as Python to get the secrets from AWS Secrets Manager and populate them in HashiCorp Vault
2. Use a wide-spread domain-specific language such as Terraform to do the same task

While both options are valid, he considers the expertise within his platform engineering team and finds that his team is more comfortable with Terraform. Moreover, they are adopting a multi-cloud strategy. They just started adding apps to Azure and within 6 months the dev team will build some apps in Google cloud to leverage GCP’s machine learning services. He wanted to encourage his team to continue working with Terraform.

One downside to using Terraform for this task is that the secrets will show up in Terraform’s state file. He needs to plan for this. He decides to use Terraform Cloud to store the state file securely. Once the secrets are moved successfully he can destroy the Terraform workspace to remove all traces of these secrets.

DISCLAIMER:

Money Leaves Bank is a fictitious bank created for demo purposes.

Money Leaves Bank Logo

Pre-requisites

The following is required to follow along:

• A free GitHub account
• Access to an AWS account, we’ll be running within the 12 months free tier
• A Vault server that you could self-host – OSS is fine (I’ll use HCP Vault for this)
• A free Terraform Cloud account (you could run this locally on your computer using TF OSS, but be careful with the statefile as it will contain the secrets)

Secrets in AWS Secrets Manager

For the purpose of this demo, we will create a couple of secrets in AWS Secrets Manager.

1. Go to AWS Secrets Manager and create a secret of type: Other type of secret.
   Make sure to create a Key/value secret. This migration won’t work with Plaintext. I have 2 keys and values here: foo/bar and zig/zag
   \

2. Click Next and put: samg-aws-migration-vault as the secret name.
   

3. Click Next then Store leaving the defaults.

4. Click on Store a new secret
   

5. Repeat the above steps to create a second secret called samg-aws-migration-vault2 with 2 keys and values: foo2/bar2 and zig2/zag2

HashiCorp Vault Setup

For this demo, I’m using HashiCorp Cloud Platform (HCP) Vault. You can also use HCP vault, you get some credits when you sign up. Alternatively, you can use a self-hosted enterprise Vault or Vault open source. Just make sure that it is reachable from Terraform cloud.

As you can see in the 2 screenshots below, I have the secret/ path running K/V version 2 secrets engine under the admin namespace.
There are currently no secrets under the secret/ path.

Vault Secrets Engines View

No Secrets Yet

Terraform Cloud Setup

If you don’t have an account on Terraform Cloud (TFC), then go ahead and sign up for one here.

1. Fork the Repos

Make sure you fork my GitHub repos.. One is a Terraform module and the other is an example to use the module.

2. Create a New Workspace

Once you log in, navigate to the Workspaces tab and create a new workspace using the button at the top right.

3. Choose the Version Control Workflow

There are 3 workflows, choose the version control workflow.

4. Connect to a Version Control Provider

If you haven’t already connected to GitHub, you can do so in this step. The steps are straight forward and you can find a guide here.

Now click on the GitHub button and choose your equivalent forked repo. Mine is: samgabrail/terraform-vault-secretsmanager-to-vault-migration.

5. Finalize the Workspace Creation

Optionally add a description and then Click on Create workspace button.

6. Add Variables

Click on the Go to workspace overview button at the top right.

Go to the variables tab and click on Add variable

Put secret_names as the key and a list of the secret names (not the secret values themselves) that live in AWS secrets manager as the value. For my demo, I’m using: ["samg-aws-migration-vault", "samg-aws-migration-vault2"].

Remember to check the HCL box and optionally the Sensitive button. You can also add a variable description. Then hit the Save variable button.

Add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to access your AWS environment. You will need to create a user under IAM in the AWS console. This allows Terraform to access the AWS API.

Add VAULT_ADDR and VAULT_TOKEN also as environment variables. These are used to access Vault. The token used should have enough privileges to write to the k/v secrets engine at the secret/ path. In my demo I use the admin token I get from HCP vault but this is a big NO NO in production.

Pro Tip: Use Variable sets in TFC so you can easily add AWS and Vault credentials to this and future workspaces without having to recreate these variables for each workspace.

Populate Variables

Using Variable Sets

Vault Variable Set

AWS Variable Set

7. Start a New Terraform Run

Click on the Actions button and Start new run then start run

8. Check the Plan

Take a look at the plan results.

9. Confirm and Apply

This will terraform apply the plan. Verify that the apply completed successfully

10. Examine the State File

Click on the State tab and notice that the secrets values show up here. It’s very important to understand this. There are methods in TFC to restrict access to the state file to certain teams.

Secrets Migrated to Vault

Now let’s check the secrets in Vault.

Secrets Revealed

Please note that this only copies the secrets from AWS Secrets Manager to HashiCorp Vault and doesn’t delete the secrets from AWS Secrets Manager.

Terraform Configuration

Let’s touch on some highlights of the Terraform configuration.

Example Repo

You can see how we call the Terraform module below in our main.tf file.

module "secretsmanager-to-vault-migration" {
  source          = "samgabrail/secretsmanager-to-vault-migration/vault"
  version         = "0.0.4"
  secret_names    = var.secret_names
  aws_region      = var.aws_region
  vault_kv_path   = var.vault_kv_path
  vault_namespace = var.vault_namespace
}

The only required variable is secret_names which is a list of the names of the AWS secrets that we saw before. The other 3 variables have the following defaults:
– aws_region: us-east-1
– vault_kv_path: secret
– vault_namespace: admin

The Terraform Module

I published the Terraform module into Terraform’s public registry. You can access it directly as we did in our demo or import it into the private module registry of TFC to use it within your organization.

You can find the code for both the example repo and the Terraform module in our code section.

Below are the main configuration points:

Terraform Data Blocks for AWS Secrets Manager

We first need to get the secrets from AWS secrets manager. To do that we need to use a data block in Terraform. We also use for_each to iterate over the list of secrets in the list variable secret_names.

data "aws_secretsmanager_secret" "mysecret" {
  for_each = toset(var.secret_names)
  name     = each.value
}

data "aws_secretsmanager_secret_version" "mysecret" {
  for_each = toset(var.secret_names)
  secret_id = data.aws_secretsmanager_secret.mysecret[each.value].id
}

Terraform Vault Provider Configuration

Once we get the secrets, we now need to store them in Vault. Here is the terraform vault generic secret that allows us to add the secret into Vault’s K/V secrets engine:

resource "vault_generic_secret" "mysecret_in_vault" {
  for_each = toset(var.secret_names)
  path = "${var.vault_kv_path}/${each.value}"
  namespace = var.vault_namespace
  data_json = data.aws_secretsmanager_secret_version.mysecret[each.value].secret_string
}

If using a Vault open-source cluster then the namespace attribute is ignored.

Conclusion

In this post, we followed along Andrew’s solution to migrating secrets from AWS secrets manager to HashiCorp Vault. We saw how he used Terraform for this task and the considerations he had to take.

If you’d like to learn more about Terraform or Vault, I have a beginners 101 course for each that gets you ready to take the associate exams. You can find them in the references below.

Finally, if you would like to check out a better migration strategy that doesn’t use Terraform, check out my blog post Migrate Secrets from AWS Secrets Manager to HashiCorp Vault with Python, Docker, and GitLab.

References

• Code for this post
• Terraform 101 – Certified Terraform Associate Course
• Vault 101 – Certified Vault Associate Course

Suggested Reading

• AWS Lambda – Terraform Example with API Gateway
• HashiCorp Vault API Tutorial and Pro Tips
• Terraform Import Example – AWS EC2 Instance
• HashiCorp Vault PKI Secrets Engine Demo for Certificate Management
• HashiCorp Vault Backup and Restore Raft Snapshots from Kubernetes to AWS S3

Code

Related Posts

• Kyverno Tutorial: A Native Policy Engine for Kubernetes
• Distributed Fragments for Secure Secrets Management
• HolmesGPT: A Kubernetes Troubleshooting Assistant

• Secure Software Development with Coder | Part 3/3
• Optimize Cloud Development Costs with Coder | Part 2/3
• Mastering Configuration as Code with Configu