Andrew at Money Leaves Bank finally convinced Claire, his CIO, that HashiCorp Vault should be their secrets management solution as they are becoming a multi-cloud company. Now he is faced with the challenge of migrating their secrets hosted in AWS Secrets Manager to HashiCorp Vault. In this blog post, learn why Andrew decides to use Terraform for this task and how he implements the solution.
Video
Below is a video explanation and demo.
Video Chapters
In this video, we look at a demo of how to use Terraform to migrate secrets from AWS Secrets Manager to HashiCorp Vault.
You can skip to the relevant timestamps below:
- 00:00 – Introduction
- 00:17 – Scenario
- 01:36 – Terraform Public Module Overview
- 03:49 – Terraform101 and Vault101 Announcements
- 07:07 – Demo Starts
- 11:52 – Secrets in the State File
- 12:39 – Important Closing Remarks
Overview
To accomplish his task, Andrew considers a couple of options:
1. He could use a multi-purpose language such as Python to get the secrets from AWS Secrets Manager and populate them in HashiCorp Vault
2. Use a wide-spread domain-specific language such as Terraform to do the same task
While both options are valid, he considers the expertise within his platform engineering team and finds that his team is more comfortable with Terraform. Moreover, they are adopting a multi-cloud strategy. They just started adding apps to Azure and within 6 months the dev team will build some apps in Google cloud to leverage GCP’s machine learning services. He wanted to encourage his team to continue working with Terraform.
One downside to using Terraform for this task is that the secrets will show up in Terraform’s state file. He needs to plan for this. He decides to use Terraform Cloud to store the state file securely. Once the secrets are moved successfully he can destroy the Terraform workspace to remove all traces of these secrets.
DISCLAIMER:
Money Leaves Bank is a fictitious bank created for demo purposes.

Code
Pre-requisites
The following is required to follow along:
- A free GitHub account
- Access to an AWS account, we’ll be running within the 12 months free tier
- A Vault server that you could self-host – OSS is fine (I’ll use HCP Vault for this)
- A free Terraform Cloud account (you could run this locally on your computer using TF OSS, but be careful with the statefile as it will contain the secrets)
Secrets in AWS Secrets Manager
For the purpose of this demo, we will create a couple of secrets in AWS Secrets Manager.
- Go to AWS Secrets Manager and create a secret of type:
Other type of secret
.
Make sure to create aKey/value
secret. This migration won’t work withPlaintext
. I have 2 keys and values here:foo/bar
andzig/zag
Click
Next
and put:samg-aws-migration-vault
as the secret name.Click
Next
thenStore
leaving the defaults.Click on
Store a new secret
Repeat the above steps to create a second secret called
samg-aws-migration-vault2
with 2 keys and values:foo2/bar2
andzig2/zag2
HashiCorp Vault Setup
For this demo, I’m using HashiCorp Cloud Platform (HCP) Vault. You can also use HCP vault, you get some credits when you sign up. Alternatively, you can use a self-hosted enterprise Vault or Vault open source. Just make sure that it is reachable from Terraform cloud.
As you can see in the 2 screenshots below, I have the secret/
path running K/V version 2 secrets engine under the admin
namespace.
There are currently no secrets under the secret/
path.


Terraform Cloud Setup
If you don’t have an account on Terraform Cloud (TFC), then go ahead and sign up for one here.
1. Fork the Repos
Make sure you fork my GitHub repos.. One is a Terraform module and the other is an example to use the module.
2. Create a New Workspace
Once you log in, navigate to the Workspaces
tab and create a new workspace using the button at the top right.
3. Choose the Version Control Workflow
There are 3 workflows, choose the version control workflow.
4. Connect to a Version Control Provider
If you haven’t already connected to GitHub, you can do so in this step. The steps are straight forward and you can find a guide here.
Now click on the GitHub button and choose your equivalent forked repo. Mine is: samgabrail/terraform-vault-secretsmanager-to-vault-migration
.
5. Finalize the Workspace Creation
Optionally add a description and then Click on Create workspace
button.
6. Add Variables
Click on the Go to workspace overview
button at the top right.
Go to the variables tab and click on Add variable
Put secret_names
as the key and a list of the secret names (not the secret values themselves) that live in AWS secrets manager as the value. For my demo, I’m using: ["samg-aws-migration-vault", "samg-aws-migration-vault2"]
.
Remember to check the HCL
box and optionally the Sensitive
button. You can also add a variable description. Then hit the Save variable
button.
Add AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables to access your AWS environment. You will need to create a user under IAM in the AWS console. This allows Terraform to access the AWS API.
Add VAULT_ADDR
and VAULT_TOKEN
also as environment variables. These are used to access Vault. The token used should have enough privileges to write to the k/v secrets engine at the secret/
path. In my demo I use the admin
token I get from HCP vault but this is a big NO NO in production.
Pro Tip: Use
Variable sets
in TFC so you can easily add AWS and Vault credentials to this and future workspaces without having to recreate these variables for each workspace.




7. Start a New Terraform Run
Click on the Actions
button and Start new run
then start run
8. Check the Plan
Take a look at the plan results.
9. Confirm and Apply
This will terraform apply the plan. Verify that the apply completed successfully
10. Examine the State File
Click on the State
tab and notice that the secrets values show up here. It’s very important to understand this. There are methods in TFC to restrict access to the state file to certain teams.
Secrets Migrated to Vault
Now let’s check the secrets in Vault.

Please note that this only copies the secrets from AWS Secrets Manager to HashiCorp Vault and doesn’t delete the secrets from AWS Secrets Manager.
Terraform Configuration
Let’s touch on some highlights of the Terraform configuration.
Example Repo
You can see how we call the Terraform module below in our main.tf
file.
module "secretsmanager-to-vault-migration" { source = "samgabrail/secretsmanager-to-vault-migration/vault" version = "0.0.4" secret_names = var.secret_names aws_region = var.aws_region vault_kv_path = var.vault_kv_path vault_namespace = var.vault_namespace }
The only required variable is secret_names
which is a list of the names of the AWS secrets that we saw before. The other 3 variables have the following defaults:
– aws_region: us-east-1
– vault_kv_path: secret
– vault_namespace: admin
The Terraform Module
I published the Terraform module into Terraform’s public registry. You can access it directly as we did in our demo or import it into the private module registry of TFC to use it within your organization.
You can find the code for both the example repo and the Terraform module in our code section.
Below are the main configuration points:
Terraform Data Blocks for AWS Secrets Manager
We first need to get the secrets from AWS secrets manager. To do that we need to use a data
block in Terraform. We also use for_each
to iterate over the list of secrets in the list variable secret_names
.
data "aws_secretsmanager_secret" "mysecret" { for_each = toset(var.secret_names) name = each.value } data "aws_secretsmanager_secret_version" "mysecret" { for_each = toset(var.secret_names) secret_id = data.aws_secretsmanager_secret.mysecret[each.value].id }
Terraform Vault Provider Configuration
Once we get the secrets, we now need to store them in Vault. Here is the terraform vault generic secret that allows us to add the secret into Vault’s K/V secrets engine:
resource "vault_generic_secret" "mysecret_in_vault" { for_each = toset(var.secret_names) path = "${var.vault_kv_path}/${each.value}" namespace = var.vault_namespace data_json = data.aws_secretsmanager_secret_version.mysecret[each.value].secret_string }
If using a Vault open-source cluster then the namespace
attribute is ignored.
Conclusion
In this post, we followed along Andrew’s solution to migrating secrets from AWS secrets manager to HashiCorp Vault. We saw how he used Terraform for this task and the considerations he had to take.
If you’d like to learn more about Terraform or Vault, I have a beginners 101 course for each that gets you ready to take the associate exams. You can find them in the references below.
Finally, if you would like to check out a better migration strategy that doesn’t use Terraform, check out my blog post Migrate Secrets from AWS Secrets Manager to HashiCorp Vault with Python, Docker, and GitLab.
References
- Code for this post
- Terraform 101 – Certified Terraform Associate Course
- Vault 101 – Certified Vault Associate Course