Terraform Cloud + Vault + envconsul - Soooo Close to Working

Vault
, ,
1

I’m so close to making Terraform Cloud and Vault with envconsul work in a CI pipeline, but hitting one roadblock. Any ideas?

All secrets are in Vault including TF Cloud token. Now I need to get that to terraform to terraform init.

So far I have envconsul authenticating to Vault and able to pull a secret TF_CLI_CONFIG that has credentials "app.terraform.io" { token = "bla_bla_bla" }

Problem 1) terraform cannot pull config from an env variable. I tried TFE_TOKEN. Nope.

Problem 2) envconsul cannot write an env variable to a file. I tried envconsul echo > .terraformrc . echo doesn’t work, don’t know why.

2

I found one thing but seems to still run into the same problem with envconsul.

terraform init -reconfigure -backend-config="token=${TF_CLOUD_TOKEN}" seems to work.

But when trying to use envconsul to bring in the value of TF_CLOUD_TOKEN the value is blank. The same as envconsul echo $TF_CLOUD_TOKEN

envconsul -once -config .vault/envconsul.hcl terraform init -reconfigure -backend-config="token=${TF_CLOUD_TOKEN}"

3

I’m getting closer, but the problem has shifted.

This works!!! (with an exception)

export $(envconsul -once -pristine -config .vault/envconsul.hcl env)
terraform init -reconfigure -backend-config="token=${TF_CLOUD_TOKEN}"

My GCP creds are a big mess of json. They are passed in via GOOGLE_CREDENTIALS from Vault. No problem, but now the command export $(envconsul -once -pristine -config .vault/envconsul.hcl env) is throwing an error because of the multi-line output from GOOGLE_CREDENTIALS.

output of envconsul…

GOOGLE_CREDENTIALS='{
  "type": "service_account",
  "project_id": "bla-bla-bla",
  ....
}'

shell reports /bin/sh: export: line 117: "type":: bad variable name

4

This is what I settled on and it’s working . Maybe someone has better ideas. I would love to hear it.

export $(envconsul -no-prefix -once -pristine -secret=secret/infrastructure/terraform env)
alias terraform="envconsul -once -config .vault/envconsul.hcl terraform"
terraform init -reconfigure -backend-config="token=${TF_CLOUD_TOKEN}"
terraform plan
terraform apply

The things I don’t love about this are:

  1. I didn’t want to call out a secret in this script because I wanted all config to come from the file, so each project could define their own secret needs.
  2. To make execution easy, I use the alias. But this means that only the terraform command has access to the secrets unless you call out envconsul directly.
  3. I like envconsul for the ability to keep secrets a bit more save, vs. just putting them into an env var. This is not true for the TF_CLOUD_TOKEN.

I wonder why secrets are not available to the command being executed?

5

You should use consul-template to write to file.

6

Have you thought about enabling Vault GCP Secrets backend to enable dynamic secrets.

and then in the TF GCP provider you should use something like this:

data "vault_generic_secret" "gcp_creds" {
    path = "secret/gcp/roleset"
}

provider "google" {
  project          = "my-project-id"
  access_token     = data.vault_generic_secret.gcp_creds.data
}

Saves you from trying to work around the issue.