I have all my secrets in aws secrets manager including secrets for access to SASS platforms I use terraform to configure
Could I create a terraform credentials help that retrieves secrets from aws secrets manager and exports environment variables expected by terraform providers before execution?
I want terraform to get api keys, etc it need to auth to providers from aws secrets manager
I don’t see why not. You could either create a wrapper script which uses the AWS CLI, or you can do it directly within Terraform.
Similar to what we do with Vault you can fetch secrets from SSM using the aws_secretsmanager_secret_version data source (https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret_version) and then pass the value to provider blocks.
Note doing that will result in the secrets being stored in the state file, so you need to treat the state file accordingly.
Credentials helpers are only for Terraform-native services. Terraform providers typically integrate with the standard (Terraform-agnostic) authentication mechanisms for whatever system they are wrapping, and so to get a similar effect for AWS would require the AWS provider to implement something for it, hopefully based on something the AWS CLI already supports.
In the AWS docs I see Sourcing credentials with an external process which seems to be aimed at the same use-case as Terraform credentials helpers. I’m not sure if the Terraform AWS provider also supports it, but I do see some issues in the AWS provider repository talking about some details of it and so I expect it is supported.
yes this is the issue. this is for provider authentication and I dont want to store secrets in state
“Credentials helpers are only for Terraform-native services” this is what I wanted to confirm
AWS/GCP/Azure have idiomatic ways of easily configuring this but datadog/postgres/pagerduty/etc do not
Right now im writing a wrapper script that retrieves secrets, and exports them as env vars the providers expect before running terraform
The dream would be a direct integration with aws secrets manager. Where I would reference an aws secret for provider creds and then terraform would fetch it for me (first using my aws credentials file, env var, whatever, to authenticate to aws to read the secrets)
This is why I was asking if I could roll my own with a custom credential helper
Hi @red8888,
The usual approach I’ve seen when working with other users is to run Terraform in automation and have that automation run some other software prior to running Terraform in order to gather any necessary credentials, and then populate whatever environment variables or configuration files each provider expects before running Terraform.
In that case, Terraform itself is unaware of where exactly the credentials come from and that concern can be handled by any other software you choose.
In the specific design you described where AWS secrets manager contains the credentials needed for your other providers it seems like your wrapping automation would need to be preconfigured with some AWS credentials (perhaps in the standard AWS configuration file, or indirectly via an external process as I linked in my previous comment) which it could then use to access AWS Secrets Manager to dynamically fetch the datadog/postgres/pagerduty/etc credentials and populate environment variables like DD_API_KEY, PAGERDUTY_TOKEN, etc before running Terraform.
Another variant of this I’ve seen often is that organizations using HashiCorp Vault will give their automation a VAULT_TOKEN which has access to issue itself credentials for all of the other services, in which case it’s only the Vault token that needs to be pre-configured in the automation and everything else can be obtained and set dynamically in environment variables in the same way.
Because Terraform’s providers consume credentials the same way as other tools for a particular service typically would, it’s possible to use existing software that isn’t Terraform specific, such as aws-vault for AWS credentials.