"Error creating IAM user: InvalidClientTokenId" when running Terraform Plan

Vault
1

Hi, I am following this doc: Inject Secrets into Terraform Using the Vault Provider

When it comes to the operator-workspace section, when I try to run terraform plan, I get this error:

$ terraform plan
│ Error: error reading from Vault: Error making API request.
│ URL: GET http://<IP>:8200/v1/dynamic-aws-creds-vault-admin-path/creds/dynamic-aws-creds-vault-admin-role?role_arn=
│ Code: 400. Errors:
│ * Error creating IAM user: InvalidClientTokenId: The security token included in the request is invalid.
│ 	status code: 403, request id: 90463cfa-6b7e-4044-92ec-de2ba6847df6
│   with data.vault_aws_access_credentials.creds,
│   on main.tf line 20, in data "vault_aws_access_credentials" "creds":
│   20: data "vault_aws_access_credentials" "creds" {

The block at line 20 is taken directly from the doc:

data "vault_aws_access_credentials" "creds" {
  backend = data.terraform_remote_state.admin.outputs.backend
  role    = data.terraform_remote_state.admin.outputs.role
}

I tried a few suggestions I found online to no avail:

  1. Unset all AWS environment vars, reauthenticated AWS on Vault instance (as well as machine I’ve running Terraform from)
  2. Regenerated, re-set AWS credentials until I got a secret access key that did not contain special characters, such as “+”, “/”, “”.
  3. Setting my AWS_PROFILE, AWS_ACCOUNT environment variables
  4. Verified that the approle I’m using (created in the doc’s previous “vault-admin-workspace” section) is using a policy that has full access to path “dynamic-aws-creds-vault-admin-path/*”

Any suggestions on what else I could try?

2

The error message is telling you the AWS credentials you provided earlier to Vault in the vault-admin-workspace part of the guide, are incorrect.

Nothing you change in your environment variables at this stage is going to help - you need to go back to the “Vault Admin” initial setup portion, and supply correct credentials there.

3

I re-did the guide (starting from the Vault Admin section) and I’m getting the same result. So I’m successfully creating the dynamic-aws-creds-vault-admin-path AWS secrets engine, but again, it’s failing with an invalid token in the Terraform Operator section.

From the start, in the Vault Admin section, I pass in my access_key and secret_key, which I’m reading from ~/.aws/credentials.

export TF_VAR_aws_access_key=<aaa>
export TF_VAR_aws_secret_key=<bbb>

I’m authenticating to AWS via the saml2aws tool. I can run AWS CLI commands, like “aws s3 ls” or “aws describe-instances” without issue, so I thought that was confirming the credentials are valid.

4

I’m unfamiliar with saml2aws, but it sounds like the kind of tool which might produce temporary (3-part) AWS credentials, consisting of access key, secret key, and session/security token.

For more details: Temporary security credentials in IAM - AWS Identity and Access Management

The Vault AWS secrets engine is not able to use such credentials as root credentials, as there is no configuration option to supply the token: AWS - Secrets Engines - HTTP API | Vault by HashiCorp

1 Like
5

Thank you! Using credentials for an existing IAM user worked.

6

Sorry, I had a follow up. I’m trying to use an IAM Role rather than an IAM User now. But I’m using this block:

provider "vault" {
  auth_login {
    path = "auth/approle/login"
    parameters = {
      role = "<IAM_Role_Name>"
      role_arn = "arn:aws:iam::<123123123>:role/<IAM_Role_Name>"
      role_id   = var.login_approle_role_id
      secret_id = var.login_approle_secret_id
    }
  }
}

And hitting this error:

Error: error reading from Vault: Error making API request.
│ 
│ URL: GET http://<IP>:8200/v1/dynamic-aws-creds-vault-admin-path/creds/dynamic-aws-creds-vault-admin-role?role_arn=
│ Code: 400. Errors:
│ 
│ * Error assuming role: NoCredentialProviders: no valid providers in chain. Deprecated.
│ 	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
│ 
│   with data.vault_aws_access_credentials.creds,
│   on main.tf line 20, in data "vault_aws_access_credentials" "creds":
│   20: data "vault_aws_access_credentials" "creds" {

Does using an IAM Role require passing in an AWS security token? I tried that as well but it doesn’t seem to be working.

7

First, an aside - you should remove role and role_arn from the above. They are currently being ignored. The only valid parameters here are the ones accepted by AppRole - Auth Methods - HTTP API | Vault | HashiCorp Developer i.e. role_id and secret_id.

Now:

IAM roles - AWS Identity and Access Management :

Also, a role does not have standard long-term credentials such as a password or access keys associated with it. Instead, when you assume a role, it provides you with temporary security credentials for your role session.

is saying that, yes, an assumed role does include a security token, and

is showing that the Vault configuration API does not support that, but hints that if you just don’t configure the Vault AWS secrets engine, it’ll just call the AWS SDK and leave it up to the SDK’s default behaviours to hopefully figure out some credentials.

I have no experience with that, though.

1 Like