Cannot login to vault with IAM instance profile: error validating X-Vault-AWS-IAM-Server-ID header

I’m starting to learn how to authenticate to vault with an IAM instance profile, but stopped by an error:

* error validating X-Vault-AWS-IAM-Server-ID header: missing header "X-Vault-AWS-IAM-Server-ID"

I followed these steps in the documentation:

On the vault server, I skipped the part running

vault write auth/aws/config/client secret_key=vCtSM8ZUEQ3mOFVlYPBQkf2sO6F/W7a5TVzrl3Oj access_key=VKIAJBRHKH6EVTTNXDHA```

Since it says it would use the IAM profile if the secret key is not defined. just writing no args won’t work.

vault write auth/aws/config/client
Must supply data or use -force

so I presume no entry is required then, since it says it would use the IAM profile if the secret key is not defined. The instance profile of the vault server mathces the profile provided by the instructions, it includes ec2:DescribeInstances and iam:GetInstanceProfile, so we should be ok to skip this.

I followed all the other steps, and I can get the client instance to login with a root token no problem, (just to verify that a connection is infact possible) using the FQDN address resolved by consul in a private subnet.

export VAULT_ADDR='https://vault.service.consul:8200'

…this will allow me to login via other methods.

There is however a problem with iam_server_id_header_value. I can update the value:

[ec2-user@ip-10-4-101-166 ~]$ vault write auth/aws/config/client iam_server_id_header_value=vault.service.consul
Success! Data written to: auth/aws/config/client

I hope the private FQDN here is acceptable, AWS wont be aware of it.

Doing this though, I cannot login with the IAM instance profile…

[ec2-user@ip-10-4-101-166 ~]$ vault login -method=aws region=ap-southeast-2 role=dev-role-iam
Error authenticating: Error making API request.

URL: PUT https://vault.service.consul:8200/v1/auth/aws/login
Code: 400. Errors:

* error validating X-Vault-AWS-IAM-Server-ID header: missing header "X-Vault-AWS-IAM-Server-ID"

So the only difference between the steps documented and what I did specifically were:

  • entering the specific role ARN for the instance profile.
  • using vault.service.consul for the private name.
  • skipping vault write auth/aws/config/client since it should be aquired by the vault server instance profile.

Based on the error what else can I check to determine what I might have configured incorrectly?

To see if I could get any further for testing sake, I tried reverting the ID header to the default empty string, although I really don’t want to do that.

vault write auth/aws/config/client iam_server_id_header_value=""
Success! Data written to: auth/aws/config/client
vault login -method=aws region=ap-southeast-2 role=dev-role-iam
Error authenticating: Error making API request.

URL: PUT https://vault.service.consul:8200/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Credential should be scoped to a valid region, not 'ap-southeast-2'. </Message>
  </Error>
  <RequestId>a34129b4-6c3f-422a-bde5-5b01b11c05fc</RequestId>
</ErrorResponse>

Apparently I can’t use region=ap-southeast-2, but I’m not sure why. removing the region does work though.

[ec2-user@ip-10-4-101-166 ~]$ vault login -method=aws role=dev-role-iam
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

WARNING! The following warnings were returned from Vault:

  * TTL of "768h" exceeded the effective max_ttl of "500h"; TTL value is
  capped accordingly

Key                      Value
---                      -----
token                    s.gaVtb3312CZ9IEjQkRXoHoX6
token_accessor           GyAcBX6BAgRezW2kXazYQidB
token_duration           500h
token_renewable          true
token_policies           ["default" "dev" "prod"]
identity_policies        []
policies                 ["default" "dev" "prod"]
token_meta_account_id    972620357255
token_meta_auth_type     iam
token_meta_role_id       aeb18575-238e-9ce0-5bd0-612090faf3de

I’m not sure why removing the region is needed.

Ahhh so the region must be us-east-1 regardless of where you are. It seems then that if that is the only accepted region for the request, its superfluous and maybe just should be not provided at all.