AWS IAM/EC2 Auth Login Failure - GovCloud

Vault
1

Hello everyone,

TL;DR

Vault aws auth login fails in GOV region with the following error:

Error authenticating: Error making API request.

URL: PUT https://vault.test.com:8200/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Credential should be scoped to a valid region, not 'us-east-1'. </Message>
  </Error>
  <RequestId>bce7b1ee-c30a-440a-8342-412221b5e9f4</RequestId>
</ErrorResponse>

I am trying to setup vault agent to cache secrets in the AWS Gov environment and I am doing it via Vault AWS Auth method, I have followed the steps mentioned here in this tutorial Vault agent caching tutorial and it works without any issues in the commercial environment.

Here is the auth/aws/config/client config:

Key                           Value
---                           -----
access_key                    n/a
endpoint                      n/a
iam_endpoint                  https://iam.us-gov.amazonaws.com
iam_server_id_header_value    vault.test.com
max_retries                   -1
sts_endpoint                  https://sts.us-gov-west-1.amazonaws.com
sts_region                    us-gov-west-1

The role that I am using to connect:

Key                               Value
---                               -----
allow_instance_migration          false
auth_type                         iam
bound_account_id                  []
bound_ami_id                      []
bound_ec2_instance_id             <nil>
bound_iam_instance_profile_arn    []
bound_iam_principal_arn           [arn:aws-us-gov:iam::12345678:role/test]
bound_iam_principal_id            [redacted]
bound_iam_role_arn                []
bound_region                      []
bound_subnet_id                   []
bound_vpc_id                      []
disallow_reauthentication         false
inferred_aws_region               us-gov-west-1
inferred_entity_type              ec2_instance
max_ttl                           800h
policies                          [vault-agent]
resolve_aws_unique_ids            true
role_id                           redacted
role_tag                          n/a
token_bound_cidrs                 []
token_explicit_max_ttl            0s
token_max_ttl                     800h
token_no_default_policy           false
token_num_uses                    0
token_period                      0s
token_policies                    [vault-agent]
token_ttl                         0s
token_type                        default

When I run this command:

vault login -method=aws header_value=vault.test.com role=vault-agent-iam -region=us-gov-west-1

I get the following error:

Error authenticating: Error making API request.

URL: PUT https://vault.test.com:8200/v1/auth/aws/login
Code: 400. Errors:

* error making upstream request: received error code 403 from STS: <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Credential should be scoped to a valid region, not 'us-east-1'. </Message>
  </Error>
  <RequestId>bce7b1ee-c30a-440a-8342-412221b5e9f4</RequestId>
</ErrorResponse>

I have verified everywhere possible and the region has been set to us-gov-west-1, I even exported the region before running the command → export AWS_REGION=us-gov-west-1 but to no avail.

And I checked the vault version across all servers it’s the same:

Vault v1.3.4

Is there anything else that I am missing?

Thanks,
Anant

2

There were some changes with this in 1.2.x and, I think, 1.4.1. With 1.3.4, quickest check since your config appears correct with sts_region and sts_endpoint, would be swap the 1.4.x latest or 1.5.0 binary and retest.
I feel this issue has shown up with gov-cloud as well as AWS China regions, if that helps your searching the issues.

3

@mikegreen - I tried it with vault 1.4.3, the error is still persistent. Also I don’t see any bug fixes for this in the CHANGELOG.

4

Very strange issue, when I ran vault agent via CLI:

vault agent -config=/etc/vault/conf.d/vault.hcl

The process started and I didn’t get any STS errors, this is the vault.hcl config:

listener "tcp" {
  address     = "localhost:8200"
  tls_disable = true
}

auto_auth {
   method "aws" {
       mount_path = "auth/aws"
       config = {
           type = "iam"
           role = "vault-agent"
           header_value = "vault.test.com"
           region = "us-gov-west-1"
       }
   }

   sink "file" {
       config = {
           path = "/tmp/vault-token-via-agent"
       }
   }
}

cache {
  use_auto_auth_token = true
}

vault {
  address = "http://vault.test.com:8200"
}
5

I was thinking
auth/aws: Fixes region-related issues when using a custom sts_endpoint by adding a sts_region parameter [GH-7922]
but oh well.
Might be worth tidying up a test for and logging and issue.

6

Has this issue fixed , I am getting this with version 1.6.2