[Solved] Vault agent - wrong aws iam principal?

Vault
1

Windows 2012R2, trying to run vault agent v1.11.0 (same? issue on 1.10.0):

[ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault-0b3fc5.mycorp.com:8200/v1/auth/aws/mycorp/login
  | Code: 400. Errors:
  |
  | * IAM Principal "arn:aws:iam::45x:user/jenkins.ec2" does not belong to the role "chef-client"

The agent’s auth config is

auto_auth {
  method "aws" {
  mount_path = "auth/aws/mycorp/"
    config = {
      type = "iam"
      role = "chef-client"
    }
}

I don’t understand why this user jenkins.ec2 is being picked up as the principal - or any user at all. My intent here, as with all my other boxes, is to use the instance itself as the principal. This instance has the same instance role profile as all of the others, so I don’t think it’s coming from there.

jenkins.ec2 is a valid IAM user, but I have no idea why the vault agent would be trying to use it? I guess this is as much an AWS/IAM question as it is a vault agent question?

Should also note that IAM instance authentication using the vault ruby gem against the server directly (no agent) works exactly as expected. For completeness, this is the ruby function:

      def auth_aws
        @logger.debug('Trying AWS authentication')

        m = Aws::EC2Metadata.new

        credential_provider = Aws::InstanceProfileCredentials.new(region: m.get('/latest/meta-data/placement/region'))
        @client.auth.aws_iam(VAULT_ROLE, credential_provider, nil, 'https://sts.amazonaws.com', VAULT_AUTH_PATH_AWS)

        @logger.debug('AWS authentication success')
      end
2

This was difficult to track down. It took decent amount of effort to finally get vault to reveal what it says above about jenkins.ec2 being the IAM authentication principal.

From there it was finding the right institutional knowledge about the history of this user, and that this particular host has the AWS sdk installed. The windows system-level environment vars had the SDK configured to authenticate as this user. Get rid of those aws sdk env vars, the vault agent is now behaving properly.

As much as I think vault is a great tool, troubleshooting when it doesn’t work right - especially the agent mode - is difficult at best.