S3 backend fails to initialize using profile

I have a very simple module using AWS (4.50.0) with terraform 1.3.7 with AWS credentials configured locally under a profile (e.g. “myprofile”)
When running with local state, all AWS commands use the profile correctly and succeed, however when I try to move the state to S3 using the same profile I get this error:

Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.

Please see Backend Type: s3 | Terraform | HashiCorp Developer
for more information about providing credentials.

Error: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors

The module looks basically like this, which works find in another project using an older terraform version (0.14)

provider "aws" {
  profile = "myprofile"
}

terraform {
  backend "s3" {
    bucket  = "mybucket"
    key     = "terraform/blah/foo.tfstate"
    region  = "us-west-2"
    profile = "myprofile"
    encrypt = true
  }
}

I had the same but pointing directly to the ‘credentials’ file helped.

backend "s3" {
    bucket = "tr-state-bucket"
    key    = "terraform.tfstate"
    region = "us-east-1"
    shared_credentials_file = "~/.aws/credentials"
    profile = "terraform"
  }

Thanks but I tried that and didn’t help. I’m actually going to see if I can file a bug after verifying that using any other profile works and even copying the values from any other profile into this one profile also works. The AWS CLI and the AWS terraform provider work, so it’s only the Terraform S3 backend with these particular values for access and secret key that fail.

The only thing I’ve noticed that’s different is that AWS is now generating secret keys with “+” signs in them. None of my other secret keys have this and all new ones I generate do, and they all fail. Adding to the confusion, it works for a coworker with + signs and when I copy the exact same lines into my credentials file it fails :roll_eyes:. Same OS, same file encoding and format. Different shell (bash vs zsh) but don’t know why that would affect Terraform.

Long story short… will file a bug for help. Working around it for now by setting credentials in the environment and the S3 backend uses that just fine.

More formal write up of findings so far

Repro Steps

  • Ubuntu 22.04 (WSL VM on Win11), bash shell, terraform 1.37, AWS provider 4.50.0
  • ~/.aws/credentials file with profile “xyz” which has full admin privileges + other profiles for other AWS accounts
  • Set aws provider to use profile “xyz” and local state
  • Create any AWS resources and apply, all operations success using “xyz” credentials
  • Change state to use “s3” backend with “xyz” profile
  • Attempt to reconfigure OR nuke local state and perform clean init

Expected

  • Terraform successfully initializes state on S3

Actual

  • Terraform returns “Error: error configuring S3 Backend: no valid credential sources for S3 Backend found”

Diagnostic Notes

  • Setting access and secret key via AWS_xxxx environment variables successfully initialized S3 backend. Access key and secret key are identical to those in the profile.
  • Using any other profile successfully detects and tries to use the profile credentials (of course fails with access denied, but at least the correct error), so it is not an issue trying to find or parse the credentials file.
  • Copying credentials from any other profile (or renaming any other profile “xyz”) again successfully detects and tries to use the profile credentials, so it is not an issue with the specific profile section definition in the credentials file.
  • Attempting to generate new keys for this account still fails. The only noticeable difference is that all these new keys have “+” signs in them, whereas none of my other secret keys do.
  • Co-worker with credentials for same account and secret key with “+” signs seems to work fine on their machine. Copying their profile info verbatim to my credentials file fails. Both using Ubuntu 22.04 VM, same terraform version, AWS provider version. Only known differences: I’m running VM via WSL on Windows with bash shell vs. VirtualBox on Mac with zsh shell

Created terraform issue Can't initialize S3 backend with AWS profile · Issue #32592 · hashicorp/terraform · GitHub