Terraform 1.4.6 - Failed to create AWS S3 buckets - The server-side encryption request configuration was not found

Terraform Providers AWS
1

Hi guys,

Due to some compliance reasons, we’re being vendor-locked to a local cloud provider who is providing an S3 compatible service.

I’m trying to create a bucket there, but it keeps failing saying

Error: getting S3 Bucket encryption: ServerSideEncryptionConfigurationNotFoundError: The server-side encryption request configuration was not found.
        status code: 404, request id: 1685976515119001 , host id:  12391987

My Terraform code for this looks like below

provider "aws" {
  access_key                    = "<ACCESS_KEY>"
  region                        = "us-east-1"
  secret_key                    = "<SECRET_KEY>"
  s3_use_path_style             = true
  skip_credentials_validation   = true
  skip_metadata_api_check       = true
  skip_requesting_account_id    = true

  endpoints {
    s3  = "https://DOMAIN-PROVIDED-BY-THE-PROVIDER.com"
    sts = "https://DOMAIN-PROVIDED-BY-THE-PROVIDER.com"
  }
}

resource "aws_s3_bucket" "example" {
  bucket = "example-may-2023"
}

However, the bucket was still created successfully. When I ran terraform apply again, it still failed with the same error, though.

Error: getting S3 Bucket encryption: ServerSideEncryptionConfigurationNotFoundError: The server-side encryption request configuration was not found.
        status code: 404, request id: 1685976743198164, host id: 12327843```

So I destroyed everything and tried again, trying to configure a aws_s3_bucket_server_side_encryption_configuration resource with an explicit dependency on the bucket as follows:

provider "aws" {
  access_key                  = "<ACCESS_KEY>"
  region                      = "us-east-1"
  secret_key                  = "<SECRET_KEY>"
  s3_use_path_style           = true
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    s3  = "https://DOMAIN-PROVIDED-BY-THE-PROVIDER.com"
    sts = "https://DOMAIN-PROVIDED-BY-THE-PROVIDER.com"
  }
}

resource "aws_s3_bucket" "example" {
  bucket = "example-may-2023"
}

resource "aws_s3_bucket_server_side_encryption_configuration" "example" {
  bucket = aws_s3_bucket.example.bucket

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }

  depends_on = [aws_s3_bucket.example]
}

Unfortunately, this one failed as well with the same error

Error: getting S3 Bucket encryption: ServerSideEncryptionConfigurationNotFoundError: The server-side encryption request configuration was not found.
        status code: 404, request id: 1685977250774298, host id: 12391987

Could you please help check if there is something wrong here? Or the problem mainly comes from our service provider?

Thank you very much for your help, guys.

2

Not compatible enough, it would seem!

Although, if you search for parts of your error messages in the terraform-provider-aws code, it shows up:

which appears to be doing some fragile-looking matching on exact strings returned in error messages, so this might be something that could be considered a bug in terraform-provider-aws, for depending overly on error strings, when it seems like a suitable error code exists.

3

Hi @maxb ,

Thank you very much for the reply

By this,

Not compatible enough, it would seem!

did you mean that their way of implementing the S3 service is not sufficient?

By the way, I forgot to mention that I could create the encryption manually.

Anyway, in my case, is it a bug in the AWS S3 plugin or from my service provider that is causing the issue in the first place please? If it’s because of the plugin, then I assume that I need to create a new topic in that group of the forum?

4

It looks like an issue with the service. The error is suggesting that the provider is making an API call to do with encryption but the service is returning an error.

5

I mean, that it’s not perfectly compatible, all the way to the exact text strings used in human-readable error messages. But, it’s debatable whether text strings used in human-readable error messages should be relied upon by computer code, so arguably it’s not the service provider’s fault.

As per the link to the code I posted, I feel that it’s a bug in terraform-provider-aws that it is insisting the human-readable text contain a specific string, when there appears to be sufficient meaning in the symbolic error code.

No, the forum is good for discussion, but once an actual problem that requires a code change has been identified, it should be raised in GitHub issues instead: Issues · hashicorp/terraform-provider-aws · GitHub

Yes, the service is returning an error, but the terraform-provider-aws code contains code designed to handle it, which is being unreasonably picky about matching the text in the human-readable error string, which seems to be slightly different in this non-Amazon implementation of the S3 API.

6

I agree that it’d be a good idea to report this as a bug in the AWS provider repository – as noted above, it seems like the machine-readable error code can provide the same information in a more robust way – but I also want to note that the hashicorp/aws provider is made for AWS only, and not tested with any other systems that try to emulate AWS behavior.

That doesn’t necessarily mean that the team won’t consider switching the error handling technique here – it seems like a good idea even for supporting real AWS – but in general you should expect that compatibility won’t be 100% if you aren’t using the real AWS.

Typically when working with another system, even if it’s advertised as being somewhat compatible with an AWS service, you should aim to use a provider designed specifically for that service. Of course, one might not exist yet if the system in question is not widely used, and so using the hashicorp/aws provider might be a pragmatic short-term workaround, but I would recommend to lock in an exact version constraint once you see it working because compatibility with third-party emulations of the API is not something the provider team will be testing for when making new releases.

7

That said, there is evidence in the code that some amount of effort has been made to support third party S3 implementations when discrepancies have been reported in the past:

and because Amazon got there first with S3, it’s quite probable there are vendors out there, for whom their S3-compatible API is their primary or only API - making the development of an entirely separate Terraform provider unlikely to be something that is ever pursued.

8

Hi @stuart-c @maxb , and @apparentlymart ,

Sorry for the late reply, and thanks a lot for your comments on this. I tried to rerun it under the DEBUG mode to collect additional info (Please see the attached text file
terraform-error.txt (51.1 KB)
).

I talked to the service provider’s support; they confirmed that they could create a bucket and enable encryption on it from the CLI without any issue. This is true because I can do it as well (please see the attached screenshot)

from_custom_s3_support
from_custom_s3_support1886×323 141 KB

I also showed them the log from which they confirmed the following:

the below operations are not implemented:
aws.operation=GetBucketWebsite
aws.operation=GetBucketAccelerateConfiguration
aws.operation=GetBucketRequestPayment
aws.operation=GetBucketLogging

Although those API calls are not needed for bucket/object encryption.

They also said that:

The issue here is that Terraform is initiating API called that are not needed to “create a bucket then enable bucket encryption on that bucket”. If you try to do so using awscli with “–debug” you won’t see any of those API calls mentioned in my previous message. So, the question should be why Terraform is initiating those un-related API calls? or how to configure Terraform to stop sending those API calls.

Could you please help check if that’s the case here?

I’m also thinking of downgrading the version of the terraform-provider-aws because if I recall correctly, recent versions of the plugin tries to enforce the encryption by default which I think may cause the issue, so maybe older versions help?

9

Nevertheless, these are things that the aws_s3_bucket resource is capable of managing, so it wants to query the current state of these things.

That is not something you can disable.

I believe you’re now desperately seeking solutions in the wrong place.

Fundamentally, your issue is that terraform-provider-aws expects the error message to contain encryption configuration was not found but your provider’s error message contains encryption request configuration was not found with the extra word “request” in there.

Should terraform-provider-aws be so picky about the contents of a human-readable error string? IMO, no, absolutely not. That is why I have suggested you open a bug report.

(Alternatively you could see if your service provider is willing to modify their error message to better match Amazon’s.)

10

Hi @maxb @stuart-c @apparentlymart ,

Thank you very much for your detailed feedback. I informed the provider about the finding, and as expected they said it would take their team many months to resolve it, which was not unexpected to me. :wink:

Meanwhile, I will report an issue in terraform-provider-aws Github repo as well. Let’s see who will fix it first.

Again, thank you very much for your help here.