Chicken and Egg: The terraform remote state S3 bucket cannot be idempotent if created by Terraform

Terraform
1

I want to share code with other accounts that allows people to automate a terraform deployment in AWS Cloud9.

One challenge I am wondering is, can terraform auto import resources?

Because to create the remote state bucket with terraform itself, unfortunately we cannot store the state of the terraform template that creates a remote state bucket in a remote state bucket itself!

Therefor if someone runs a template to create that bucket again in another instance, there will be a conflict. Is there a terraform only solution?

Alternatively I could switch to Ansible for this, but I’d prefer not to.
Here is the present example code from gruntwork that I use to create the bucket:

resource "aws_s3_bucket" "terraform_state" {
  bucket = "state.terraform.${var.bucket_extension}"
  acl    = "private"
  # Enable versioning so we can see the full revision history of our
  # state files
  versioning {
    enabled = true
  }
  # Enable server-side encryption by default
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
  tags = merge(
    {"description" = "Used for Terraform remote state configuration. DO NOT DELETE this Bucket unless you know what you are doing."},
    local.common_tags,
  )
}

resource "aws_s3_bucket_public_access_block" "backend" { # https://medium.com/dnx-labs/terraform-remote-states-in-s3-d74edd24a2c4
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "locks.state.terraform.${var.bucket_extension}"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}
1 Like
2

You can’t store/check state in a bucket that doesn’t yet exist.

The way I overcome this is to run the Terraform locally before the backend code has been created (so the bucket & DynamoDB [for locking] are created) producing a local state. I then add the backend code and run Terraform init again, which asks me if I want to transfer the local state into the bucket (which I do).

From then on everything is now happy.

3 Likes
3

That is pretty cool!

Is it possible to provide an arg instead of having to answer the prompt? That way initialisation could be automated too.

4

One problem I encountered attempting this and updating my init script with a backend:

# Init the s3 backend if it doesn't exist.
terraform init \
    -input=false \
    -backend-config="bucket=state.terraform.$TF_VAR_bucket_extension" \
    -backend-config="key=$TF_VAR_resourcetier/$namespace/terraform.tfstate" \
    -backend-config="region=$AWS_DEFAULT_REGION" \
    -backend-config="dynamodb_table=locks.state.terraform.$TF_VAR_bucket_extension"
terraform plan -out=tfplan -input=false

was suddenly hitting the error:

Warning: Missing backend configuration

-backend-config was used without a "backend" block in the configuration.

and if you add an s3backend.tf file with someting like:

terraform {
  backend "s3" {
    encrypt        = true
  }
}

Then you suddenly have some hacky config that isn’t idempotent.

I want to be wrong, but I dont think terraform is good at this.

Ansible might be sadly better suited to ensuring the backend exists regardless of state. Terraform should be able to do the same thing.

5

Im not sure if it related, but now I’m seeing:

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

module.iam_policies_s3_shared_bucket.data.aws_caller_identity.current: Refreshing state...
module.iam_policies_s3_shared_bucket.data.aws_s3_bucket.shared_bucket: Refreshing state...
data.aws_caller_identity.current: Refreshing state...
data.aws_iam_policy_document.multi_account_assume_role_policy: Refreshing state...

Error: Failed getting S3 bucket: NotFound: Not Found
        status code: 404, request id: W1N3EYJ3KJ83B0ZX, host id: a+v20RQQGJU6bPzbFTrYX73xtZyLSjZPM/8c2WG9pZw6ayFnSFhGrZmKaJzJpZiDZYG2Q3mV3JM= Bucket: "software.dev.firehawkvfx.com"

I tried deleting everything in my AWS account (except my cloud 9 instance that I’m launching from) but this problem persisted. My terraform template is supposed to create that bucket, so I dont know why it is complaining it doesn’t exist.

6

In case anyone comes across this again, adopting terragrunt into my workflow solved this problem, by its ability to auto configure remote state.

1 Like