Errors with upgrading aws-provider from v4.50 to v5.11

Terraform Providers AWS
1

Context

So I am upgrading terraform AWS provider so I can take advantage of python 3.11 lambda. The biggest issue is that I have to change aws_iam_policy_document.policy to aws_iam_policy_document.source_policy_documents which is a list of strings and not a string. ( I am using -- and ++ to show changes)

data "aws_iam_policy_document" "bucket" {
  count = local.policy_enabled ? 1 : 0

  -- source_json = var.policy
  ++ source_policy_documents =  var.policy

I then changed variable policy to a list of strings:

variable "policy" {
  description = "A bucket policy in JSON format."
  -- type        = string
  -- default     = null
  ++ type        = list(string)
  ++ default     = []
}

Added a bracket around policy on this loggin bucket module

module "ehl_telemetry_access_logging_s3_bucket" {
  source                                  = "./modules/infra/s3bucket"
  count                                   = var.ehl_telemetry_access_logging_s3_bucket.create ? 1 : 0
  bucket                                  = local.ehl_telemetry_access_logging_s3_bucket_name
  versioning                              = var.ehl_telemetry_access_logging_s3_bucket.versioning
  block_public_acls                       = var.ehl_telemetry_access_logging_s3_bucket.block_public_acls
  block_public_policy                     = var.ehl_telemetry_access_logging_s3_bucket.block_public_policy
  ignore_public_acls                      = var.ehl_telemetry_access_logging_s3_bucket.ignore_public_acls
  restrict_public_buckets                 = var.ehl_telemetry_access_logging_s3_bucket.restrict_public_buckets
  policy                                  = [replace(replace(replace(jsonencode(var.ehl_telemetry_access_logging_s3_bucket.policy), "VAR_S3_BUCKET", local.ehl_telemetry_access_logging_s3_bucket_name), "VAR_ACCOUNT_ID", data.aws_caller_identity.current.account_id), "VAR_REGION", var.aws_region)]
  apply_server_side_encryption_by_default = {kms_master_key_id: module.kms_key.keyarn, sse_algorithm: "aws:kms"}
  lifecycle_rules                         = var.ehl_telemetry_access_logging_s3_bucket.lifecycle_rules
  tags                                    = merge(local.common_tags, var.ehl_telemetry_access_logging_s3_bucket.tags)
}

The main issue is on this resource (original with just string type as policy):

resource "aws_s3_bucket_policy" "bucket" {
  count = local.policy_enabled ? 1 : 0

  bucket = local.bucket_id

  # Debugging outputs
  original_policy = local.original_policy
  policy_string = local.policy_string
  encoded_policy  = local.encoded_policy

  # remove whitespaces by decoding and encoding again to suppress terrform output whitespace cahnges
 policy = try(jsonencode(jsondecode(data.aws_iam_policy_document.bucket[0].json)), null)

  depends_on = [
    var.module_depends_on,
    aws_s3_bucket_public_access_block.bucket,
  ]
}

It is trying to use this iam_policy_document.

data "aws_iam_policy_document" "bucket" {
  count = local.policy_enabled ? 1 : 0

  source_policy_documents =  var.policy

  dynamic "statement" {
    for_each = local.cross_account_bucket_actions_enabled ? [1] : []

    content {
      actions   = var.cross_account_bucket_actions
      resources = [local.bucket_arn]

      principals {
        type        = "AWS"
        identifiers = var.cross_account_identifiers
      }
    }
  }

  dynamic "statement" {
    for_each = local.cross_account_object_actions_enabled ? [1] : []

    content {
      actions   = var.cross_account_object_actions
      resources = ["${local.bucket_arn}/*"]

      principals {
        type        = "AWS"
        identifiers = var.cross_account_identifiers
      }
    }
  }

  dynamic "statement" {
    for_each = local.cross_account_object_actions_with_forced_acl_enabled ? [1] : []

    content {
      actions   = var.cross_account_object_actions_with_forced_acl
      resources = ["${local.bucket_arn}/*"]

      principals {
        type        = "AWS"
        identifiers = var.cross_account_identifiers
      }

      dynamic "condition" {
        for_each = length(var.cross_account_forced_acls) == 0 ? [] : [1]

        content {
          test     = "StringEquals"
          variable = "s3:x-amz-acl"
          values   = var.cross_account_forced_acls
        }
      }
    }
  }

  dynamic "statement" {
    for_each = local.origin_access_identities_enabled ? [{
      actions   = ["s3:GetObject"]
      resources = ["${local.bucket_arn}/*"]
      }, {
      actions   = ["s3:ListBucket"]
      resources = [local.bucket_arn]

    }] : []
    content {
      actions   = statement.value.actions
      resources = statement.value.resources

      principals {
        type        = "AWS"
        identifiers = local.oai_identities
      }
    }
  }

  dynamic "statement" {
    for_each = local.elb_log_delivery ? [1] : []

    content {
      actions   = ["s3:PutObject"]
      resources = ["${local.bucket_arn}/*"]

      principals {
        type        = "AWS"
        identifiers = sort(local.elb_accounts)
      }
    }
  }
}

Attempts

  1. policy = try(jsonencode(jsondecode(data.aws_iam_policy_document.bucket[0][0].json)), null)
    Error: Only attribute access is allowed here, using the dot operator.

  2. policy = local.policy_enabled ? jsonencode(tolist(data.aws_iam_policy_document.bucket[0])) : null
    Error: data.aws_iam_policy_document.bucket[0] is object with 7 attributes
    Invalid value for “v” parameter: cannot convert object to list of any single

  3. policy = data.aws_iam_policy_document.bucket[0].json
    Error: MalformedPolicy: Missing required field Statement

  4. policy = data.aws_iam_policy_document.bucket.json
    Error:

Missing resource instance key
│ 
│   on modules/infra/s3bucket/main.tf line 202, in resource "aws_s3_bucket_policy" "bucket":
│  202:   policy = data.aws_iam_policy_document.bucket.json
│ 
│ Because data.aws_iam_policy_document.bucket has "count" set, its attributes
│ must be accessed on specific instances.
│ 
│ For example, to correlate with indices of a referring resource, use:
│     data.aws_iam_policy_document.bucket[count.index]
  1. policy = try(data.aws_iam_policy_document.bucket.json, null)
    Error:
Missing resource instance key
│ 
│   on modules/infra/s3bucket/main.tf line 202, in resource "aws_s3_bucket_policy" "bucket":
│  202:   policy = try(data.aws_iam_policy_document.bucket.json, null)
│ 
│ Because data.aws_iam_policy_document.bucket has "count" set, its attributes
│ must be accessed on specific instances.
│ 
│ For example, to correlate with indices of a referring resource, use:
│     data.aws_iam_policy_document.bucket[count.index]
  1. policy = local.policy_enabled ? data.aws_iam_policy_document.bucket[count.index].json : null
    Error: putting S3 policy: MalformedPolicy: Missing required field Statement

Any Ideas on what to try

2

Attempt 3 was correct.

Some of the others were weird.

However it seems that you somehow are getting JSON out of the datasource with no statements, so the next step is to look more closely at the inputs and output of the datasource, not make changes to the expression that comes after that in the flow of logic.

3

How can I view input and output like I’m debugging? To make this a little more annoying, I have to run this as part of a gitlab pipeline

4

I was able to run TFLOG=DEBUG and this is the result

json:
"{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailAclCheck",
      "Effect": "Allow",
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::-logging-XXX",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      }
    },
    {
      "Sid": "AWSCloudTrailWrite",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::-logging-XXX/*",
      "Principal": {
        "Service": "cloudtrail.amazonaws.com"
      },
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": [
            "bucket-owner-full-control"
          ]
        }
      }
    },
    {
      "Sid": "AWSServerLoggingAllow",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::-logging-XXX/*",
      "Principal": {
        "Service": "logging.s3.amazonaws.com"
      }
    },
    {
      "Sid": "AWSLogDeliveryAclCheck",
      "Effect": "Allow",
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::-logging-XXX",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": [
            "arn:aws:logs:us-east-1:XXX:*"
          ]
        },
        "StringEquals": {
          "aws:SourceAccount": [
            "XXX"
          ]
        }
      }
    },
    {
      "Sid": "AWSLogDeliveryWrite",
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3::access-logging-XXX/*",
      "Principal": {
        "Service": "delivery.logs.amazonaws.com"
      },
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": [
            "arn:aws:logs:us-east-1:XXX:*"
          ]
        },
        "StringEquals": {
          "aws:SourceAccount": [
            "XXX"
          ],
          "s3:x-amz-acl": [
            "bucket-owner-full-control"
          ]
        }
      }
    }
  ]
}"

Here is the aws_iam_policy_document.json and it does contain a STATEMENT but maybe I’m not understanding why the expression policy = data.aws_iam_policy_document.bucket[0].json is not working