Sporadic permission error when creating aws_flow_log for multiple VPCs

Terraform Providers AWS
1

I use the following terraform code in a separate vpc_flow terraform config to create aws_flow_logs (destination is the same S3 bucket for all VPCs) for all VPCs.

resource "aws_flow_log" "flow_log" {
  for_each = {
    for vpc in data.terraform_remote_state.vpc_state.outputs.vpc_list : "${vpc.network_key}" => vpc
  }
  
  log_destination      = aws_s3_bucket.vpc-flow-bucket.arn
  log_destination_type = "s3"
  traffic_type         = "ALL"
  vpc_id               = each.value.network_id
}

When I run terraform apply on this , some times I get an error as follows, with some VPC’s flow logs created, and some not:

Error: Error creating Flow Log for (vpc-027c46aab58f8c7ee), error: Failed to set permission for LogDestination: 89556522452vpcdflowlogs

When I re-run the same script without any changes, it does run successfully for all VPCs. However, many times, the script runs the first time successfully, creating all the flow logs in the same S3 bucket. Is this a race condition of some kind while setting bucket-policy/ACL on target S3 bucket for multiple VPCs (as flow logs of all VPCs are targeted in the same bucket)?

2

I just came across this issue too. I believe the problem is to do with the bucket permissions for the flow logs - see the AWS documentation, particularly:

If the user creating the flow log owns the bucket, has PutBucketPolicy permissions for the bucket, and the bucket does not have a policy with sufficient log delivery permissions, we automatically attach the preceding policy to the bucket

Essentially, when you create the flow log resource if the bucket policy specified doesn’t exist, it tries to add it. While it’s being added, the flow log creation will fail.

To fix the issue, add the following policy when creating the bucket:

{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “AWSLogDeliveryWrite”,
“Effect”: “Allow”,
“Principal”: {“Service”: “delivery.logs.amazonaws.com”},
“Action”: “s3:PutObject”,
“Resource”: “arn:aws:s3:::bucket_name/*”,
“Condition”: {“StringEquals”: {“s3:x-amz-acl”: “bucket-owner-full-control”}}
},
{
“Sid”: “AWSLogDeliveryAclCheck”,
“Effect”: “Allow”,
“Principal”: {“Service”: “delivery.logs.amazonaws.com”},
“Action”: “s3:GetBucketAcl”,
“Resource”: “arn:aws:s3:::bucket_name”
}
]
}