Failed to create the CloudWatch log–related resources due to an AccessDeniedException.

Terraform
1

Background

When using Terraform to create an ElastiCache replication group, we also need to create the associated engine log resources in CloudWatch. However, during the deployment, Terraform reported an error indicating that it failed to create these log-related resources(log group & resource policy). The error message is as follows:

│ Error: creating CloudWatch Logs Log Group (/aws/elasticache/xxx): operation error CloudWatch Logs: CreateLogGroup, https response error StatusCode: 400, RequestID: xxx, api error AccessDeniedException: User: arn:aws:sts::xxxx:assumed-role/xxxx-role/xxxx is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:ap-southeast-1:xxxx:log-group:/aws/elasticache/xxxx:log-stream: because no identity-based policy allows the logs:CreateLogGroup action
╷......
│ Error: creating CloudWatch Logs Resource Policy (xxxxx): operation error CloudWatch Logs: PutResourcePolicy, https response error StatusCode: 400, RequestID: xxxx, api error AccessDeniedException: User: arn:aws:sts::xxxx:assumed-role/xxxx-role/xxxx is not authorized to perform: logs:PutResourcePolicy on resource: arn:aws:logs:ap-southeast-1:xxx:log-group::log-stream: because no identity-based policy allows the logs:PutResourcePolicy action

Following is the code about the log group & resource policy:

resource "aws_cloudwatch_log_resource_policy" "elasticache_log_delivery_policy" {
  provider        = aws.location
  policy_document = data.aws_iam_policy_document.elasticache_log_delivery_policy.json
  policy_name     = "log-delivery-policy-xxxx"
}

resource "aws_cloudwatch_log_group" "engine-logs" {
    provider = aws.location
    name = "/aws/elasticache/xxxxx"
    log_group_class = "STANDARD"
    retention_in_days = "90"
}

The aws provider user executing the deployment has the two permissions mentioned above. However, there are IP restrictions in place—only requests originating from whitelisted IPs are allowed to perform these actions. The relevant permission policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Condition": {
                "Bool": {
                    "aws:ViaAWSService": "false"
                },
                "IpAddress": {
                    "aws:SourceIp": [
                        "1.1.1.1"
                    ]
                }
            },
            "Action": [
                "logs:CreateLogGroup",
                "logs:PutRetentionPolicy"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/elasticache/fb-*",
            ],
            "Effect": "Allow"
        }
    ]
}

We can also confirm that the public IP of the machine executing the Terraform commands is included in this whitelist.

I was just run 2 command: terraform init and terraform apply.

Some Findings

  1. Other AWS resources(elasticache subnet_group/parameter_group) are all create successfully.
  2. We also checked CloudTrail found that the recorded Source IP address for these evens(CreateLogGroup & PutResourcePolicy) are NOT the public IP of our local machine. For other AWS resource creation API calls, the recorded IP matches our local public IP correctly. Due to security concerns, I am unable to provide screenshots from CloudTrail.
  3. CloudTrail also shows the “User Agent” are all from HashiCorp, like APN/1.0 HashiCorp/1.0 Terraform/1.5.7 (+https://www.terraform.io) terraform-provider-aws/6.21.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.39.6 ua/2.1 os/linux lang/go#1.24.10 md/GOOS#linux md/GOARCH#amd64 api/cloudwatchlogs#1.58.9 m/i.
  4. After adding the above IP to the whitelist, both the CloudWatch log group and resource policy resources can be created successfully.

Questions

  1. Could you please help clarify this IP switching phenomenon? During Terraform execution, might there be any proxy or routing operation that causes the requests sent to AWS to originate from an IP different from my local public IP? I have reviewed extensive documentation but was unable to find an explanation.
  2. If Terraform itself does not perform any proxy or routing operations as mentioned above, could you please advise on other methods I might use to debug this issue?

I would be very grateful if anyone could help shed some light on the questions above. Many thanks!

2

After checking internally, we found this is not a Terraform issue.
Our IT team added a custom route for creating log-related resources, which changed the behavior compared to before and caused the confusion on our side.

sorry for the noise, I will close the issue.