On Terraform 1.4, the Terraform Plan showed no changes. Having recently upgraded to the latest v1.6 (didn’t change the AWS Provider which was a v4, but issue still exists on the latest v5), the plan shows (single example clipped):
# module.lambdas["queue_housekeeper"].data.aws_iam_policy_document.lambda_permissions_policy_document will be read during apply
# (depends on a resource or a module with changes pending)
<= data "aws_iam_policy_document" "lambda_permissions_policy_document" {
+ id = (known after apply)
+ json = (known after apply)
+ statement {
+ actions = [
+ "logs:CreateLogGroup",
+ "logs:CreateLogStream",
+ "xray:PutTelemetryRecords",
+ "xray:PutTraceSegments",
]
+ effect = "Allow"
+ resources = [
+ "*",
]
+ sid = "LambdaMonitoring"
}
+ statement {
+ actions = [
+ "logs:DescribeLogGroups",
+ "logs:DescribeLogStreams",
+ "logs:PutLogEvents",
+ "logs:PutRetentionPolicy",
]
+ effect = "Allow"
+ resources = [
+ "arn:aws:logs:*:*:*",
]
+ sid = "Logs"
}
+ statement {
+ actions = [
+ "logs:PutLogEvents",
]
+ effect = "Allow"
+ resources = [
+ "arn:aws:logs:*:*:log-group:/aws/lambda-insights:*",
]
+ sid = "LambdaInsights"
}
+ statement {
+ actions = [
+ "ec2:CreateNetworkInterface",
+ "ec2:DeleteNetworkInterface",
+ "ec2:DescribeNetworkInterfaces",
]
+ effect = "Allow"
+ resources = [
+ "*",
]
+ sid = "EC2"
}
+ statement {
+ actions = [
+ "ecr:Describe*",
+ "ecr:Get*",
+ "ecr:List*",
]
+ effect = "Allow"
+ resources = [
+ "*",
]
+ sid = "ECR"
}
}
# module.lambdas["queue_housekeeper"].aws_iam_policy.lambda_permissions_policy will be updated in-place
~ resource "aws_iam_policy" "lambda_permissions_policy" {
id = "arn:aws:iam::745249840871:policy/lambda_access_policy_for_queue_housekeeper"
name = "lambda_access_policy_for_queue_housekeeper"
~ policy = jsonencode(
{
- Statement = [
- {
- Action = [
- "xray:PutTraceSegments",
- "xray:PutTelemetryRecords",
- "logs:CreateLogStream",
- "logs:CreateLogGroup",
]
- Effect = "Allow"
- Resource = "*"
- Sid = "LambdaMonitoring"
},
- {
- Action = [
- "logs:PutRetentionPolicy",
- "logs:PutLogEvents",
- "logs:DescribeLogStreams",
- "logs:DescribeLogGroups",
]
- Effect = "Allow"
- Resource = "arn:aws:logs:*:*:*"
- Sid = "Logs"
},
- {
- Action = "logs:PutLogEvents"
- Effect = "Allow"
- Resource = "arn:aws:logs:*:*:log-group:/aws/lambda-insights:*"
- Sid = "LambdaInsights"
},
- {
- Action = [
- "ec2:DescribeNetworkInterfaces",
- "ec2:DeleteNetworkInterface",
- "ec2:CreateNetworkInterface",
]
- Effect = "Allow"
- Resource = "*"
- Sid = "EC2"
},
- {
- Action = [
- "ecr:List*",
- "ecr:Get*",
- "ecr:Describe*",
]
- Effect = "Allow"
- Resource = "*"
- Sid = "ECR"
},
]
- Version = "2012-10-17"
}
) -> (known after apply)
tags = {
"Name" = "queue_housekeeper"
}
# (5 unchanged attributes hidden)
}
# module.lambdas["queue_housekeeper"].aws_iam_role.lambda_role will be updated in-place
~ resource "aws_iam_role" "lambda_role" {
~ assume_role_policy = jsonencode(
{
- Statement = [
- {
- Action = "sts:AssumeRole"
- Effect = "Allow"
- Principal = {
- Service = "lambda.amazonaws.com"
}
- Sid = "LambdaAssumeRole"
},
]
- Version = "2012-10-17"
}
) -> (known after apply)
id = "lambda_role_for_queue_housekeeper"
name = "lambda_role_for_queue_housekeeper"
tags = {
"Name" = "queue_housekeeper"
}
# (8 unchanged attributes hidden)
}
This is occurring for every lambda function and on every deployment.
The volume of them is now essentially hiding anything really worth looking at due to the number of lines output.
If we run the plan after making zero changes after a deployment (which includes the same overload of issues), the same output, so something isn’t quite right here.
We have a “Terraform Plan Summary” which is a jq processing of the plan. The output in v1.4 of Terraform was as expected, empty, as nothing had changed. In v1.5 we see:
Plan Summary
============
Resource address Read Deleted Created Updated Trigger
---------------- ---- ------- ------- ------- -------
module.lambdas["email_notification"].data.aws_iam_policy_document.lambda_assume_role_policy_document *
module.lambdas["email_notification"].data.aws_iam_policy_document.lambda_permissions_policy_document *
module.lambdas["email_notification"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["email_notification"].aws_iam_role.lambda_role *
module.lambdas["qr_code_generator"].data.aws_iam_policy_document.lambda_assume_role_policy_document *
module.lambdas["qr_code_generator"].data.aws_iam_policy_document.lambda_permissions_policy_document *
module.lambdas["qr_code_generator"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["qr_code_generator"].aws_iam_role.lambda_role *
module.lambdas["queue"].data.aws_iam_policy_document.lambda_assume_role_policy_document *
module.lambdas["queue"].data.aws_iam_policy_document.lambda_permissions_policy_document *
module.lambdas["queue"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["queue"].aws_iam_role.lambda_role *
module.lambdas["queue_admin"].data.aws_iam_policy_document.lambda_assume_role_policy_document *
module.lambdas["queue_admin"].data.aws_iam_policy_document.lambda_permissions_policy_document *
module.lambdas["queue_admin"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["queue_admin"].aws_iam_role.lambda_role *
module.lambdas["queue_housekeeper"].data.aws_iam_policy_document.lambda_assume_role_policy_document *
module.lambdas["queue_housekeeper"].data.aws_iam_policy_document.lambda_permissions_policy_document *
module.lambdas["queue_housekeeper"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["queue_housekeeper"].aws_iam_role.lambda_role *
Completed
None of these things have changed. Nothing that they are dependent upon (names of resources mainly) have changed.
Nothing is using depends_on.
The plan summary output is there to help guide expectation. Upon reading the actual plan summary, nothing in it is helping.
If there IS a dependency, where in the plan is that information accessible? If we can extract that from the JSON data then we can compartmentalise the Plan Summary in some way (things we expect to change and things we know about that have not really changed unless they are actually present in the first set). By having a known “trigger” (as we call it) for the changes we currently see, then we can add additional processing to help render the list in a more meaningful way.
Some of the policies are hard-coded.
As this is only present in TF 1.6 and not dependent upon the AWS provider (a version v4 or a v5) … we are very stuck on what to do.
The “noise” being generated repeatedly is not useful.
So, as an exercise, I changed the aws_iam_role setup from:
data "aws_iam_policy_document" "lambda_assume_role_policy_document" {
statement {
sid = "LambdaAssumeRole"
effect = "Allow"
actions = ["sts:AssumeRole"]
principals {
identifiers = ["lambda.amazonaws.com"]
type = "Service"
}
}
}
resource "aws_iam_role" "lambda_role" {
name = format("lambda_role_for_%s", var.name)
assume_role_policy = data.aws_iam_policy_document.lambda_assume_role_policy_document.json
tags = {
Name = var.name
}
}
to
resource "aws_iam_role" "lambda_role" {
name = format("lambda_role_for_%s", var.name)
assume_role_policy = jsonencode({
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "lambda.amazonaws.com"
}
Sid = "LambdaAssumeRole"
},
]
Version = "2012-10-17"
})
tags = {
Name = var.name
}
}
and that solved the problem. The downside is that any issue with the the content of the JSON (a typo in a property name for example) is not going to be known until apply and, hopefully, AWS reject the policy. But this just feels wrong and would seem the whole point of a data "aws_iam_policy_document" "lambda_assume_role_policy_document".
Any insight would be REALLY good!!!
EDIT: I have looked into the JSON and found "action_reason": "read_because_dependency_pending". Added this to the Plan Summary and now get:
Plan Summary
============
Resource address Read Deleted Created Updated Trigger Reason
---------------- ---- ------- ------- ------- ------- ------
module.lambdas["email_notification"].data.aws_iam_policy_document.lambda_assume_role_policy_document * read_because_dependency_pending
module.lambdas["email_notification"].data.aws_iam_policy_document.lambda_permissions_policy_document * read_because_dependency_pending
module.lambdas["email_notification"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["email_notification"].aws_iam_role.lambda_role *
module.lambdas["qr_code_generator"].data.aws_iam_policy_document.lambda_assume_role_policy_document * read_because_dependency_pending
module.lambdas["qr_code_generator"].data.aws_iam_policy_document.lambda_permissions_policy_document * read_because_dependency_pending
module.lambdas["qr_code_generator"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["qr_code_generator"].aws_iam_role.lambda_role *
module.lambdas["queue"].data.aws_iam_policy_document.lambda_assume_role_policy_document * read_because_dependency_pending
module.lambdas["queue"].data.aws_iam_policy_document.lambda_permissions_policy_document * read_because_dependency_pending
module.lambdas["queue"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["queue"].aws_iam_role.lambda_role *
module.lambdas["queue_admin"].data.aws_iam_policy_document.lambda_assume_role_policy_document * read_because_dependency_pending
module.lambdas["queue_admin"].data.aws_iam_policy_document.lambda_permissions_policy_document * read_because_dependency_pending
module.lambdas["queue_admin"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["queue_admin"].aws_iam_role.lambda_role *
module.lambdas["queue_housekeeper"].data.aws_iam_policy_document.lambda_assume_role_policy_document * read_because_dependency_pending
module.lambdas["queue_housekeeper"].data.aws_iam_policy_document.lambda_permissions_policy_document * read_because_dependency_pending
module.lambdas["queue_housekeeper"].aws_iam_policy.lambda_permissions_policy *
module.lambdas["queue_housekeeper"].aws_iam_role.lambda_role *
Completed
The generator of this is:
TF_PLAN_JSON=test-reports/tf-plan.json
terraform show -json terraform.tfplan > $TF_PLAN_JSON
jq -r '
# Create an array of object from the following logic
[
# Delete any resource_change object that have a change action of "no-op".
del(.resource_changes[]?|select(.change.actions[0]|startswith("no-op")))
|
# Extract just the resource_change that remain
.resource_changes[]?
|
# Extract the required elements from each resource_change into a simpler object
{
"Resource address":.address,
"Read":(if .change.actions | contains(["read"]) then " *" else " " end),
# As a resource can be created and deleted, determine the position of the delete
"Deleted":(if .change.actions | contains(["delete"]) then
if .change.actions | length == 1 then
" *"
elif .change.actions[0] == "delete" then
" 1st"
else
" 2nd"
end
else
" "
end),
# As a resource can be created and deleted, determine the position of the create
"Created":(if .change.actions | contains(["create"]) then
if .change.actions | length == 1 then
" *"
elif .change.actions[0] == "create" then
" 1st"
else
" 2nd"
end
else
" "
end),
"Updated":(if .change.actions | contains(["update"]) then " *" else " " end),
"Trigger":(if .change.replace_paths?[0][0] then .change.replace_paths[0][0] else " " end),
"Reason":(if .action_reason then .action_reason else " " end)
}
]
|
if (.[0] | length) == 0 then
"No changes"
else
# Generate the column headings for the resultant table.
(
# Use the first object
.[0]
|
# Get the keys in their defined order
keys_unsorted
|
# For each key, create an underline and have these underlines in a new array (so, keys, then underlines).
(
.,
map(length*"-")
)
),
# Join on the objects to the set of column headings.
.[]
|
# Get just the values.
map(.)
|
# Output everything using tab separation which is the picked up by the column command that follows to make a nice
# tabular output.
@tsv
end
' $TF_PLAN_JSON | column -ts $'\t'