I’ve been attempting for days to figure out why something I thought would be so simple turns out to be a nightmare of types and nested loops, and I finally need to ask for help. Here’s the situation:
I have several AWS S3 buckets, each created in one of multiple for_each looped resources with slightly different parameters as required. Up until now, each of those resource blocks contained a “policy” section that was identical to the one in every other s3 resource in question.
“Policy” inline with a bucket resource is now deprecated, and I’ve attempting to make a single for_each-enabled aws_s3_bucket_policy resource which finds and attaches the same policy to all of these several other buckets.
I know that for_each can be chained, and have done so before, but only with a one-to-one mapping.
I found the “merge()” function, which theoretically takes in several maps and returns one map with all of their members.
The documentation for for_each says that a resource using it references as a map of objects.
the error in the resulting plan shows that what really happened is the merge returned a single object whose contents were equal to the final overriding values in the last object in the list (as it documents that it does when merging objects).
I expected that merge would allow me to simply treat these multiple maps of objects as one, but something isn’t right, and since I don’t understand where my understanding deviates from reality, I also can’t make any progress using something like a for loop to construct an appropriate mapping, as is documented in plenty of places (but never, it seems, for this particular use case)
As a follow on to this statement, I’ve begun just throwing objectively incorrect code into the interpretter to see what the error messages tell me about the data structure. and after trying to, for a string, input the entire for_each resource reference just to see what it would say… I get the following error:
138: “Effect”: “Deny”,
│ 139: “Resource”: [
│ 140: “${aws_s3_bucket.images}”,
│ 141: “test/*”
│ 142: ]
│ 143: }
│ 144: ]
│ 145: }
│ 146: EOF
│ ├────────────────
│ │ aws_s3_bucket.images is object with 11 attributes
│
│ Cannot include the given value in a string template: string required.
Using merge also proves difficult when the keys of two of the types of buckets are identical (because each one needs two buckets) because it will overwrite with the second set as it merges based on keys. To get around that, I am attempting to use a for loop
locals {
allbuckets = flatten([
for bucketobject in [
aws_s3_bucket.static,
aws_s3_bucket.scms,
aws_s3_bucket.images,
aws_s3_bucket.background
] : {
for bucket in bucketobject : bucket.id => bucket
}
])
}
resource "aws_s3_bucket_policy" "secure_transport" {
for_each = local.allbuckets
bucket = each.key
The error that results from this is
| Error: Invalid for_each argument
│
│ on modules/s3/buckets.tf line 132, in resource "aws_s3_bucket_policy" "secure_transport":
│ 132: for_each = local.allbuckets
│ ├────────────────
│ │ local.allbuckets is tuple with 4 elements
I think that this and the previous merge example provides enough context for what I’m going for.
It sounds like the three S3 bucket resources you referred to in your example are just single-instance resources, without for_each set themselves, and so their values are just single objects rather than a map of objects. In that case, merge would try to meld all of their individual attributes into a single object, which I think is what you observed here if I understood correctly.
merge would be reasonable to use for multiple other resources that each have for_each set themselves, as long as their instance keys are all distinct, but for single objects you could instead construct a mapping directly.
for_each = {
a = aws_s3_bucket.a
b = aws_s3_bucket.b
}
As for why three results might be objects rather than maps: Terraform will often use an object as a mapping type, instead of a map, in situations where it can’t guarantee that all of the elements would have the same type. A map has only a single element type for all elements, whereas an object tracks a separate type for each attribute. Most Terraform language features will accept both as equivalent or convert when needed, so this is a pragmatic way to avoid lots of explicit type conversions in the common case. You can use explicit type conversions like tomap(...) if you’d like to ensure a particular kind of value in a given location, without relying on the implicit conversions.
I can 100% confirm that 4 of the ones I listed are each a for_each themselves. I mentioned that there were 5 in the original post, and I noticed that it was the 5th which was not (which explains the merge into single object with overrides from that resource). But when I removed that 5th resource and left the 4 remaining ones, the merge still breaks because of the duplicate keys across two of them. This leaves the for-loop (or some other remapping) necessary to bypass that.
My question is now how you would construct this as the for-loop required, because while it’s very useful knowledge I hadn’t thought of to turn the final resource into a map on the fly within the merge, that still leaves the key collision
locals {
allbuckets = [ for bo in [
aws_s3_bucket.static,
aws_s3_bucket.scms,
aws_s3_bucket.images,
aws_s3_bucket.background,
{"log-bucket" = aws_s3_bucket.log-bucket}] : {
for k, v in bo : v.id => v }
]
}
After constructing the above tuple, and using outputs to print it out and get a true understanding of what I’m working with, I know I’m very close. I now have a 5-tuple, each of which is an object containing several maps.
I now need all of those inner maps (2 layers in) to become one continuous list, set, whatever, of maps, rather than being nested like this. I know I can now call merge on the inner objects, but I can’t do it directly with them in the tuple.
Problem resolved! Thank you @apparentlymart for the push I needed to put this all together.
To create a fanned-in tuple of multiple resource attribute sets, both with and without for_each, creating new unique mapping so as to allow for parents which have the same keys for their maps:
locals {
allbuckets = flatten([ for bucketobject in [
aws_s3_bucket.static,
aws_s3_bucket.scms,
aws_s3_bucket.images,
aws_s3_bucket.background,
{"log-bucket" = aws_s3_bucket.log-bucket}] : {
for k, v in bucketobject : v.id => v }
])
}
To then call it in the for_each of a subsequent resource, allowing one child resource to multiple parent inheritance:
For those who come here without knowing yet, ... expands the tuple, so it is like I typed each of the 5 entries within it into the merge command individually.
One little note I have from reading your example is to watch out for exactly which attribute you choose to use as a map key. Although I’m not sure if this is true for aws_s3_bucket in particular, for many resource types the id attribute is something decided only during the apply step and so would not be a suitable to use as an instance key during the initial “create everything” step for a new instance of the module. (On the initial run when the objects don’t exist yet they won’t yet have known ids.)
If you’ve seen this work during create with this particular resource type then you can feel free to ignore this note and consider it as being for others who might see this topic in future but I would typically suggest using one of the attributes that was specified directly in the configuration, rather than an attribute decided dynamically by the provider or remote system, which therefore ensures it’ll always be known during planning. For example, bucket might be a good attribute to use for S3 buckets, assuming that the configuration uses a statically-decided name for each bucket rather than deriving the bucket name from other resource attributes.
but I would typically suggest using one of the attributes that was specified directly in the configuration, rather than an attribute decided dynamically by the provider or remote system, which therefore ensures it’ll always be known during planning. For example,