Background
Creating some JSON in a template file (table_mappings for aws_dms_replication_task)
The problem with template syntax
I had something like this, which was working fine until I didn’t want the trailing rule, then I ran into a trailing comma problem. Unfortunately you can’t access array length, so an inline conditional to get rid of the comma isn’t possible.
{
"rules": [
%{ for itablemapping, table_map in table_mappings}
%{ for itable, table in table_map.tables}
{
"rule-type": "selection",
# More props
},
{
"rule-type": "object-mapping",
"mapping-parameters": {
# Complex object
}
},
%{ endfor }
%{ endfor }
{
"rule-type": "old-rule", # Removed this rule
}
]
}
The problem with jsonencode and expression syntax
A couple things I read implied that jsonencode would ensure valid JSON, but it appears this is only true when inputting HCL; it does not strip out trailing commas, for example.
It looks like the recommended approach is jsonencode with HCL and expression syntax, so I tried that, but expression syntax for loops are severely limited, the biggest problem being that they can’t output anything except a flat object and I have complex objects. The documentation suggests dynamic blocks when it notes this limitation, but that does not apply to templates.
Workarounds
I tried numerous workarounds but was not able to find a way to generate valid JSON with complex objects in loops. For now I’m just using replace to find and remove the offending comma with regex. However, this feels dirty and may be difficult to maintain if the JSON changes.
Question
What is the correct way to create JSON with loops when you have nested objects?
Unfortunately I don’t have anything new to tell you that you haven’t already learned: the jsonencode function is the best way to generate valid JSON.
If you would like to discuss some details about how to construct a suitable value to pass into jsonencode to get the result you need, I’d be happy to do so. Some structures are easier to create than others, but I expect it should be possible to create a suitable structure somehow, if you can share what you tried before and how its result differed from what you needed.
Sure. Basically I want to do the same as above, but it doesn’t appear possible due to for loop limitations. Here’s an example of what I originally tried with most of the properties removed, just leaving a few to show structure:
This errors with Invalid 'for' expression. Between docs and experimenting I found that you can’t return multiple objects like that in a for, and more importantly you can’t return complex objects. For example, this works
[ for itablemapping, table_map in table_mappings :
{
"rule-type" = "selection"
}
]
But this does not.
[ for itablemapping, table_map in table_mappings :
{
{
"rule-type" = "selection"
}
}
]
The for that returns an object can’t do this either. So unless there’s something else we can use that I missed in the docs, I don’t see a clean way to create these types of JSON structures. If you have ideas I would love to hear them though.
Indeed, I think the crucial trick here is to keep generating as many nested lists as are needed to make the for expressions valid, and then rely on flatten to reduce all of that nesting into a flat list for the final result. flatten will reduce any number of nested lists or tuples and stop only once it finds something that isn’t a list or tuple, which means that in your case you should end up with a flat list of objects.
Working from your specific example, I expect it should work if you make the for expression results be either individual objects or lists of objects:
[ for itablemapping, table_map in table_mappings :
{
"rule-type" = "selection"
}
]
or
[ for itablemapping, table_map in table_mappings :
[
{
"rule-type" = "selection"
},
]
]
Thanks! Explaining the flatten behavior helped me get it right on the first try after realizing my mistake testing for the ability to handle nested objects.
Note that this process was much more difficult than with any other templating language I’ve used. A couple things I think would help:
Call out the nesting behavior earlier and more clearly in the documentation. It is only mentioned in a footnote at the end and says things like “You can’t dynamically generate nested blocks” which is part of what led me to believe I couldn’t have nested objects. Perhaps a small section that explains how to use them in templates (like you just did)?
I still consider the “extra list + flatten” solution a little hacky. Would it be possible to remove some of the limitations so that these loops work more like the template syntax loops? If jsonencode (and thus HCL + expression syntax) is supposed to be the go-to solution for generating JSON it really ought to be cleaner than this.
In a perfect world I could just do something like for i, thing in things : followed by any HCL, not needing a wrapper outside the for or for its content.
Failing that, any option that avoids the need for flatten would be very welcome. It’s a bit clumsy and not intuitive that we would need to add extra layers and flatten them just to generate some JSON blocks.
Not trying to be negative, very thankful for the help, just trying to give some feedback from the dev perspective.
Bear in mind that you are not using a templating language here, but an expression language.
On the plus side, this means you’re guaranteed to get syntactically valid JSON out the end. On the minus(?) side, since you’re no longer arbitrarily concatenating text and hoping it’s being joined syntactically correctly, the concatenation needs to be a bit more explicit.
Nested blocks are a specific part of HCL syntax, but you’re not working with any blocks here, so that doesn’t apply.
You can do that, followed by any HCL expression. Your case ends up being more complicated in two ways:
You have two dimensions to iterate over, and you want to join the output of that iteration into a single flat list - this is the first thing requiring flatten.
In an imperative programming language, you might avoid the use of an explicit flatten operation by appending items to a single output list as the iteration proceeds, but since Terraform only provides an expression language, not an imperative data manipulation language, you need the flatten. For that matter, in a templating language you could also skip any kind of explicit flatten operation by just outputting chunks of text as you go.
You want to produce two output objects for each step of your iteration (selection & object-mapping rules) - this is the second thing requiring flatten.
Again, in an imperative programming language, or templating language, the above-mentioned workarounds would apply for avoiding an explicit flatten.
But working with an expression language, there is no kind of expression for “some elements of a list, but not the whole list” - so you end up returning a two-element list, which then later gets flattened.
So there you have it - a simpler JSON generation situation wouldn’t have required flatten - it just happens that your particular use-case has two extra nuances that required it.
You hit the nail on the head as far as the two complications of my case go, and I think the reason this was more difficult than other times I’ve worked with templating languages is in fact because of the expression language.
I doubt I’m the first person to want to output two objects or iterate over two dimensions, so a cleaner or at least more obvious way of handling this would still be helpful; it just seems from my understanding of your explanation that it’s an inherent limitation of the expression language, so I’m not sure what that would look like.
Perhaps jsonencode could be made smart enough to remove trailing commas? Then the JSON could be created in the more straightforward template syntax, and just put through jsonencode to fix things like that. (I actually tried this approach when I was working on this and was disappointed to find it did not remove trailing commas.)
This doesn’t make sense, really… jsonencode’s input is objects, not a string. It receives no commas at all - just some structure of maps, lists, and scalar data types.
Indeed, I think this would amount to a function that takes some syntax that is a superset of JSON – let’s call it “JSON but with trailing commas allowed” – and converts it into valid JSON.
With that possibility in mind it’s interesting to note that YAML is already a superset of JSON and I believe it allows trailing commas, so it would also be a superset of our hypothetical “JSON but with trailing commas allowed”.
Therefore it might work to first parse the string as YAML and then immediately encode the result as JSON. You could still use “flow-style” YAML (the form that looks like JSON) rather than the whitespace-sensitive form of YAML.
If that works and it makes your particular situation easier to read then that could be a reasonable alternative, although I would suggest still using the expression language to construct values where possible because that way you are working with the full set of data types rather than just concatenating strings together. (Template for ... is essentially the same as ${ join("", [ for ... ]) }, implicitly concatenating all of the results together into a single string before returning it.)