I wonder if anyone has come across any performance issue when a list attribute is set with 400+ items?
One resource in our provider has an attribute that will be filled with lots of strings. Testing with <50 shows no performance issue. However, when our customer assigns 400+ strings for that attribute, Terraform does not finish processing.
With TF_LOG set to ‘debug’, I can see that Terraform took 7 minutes from start of the process to planning. Then the plugin stuck doing something (CPU at 145%+) when it encounters the attribute with lots of item. The plugin has been running for 100 minutes currently (with CPU at 145%+ the entire time) and no end in sight.
Is there something fundamental to Terraform core or plugin framework that induce this performance slowdown?
What can I do pinpoint the source of this issue?
I have TF configuration and debug logs to share if anyone is interested.
Yes, using large sets of values is going to be an inherently slow operation:
Depending on what exactly is slowing you down the most though, there have been a number of performance enhancements in the v1.10 branch, though I would guess you are seeing most of the time taken by the above issue.
It does look to be the likely culprit. The attribute in question in my provider is nested. See example in this GitHub issue.
Is there any workaround I can apply in the meantime? Would switching to TypeList (and forego the uniqueness validation of a set) help?
In the worst case scenario, I think making this attribute a TypeString and force the users to supply a JSON string of array would pretty much bypass this set validation.
Yes, a list would probably avoid the issue as long as the order of items can be made consistent. List items are identified by their index, so Terraform doesn’t need to compare every possible combination of elements, just each pair of elements at the same index.