Hi @jorhett,
For my part Iâm sorry that in my earlier communications I was unclear as to what weâd require to make a change here. Iâm going to try to elaborate here just for the sake of explaining myself, but I understand that youâre frustrated and thus may not be willing to engage with this further, and so Iâm not meaning this to suggest that you ought to do anything or ought to have done anything, just to give some more context.
When I say that something like this should be implemented without adding any additional options, I donât mean that we should find another existing option to attach it to but rather that we should design a new behavior that will unconditionally replace the old behavior and will better meet the design goals, by attempting to strike a better compromise between the different constraints.
In this case, it seems like the most significant conflicting requirements are:
- I infer from the PR that originally implemented these periodic announcements that this was intended to address a situation where Terraform can appear stuck/hung for particularly long-running operations such as creating an Amazon RDS instance. That PR was from before I joined the Terraform team and so I donât know what specifically prompted it, but I have to trust that Mitchell was doing this in response to some feedback that a long silence from Terraform during an operation was disconcerting, possibly leading to folks killing/interrupting the process.
- From your feedback here (and other similar feedback) we can see that there are situations where a notification every 10 seconds per resource instance can produce an excessive amount of notification messages, which can be particularly frustrating if you are already familiar with how long operations take for a particular configuration and so donât need the reassurance that Terraform is âstill workingâ.
The design challenge here, then, is to find a compromise that makes things tolerable (though likely not ideal) for both of these constituents, so that we can at least partially address various problems without creating various competing codepaths that would be a drag on future maintenence.
One simple change I could imagine here to start would be to extend the notification timeout to happen less often than every 10 seconds. I donât know what prompted Mitchell to select 10 seconds in his original proposal here, but I suspect it was a somewhat-arbitrary number chosen based on experience with how long commonly-used operations of that time took. With the benefit of a few more years of Terraform experience, we can try to determine what is a âtypicalâ length of time for an operation and try to set this timeout so that itâs more likely that it will appear only for the most egregious cases (like RDS) and even in those cases will generate less output, while still producing new output somewhat often to provide the comfort that Terraform hasnât got stuck.
I can also imagine some other designs that would likely strike a better compromise between the two goals but at the expense of a more complicated implementation. For example, the inferred requirement of periodically generating some output periodically for reassurance doesnât seem like it requires per-operation status updates. Perhaps the UI layer could roll up all of the pending operations into a single periodic message like â3 operations still in progressâŚâ, and thus reduce the total number of bytes written out while still giving the feedback that I expect the original change was aiming for.
Typically we start on something like this by doing research and development work rather than immediately writing code. Research can of course involve writing prototype code, but we typically donât engage with final implementation until weâve arrived at a design proposal that we believe strikes the best compromise between the competing goals. In my earlier responses on this topic I was intending to communicate that weâd be interested in that research but wouldnât be able to prioritize it yet.
This sort of open-ended iterative research prior to having a concrete proposal is, in our experience, not something that external contributors can typically commit to, and frankly we donât really have a great process for collaborating with folks outside of HashiCorp on design work of this kind even if they did want to.
It is something I personally have wanted to improve for some time, but Iâm sorry to say that itâs just another example of there only being so many hours in a day, and so we have to make some tough decisions about what to work on first.
I would love to spend time on researching and discussing different behaviors that better meet these needs, but right now everyone on the team is focused elsewhere and so regrettably this is a problem that needs to wait for later.
With all of that said, I understand and recognize your frustration, and I understand that the above will likely not make you any less frustrated, but I hope it will at least give you some context to understand the Terraform team behaviors youâve observed.