We have developed a Terraform solution by creating smaller, modular components that are deployed across several AWS accounts to complete the solution.
Throughout this process, we encountered situations where certain modules depend on output values generated by other modules in different AWS accounts.
Now, we are seeking the best approach to read and utilize these cross-account Terraform output values in subsequent deployments.
Can anyone help on this, would be appreciated ?
It really depends.
Options include:
Hard-coding fixed values within the code/variables. This is pretty simple, but means any changes have to be manually updated anywhere they are used. Can be a good fit for things which are never likely to change.
The same as before, but the values contained within a reusable module. The advantage is that those hard-coded values are together, and therefore easier to manage. Depending how you manage modules you can also take advantage of versioning and automated updates.
The use of data sources to fetch live values from different accounts. Probably needs more complex access control policies (and likely multiple AWS provider configurations), but benefits from always being up to date. Useful for things which are likely to change fairly frequently.
The use of the remote state data source with output blocks. This way a root module exposes certain data which can then be consumed by other bits of Terraform. It will more closely couple things together, but sometimes that is desirable (clearer dependencies compared with data sources). You would need to have access to the source remote state, which could expose sensitive data (depending on the remote state type, and if it contains sensitive information).
In our situation we actually use a combination of all these options, as different cases fit each option better. For example we have a shared module which contains hard coded values such as AD group IDs, CIDRs of our internal office networks and other cross account static shared resources, but then we also use exposed output values and the remote state data source where things are more tightly coupled and specific.
Thank you for your valuable suggestions on this. I will take this up for sure.
Do you think Consul will help on this regard ?