Sharing state data across different terraform configurations using azure app configuration store

I’ve been reading many different opinions on how to share state between terraform configurations.

Currently refactoring a monolith into environments/components…

Google terraform best practices recommend to not use data sources for resources managed in other TF configurations as “it creates implicit dependencies on resource names and structures that normal Terraform operations might unintentionally break.”

They recommend terraform_remote_state. But I’ve also read mixed opinions on using that on the Hashicorp developer site as well, and recommendation to use data sources or other alternatives.

I’m currently experimenting with using Azure app configuration store to store key/value pairs of tfstate data I need to share across my configurations.

Each configuration pushes key/value (either as json for object type, or simple string) to config store.

I have a data-only module that abstracts the config store access and outputs specific key values which are referenced where/when needed in other configurations.

Is this a valid approach? Seems like it was suggested as valid in 2019… wondering if that is still the case in 2023.

Context: Using Terraform OSS on latest Azurerm provider (3.44.1) with Terraform cli v1.3.2

Like many things “it depends”.

There are various advantages and disadvantages of each option, and the decision of which to use (and you might choose to use more than one option) depends on the situation at your place of work, what you are doing & personal preference. The only thing I would strongly suggest is to be consistant - don’t do the same thing in different ways without there being a clear reason why.

The difference between using data sources & remote state is really the strength of the link between the two root modules. With remote state it is a very strong link - one root module exposes some outputs, which other root modules can then consume. Whereas for data sources there isn’t such an explicit link.

One advantage of having an explicit link (i.e. remote state) is you have clear links between root module, which you can more easily understand and document. A root module has to decide to expose something, whereas a data source could reference something that the producer doesn’t realise is being used/needed.

On the other had the advantage that data sources have is this looser association. This is particularly useful for example if something is produced that maybe isn’t managed by Terraform at all (and therefore remote state wouldn’t be possible) or is managed by Terraform but you shouldn’t have access to the root module’s state file (remote state requires access to the state file - for example S3 bucket - which includes more than just the exposed outputs, so could be a security issue in some situations). It is also useful when what is returned doesn’t matter as much - an example is where you just need the latest AMI instead of a specific one.

The idea of storing things in a totally different system is fairly similar to remote state - you have to explicitly send the data to that remote statem for it to be usable by someone else. One advantage is that only the shared information is available (instead of the whole state file), as well as the possibility of more granular access control, depending on what system you choose (so you could expose several items of information but restrict who can use them differently, whereas for remote state it is an all or nothing). The disadvantage is that there is additional complexity - you need to setup/maintain that extra system as well as any access control.

One other option you didn’t mention is just using hard coded values. This could be ARNs just included in a module as a string, or using a module that just contains such hard coded string/list values. With them being contained within a module you could then have loose versioning (such that you always get the latest version of that module). This can actually work pretty well for things which are pretty fixed - for example AWS account numbers or ARNs for a transit gateway (if they change it is likely to be due to some very major adjustment, which is unlikely and also likely to need other work anyway to accomodate).

Finally the other key thing you need to consider regardless of the option you use for sharing information between different root modules is how you perform changes. Within a single root module Terraform will produce a dependancy graph and figure out what is changing and the order to do things in. When you have coupled root modules that doesn’t exist, but there is still the same need.

You therefore need to have a way of rerunning moduleB if moduleA changes (assuming moduleB uses a value exposed by moduleA). This could be something totally manual (a process to run things in a certain order) or something more automated/complex (for example many CI systems can auto trigger other jobs once they complete).

1 Like

Thanks for the detailed reply!

Part of the problem too is I’m kinda stuck in a bit of analysis paralysis and need to commit.

So far, using azure app config store seems to be “working”… The one benefit I’m seeing is it decouples the outputs from underlying source. I could change how the value is populated without changing the key, so to no impact consumer configurations.

I get the hard-coded option as well… but I’m using the azure_caf provider to generate random names for me, using CAF rules which is kind of nice. I do have some hard-coded values in the provider, subscription ID (I’m deploying to 2 separate Azure subscriptions), some high level static strings in yaml files (apex domains etc…) that are pulled into locals where/when needed)

The other hard part is breaking out the monolith into distinct components that makes sense while keeping dependencies in check.

Currently, I have directories for each environment, with component subdirectories under each of those. My non-prod is provisioned differently than my prod, in that I’m sharing core infrastructure across all non-prod environments (app gateway, vnet/subnets, keyvault, app config, private dns zone etc.) The only distinct environment specific resources will be my linux webapp services.

Azure app gateway is limited in that it has to be provisioned entirely in one shot. Backend pools and http listeners can’t be added later as separate managed resources. Which means backend app services FQDNS need to be known when the app gateway is provisioned. Unless I want to get into complex helper bash scripts using the azure cli to directly manipulate parts of the app gateway, but that means TF won’t have any view into that particular state.

We have primarily gone with the remote state method.

It is nice that is clearly shows the dependencies between different root modules, although there is extra work involved as you have to remember to expose the output before it can be used. In our situation we aren’t too worried about having access to the full state file.

As to splitting things up for our main chunk of code we have split things based on ownership (some code is owned by a central team, while others is owned by various product dev teams), practicality (we don’t create a Kubernetes cluster & deploy into it in the same root module), number of environments (as we use workspaces as each environment is pretty much identical) and change cadence (we have a root module which just sets up the remote state S3 bucket and Jenkins IAM permissions which basically never changes once initially provisioned).