I have an issue where my terraform code reads terraform_remote_state generated by another module.
In some cases this remote state do not exist yet and it will be created at later stage. In this case terraform exists with confusing error:
data "terraform_remote_state" "resource" {
backend = "s3"
config = {
bucket = "some_S3_Bucket"
key = "${var.account_name}/resource.tf"
region = "eu-west-1"
}
}
Error: AccessDenied: Access Denied
status code: 403, request id: 53AA619CF7AEAD08, host id: 0HnRlk=
Error: Unable to find remote state
on data.tf line 29, in data "terraform_remote_state" "resource":
29: data "terraform_remote_state" "resource" {
I’m not sure if there is a way that this code can continue even tho that key doesn’t exist in remote s3 bucket.
What I’m trying to achieve is to terraform to continue even tho remote tfstate file doesn’t exist (say mark it as Optional).
Probably something like conditionals to check if value exist would work here, but I cannot use conditionals like that inside datasource:
When you call this module, you can optionally set use_remote_state = false for situations where you don’t expect the remote state to exist yet.
A different way to solve this problem is to use module composition so that the data from the remote state is passed in to your module as a variable rather than having it be fetched directly.
For example, that guide shows a module that depends on the existence of a VPC but rather than going out and fetching the VPC itself it instead expects the caller to pass it in. The caller can then either pass an aws_vpc resource it is directly managing or a result from the aws_vpc data source, depending on whether the calling configuration is the one that manages this VPC.
You could do a similar thing with whatever data that might potentially be coming from this terraform_remote_state: rather than requesting it directly, instead declare a variable that expects to receive an object type that terraform_remote_state.resource.outputs would conform to and in configurations that run before that remote state exists, pass in the necessary data some other way instead, or perhaps set it to null if the data is optional and the module is able to operate without it.
Hi @apparentlymart,
Thanks for update. I do actually use Module Composition in this case.
I have a root module which does have that remote state and then output of that state is passed as variable to nested module for processing.
My problem is with remote backend which haven’t yet been created but it’s an dependency for this module.
I do not see how I could use count here since as you suggested count would basically enable/dissable this resource and then I would have to take decision if I want to enable or dissable it.
Above mentioned solution works great in scenario where you have say complex infrastructure made out of multiple resources (say s3, ec2, sns, dynamo, alb) and in some cases you want to have flexibitliy to dynamically enable dissable certain modules so one can have just 2 of those and other can have all and all you do is change few variables with True or False (0 or 1) and then that would work.
My case here is a bit different cause I do not want to manually control it.
In my case I have 3 portals which I’m building. Each of those have small or bigger dependency on other module output. Those outputs are being then taken as remote state from s3.
Problem here is that aws_terraform_remote_state resource will throw 2 errors. One saying that key do not exist (Wrongly Access Denied even tho you have 404 from AWS) and straight after that not being able to parse key of terraform_remote_state resource like shown in original post.
So what I’m looking for is option where aws_terraform_remote_state will continue without an error where key (object) in bucket do not exist. next step would be do so something like this in key portion of data or module declaration:
In above situation remote state finds missing key but contninues terraform execution and then I’m validating if remote state have any outputs in another resource declaration and take action depending on condition like above.
Example code:
data "terraform_remote_state" "resource" {
backend = "s3"
config = {
bucket = "some_S3_Bucket"
# Below key does not exist in S3 bucket
key = "${var.account_name}/resource.tf"
region = "eu-west-1"
}
}
module "module_name" {
module = "./somemodule"
somevar = length(data.terraform_remote_state.resource.outputs) != 0 ?data.terraform_remote_state.resource.outputs.some_output : ""
}
There is actually a defaults in terraform_remote_state resource but this seem to work only if key does in fact exist and doesn’t work if key is not present:
Indeed, what you are trying to do here isn’t really within the spirit of how Terraform is designed. Terraform wants you to specifically state for each situation whether or not this object is supposed to exist, rather than to dynamically select different behavior based on checking it.
In other words, the assumption is that in any specific situation you as the author of the configuration will decide statically how that data is to be obtained and will write the configuration to make that be true. If some situations require that information to be accessed a different way then Terraform expects you will write a different configuration for each scenario, describing those differences.
Input variables allow for a lighter-weight answer when the two situations are similar enough that they can share the same overall configuration and just tweak some values, but Terraform still expects that ultimately the author or the user will explicitly state what is supposed to happen, rather than having behavior emerge dynamically.
If you need more dynamic behavior then it’s possible that Terraform is not the best tool for the job here. You could potentially wrap Terraform in another bit of software of your own design that essentially automates the decision-making and sets variables accordingly (so that from Terraform’s perspective it’s still a fixed decision, but now being made by this other software rather than by a human operator) or you might find that it’s better to solve the whole problem with a different system that is designed for more dynamic decision-making.
One key thing that any alternative solution must address (and this is the primary reason Terraform doesn’t encourage dynamic decision-making) is what happens when you later apply another configuration that changes the decision. In that case, you might find that re-running the original configuration now wants to make a change, because the circumstances that decided the target infrastructure have now changed. You’d either need to ensure that the later creation of that remote state object doesn’t actually change the outcome, or to have your other program keep a persistent record of what decision it made so that it can guarantee to always make the same decision in the future, regardless of what else changes in the world.
You can control variable value outside the terraform (Eg bash script that checks file existence using aws CLI). somevar = var.use_remote_state ? data.terraform_remote_state.resource.outputs[0].some_output : ""
You can use the lookup function and check the outputs object for entries.
Uses the map, key (field that you know would be in the output) and default value.