Module inter-dependency from another folder

Terraform
1

Hi @apparentlymart I am trying to achieve module inter-dependency amongst different folder levels.

Here’s what I want to achieve.

This is the folder structure:

org:
  modules:
    gcp_buket: 
        main.tf
        variables.tf
        data.tf
        outputs.tf
    gcp_service_account: 
        main.tf
        variables.tf
        data.tf
        outputs.tf
  environment:
    dev:
      bucket_module:
        main.tf (please refer [1] )
        outputs.tf
        variables.tf
      service_account_module:
        main.tf (please refer [2] )
        outputs.tf
        variables.tf

The above is the folder structure, where the org/modules have all the reusable code which has the “resource” types, and below are the “module” types re-using the above resources.

In the below bucket_module, I am creating two buckets.

[1] bucket_module: main.tf

module "bucket_1" {
source = "../../../org/modules/gcp_buket"
 a = var.a
 b = var.b
 ..... more variables
}

module "bucket_2" {
source = "../../../org/modules/gcp_buket"
 a = var.a
 b = var.b
 ..... more variables
}

and here I am creating two service account modules.

[1] service_account_module: main.tf

module "service_account_module_1" {
source = "../../../org/modules/gcp_service_account"
 a = var.a
 b = var.b
 ..... more variables
}

module "service_account_module_2" {
source = "../../../org/modules/gcp_service_account"
 a = var.a
 b = var.b
 ..... more variables
}

Now my question is how do I achieve module inter-dependency from another folder ?

basically I want to achieve the below.

module "bucket_1" {
source = "../../../org/modules/gcp_buket"
 a = var.a
 b = var.b
 ..... more variables

depends_on = [module.service_account_module_1]
}

but since the service_account_module_1, is in another folder I cannot do depends_on cause it says, module not found. so I tried using “data” source like below.

data "terraform_remote_state" "service_account" {
  backend = "local"
  config = {
    path = "../service_account/.terraform/terraform.tfstate"
  }
}

module "bucket_1" {
 source = "../../../org/modules/gcp_buket"
 service_account = data.terraform_remote_state.service_account.outputs.service_account.email
 a = var.a
 b = var.b
 ..... more variables

}

If I run terraform plan and apply inside the service_account folder, everything works. I can see the outputs as well, but when i try to run it at the root_level, it gives me the below error. Ideally it should wait for the service account to create it first, export the outputs, then pass it to the bucket, but since the service_account is yet to be created, and there are no outputs in the local “terraform.tfstate” , it fails.

│     │ data.terraform_remote_state.service_account.outputs is object with no attributes
│
│ This object does not have an attribute named "service_account_a_name".

Can you please advise how do i achieve this inter-module dependency, for the modules that are in a different folder. Thanks in advance.

2

It’s a little difficult to tell what you’re aiming for here, as you’re mixing incompatible concepts.

Each Terraform configuration consists of a root module (the directory in which you run the terraform command), and optionally other modules you reference via module blocks.

Terraform state files are one per configuration, or ROOT module. Therefore, it’s only possible to point a terraform_remote_state data source at a root module. For you to have that data source in your configuration, pointing at the service_account directory, yet be aiming to run Terraform from a different directory, won’t work.

I think what you want to do here is get rid of all use of terraform_remote_state in this configuration, and commit to only running terraform commands at the root level.

Then, you need to wire up the dependencies - but there’s no way to reach into a module from outside to depend on specific bits of it, you have to configure output variables to expose an attribute of the resource you want to depend on from the service account modules, up to the root module, and then pass them down as input variables to the bucket modules.

This might give you a useful introduction to dependencies in Terraform: https://developer.hashicorp.com/terraform/tutorials/configuration-language/dependencies

It would help a lot if you could give a more representative example of your configuration, including all resource blocks - that would make it a lot easier to recommend additions.

3

Hi Max,

Thanks for your reply. Perhaps I was not clear in explaining myself. I shall try again with the below example. I was not able to upload a .zip attachment on the forum, hence am sending it via email.

Basically I want to achieve inter-module dependency amongst different folder level. if you would be kind enough to checkout the attachment, you’ll see the modules folder - which are of types resource (reusable resources) and under “projects >> dummy_project >> stacks” you will see the modules.

currently I have “gcs” and “service_account” modules, which are using the above “modules” folder as source and are creating the infrastructure components.

Now I am using terragrunt, cause it helps me a lot in many ways, where in I can just run “terragrunt run-all plan” at root level and all the infra resources gets invoked. however, the challenge here is to specify dependencies that too on a specific module not on the entire folder. for eg, in the “service_account” folder I can have multiple modules for “service_account” , and same for the “gcs” folder, but I want to have a dependant relationship on “gcs-module-a” with the “service_account-module-a” , and not the entire folder. (I hope I’ve made my point here) . I can achieve the entire folder level dependency in terragrunt (https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#dependency ) but then it create unnecessary dependencies for all modules inside that folder, which i don’t want.

I only want “gcs module a” should have a implicit/explicit dependency on " service account module a" , irrespective of how many modules are present in each of that folder. so it means that, any other module say " gcs module b " or “service account module b or c or d” will NOT affect “module a”.

Can you please advise how can i achieve such kind of dependency ?

Also i want to highlight that i cannot use outputs or data cause the outputs are not available until the attributes are exported , so that terraform or terragrunt plan fails here.

What ideally it should do is that it should lookup for the dependant module, run that first, and then run the actual module.

This is really breaking our system, and i really need some advise how do i tackle this. Thanks a ton in advance.

edit : seems like even via mail i cannot upload zip, so here’s the git repo for the code.

(Attachment infrastructure.zip is missing)

4

Oh, now I finally understand where all the confusion is coming from.

You’re actually trying to configure dependencies in Terragrunt, a third party tool with its own dependency concept separate to Terraform’s dependencies.

Unfortunately, I’ve never used Terragrunt so I am unable to help you here.

However, I can take a pretty good guess that Terragrunt must, necessarily, only support setting dependencies between the various Terraform executions it manages… and so to cut a long story short, it appears that the dependencies you want are impossible, given how you’ve structured your composition of Terraform and Terragrunt.

As for where to go from here:

  1. I think you are using too many modules. When you have a module that contains only one resource block, and most of the module is just wiring variable and output blocks around that, it’s usually a sign that you’d be better off getting rid of the module entirely, and just inlining the resource block.

  2. When you create a resource, and then immediately use a data block to read back that same resource, it’s usually a sign you’ve made things unnecessarily complicated. It is very rare that this is a good pattern to use - usually only as a workaround for provider bugs. Normally, any information you need, you can read directly from the resource block.

  3. I can’t see why you want to have a dependency at all here. The google_service_account and the google_storage_bucket are independent.

  4. Even if you do need to separate Terraform configuration into multiple root modules, putting all resources of one type (service account) in one, and all resources of another type (bucket) in another, isn’t a separation that makes sense, in my opinion. If you have logical products you want to update separately, that is the separation to make.