Modify a resource created in another module

I have a standard_customers module for onboarding customer teams into Vault. It starts from a list variable that defines each customer, and then a series of for_each resources that create the engines, policies, entities, and assigns the expected policies to the entities.
How can I best handle arbitrary exceptions for policy assignment?
Example:
I have a variable that looks like this ( but will have a lot more members in the list). The policies, notably, are calculated values based on other fields I’m not showing here; they’re not a static map I can simply edit.

	"value": [
  {
	"app1_cms_dev": {
	  "app_boundary": "app1_cms_dev",
	  "app_name": "app1_cms",
	  "policies": [
		"app1_cms_pki_dev_r",
		"app1_cms_pki_shared_r"
	  ],
	},
	"app1_cms_prod": {
	  "app_boundary": "app1_cms_prod",
	  "app_name": "app1_cms",
	  "policies": [
		"app1_cms_pki_prod_r",
		"app1_cms_pki_shared_r"
	  ],	}  }	]

My resource creates roles and assigns policies:

resource "vault_aws_auth_backend_role" "aws_auth_reader" {
  for_each = local.aws_auth_roles

  provider  = vault.admin
  backend                  = vault_auth_backend.aws[each.value.auth_path].id
  role                     = each.key
  auth_type                = "iam"
  bound_iam_principal_arns = ["arn:aws:iam::${each.value.auth_id}:role/${each.value.rolename}"]
  token_policies           = each.value.policies[*]
}

I need to be able to override the token_policies for an arbitrary backend_role (or vault_identity_entity). If I create a static resource block, “terraform apply” will keep adding and removing/replacing the token_policies, so this clearly isn’t working as written.

resource "vault_aws_auth_backend_role" "app1_cms_dev_pki_issuer" {
  provider       = vault.admin
  backend        = "aws_auth_app1_dev"
  role           = "app1_cms_dev"
  token_policies = ["app1_cms_pki_dev_rw"]
}

Log output from multiple runs, with “tf state show” output after each run won’t fit in this post; I will post them in comments. You can see that the same TF files (i don’t make edits between runs) keeps flip-flopping between assigned policies.

Looking at this other question from a few years ago, am I just going down the wrong path? Do I definitely need to fix up the variable being operated on by the for_each to get this to work?

Sets the desired custom policy

PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> terraform apply
<snip>

Terraform will perform the following actions:

  # module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer will be updated in-place
  ~ resource "vault_aws_auth_backend_role" "app1_cms_dev_pki_issuer" {
      ~ bound_iam_principal_arns        = [
          - "arn:aws:iam::accesskey:role/AmazonEKSNodesRole",
        ]
        id                              = "auth/aws_auth_app1_dev/role/app1_cms_dev"
      ~ token_policies                  = [
          - "app1_cms_pki_dev_r",
          - "app1_cms_pki_shared_r",
          + "app1_cms_pki_dev_rw",
        ]
        # (23 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

2023-08-08T12:42:45.638-0700 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/terraform"]
module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer: Modifying... [id=auth/aws_auth_app1_dev/role/app1_cms_dev]
module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer: Modifications complete after 0s [id=auth/aws_auth_app1_dev/role/app1_cms_dev]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Outputs:

infra = <sensitive>
readers = <sensitive>
PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> vault read auth/aws_auth_app1_dev/role/app1_cms_dev
Key                               Value
---                               -----
allow_instance_migration          false
auth_type                         iam
bound_account_id                  []
bound_ami_id                      []
bound_ec2_instance_id             <nil>
bound_iam_instance_profile_arn    []
bound_iam_principal_arn           [arn:aws:iam::accesskey:role/AmazonEKSNodesRole]
bound_iam_principal_id            [iam_id_removed]
bound_iam_role_arn                []
bound_region                      []
bound_subnet_id                   []
bound_vpc_id                      []
disallow_reauthentication         false
inferred_aws_region               n/a
inferred_entity_type              n/a
resolve_aws_unique_ids            true
role_id                           d674ea67-9d6b-7f6a-87a6-434886feb6c8
role_tag                          n/a
token_bound_cidrs                 []
token_explicit_max_ttl            0s
token_max_ttl                     0s
token_no_default_policy           false
token_num_uses                    0
token_period                      0s
token_policies                    [app1_cms_pki_dev_rw]
token_ttl                         0s
token_type                        default
PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> terraform state show 'module.standard_customers.vault_aws_auth_backend_role.aws_auth_reader[\"app1_cms_dev\"]'
# module.standard_customers.vault_aws_auth_backend_role.aws_auth_reader["app1_cms_dev"]:
resource "vault_aws_auth_backend_role" "aws_auth_reader" {
    allow_instance_migration        = false
    auth_type                       = "iam"
    backend                         = "aws_auth_app1_dev"
    bound_account_ids               = []
    bound_ami_ids                   = []
    bound_ec2_instance_ids          = []
    bound_iam_instance_profile_arns = []
    bound_iam_principal_arns        = [
        "arn:aws:iam::accesskey:role/AmazonEKSNodesRole",
    ]
    bound_iam_role_arns             = []
    bound_regions                   = []
    bound_subnet_ids                = []
    bound_vpc_ids                   = []
    disallow_reauthentication       = false
    id                              = "auth/aws_auth_app1_dev/role/app1_cms_dev"
    resolve_aws_unique_ids          = true
    role                            = "app1_cms_dev"
    role_id                         = "d674ea67-9d6b-7f6a-87a6-434886feb6c8"
    token_bound_cidrs               = []
    token_explicit_max_ttl          = 0
    token_max_ttl                   = 0
    token_no_default_policy         = false
    token_num_uses                  = 0
    token_period                    = 0
    token_policies                  = [
        "app1_cms_pki_dev_r",
        "app1_cms_pki_shared_r",
    ]
    token_ttl                       = 0
    token_type                      = "default"
}
PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> terraform state show 'module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer'
# module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer:
resource "vault_aws_auth_backend_role" "app1_cms_dev_pki_issuer" {
    allow_instance_migration        = false
    auth_type                       = "iam"
    backend                         = "aws_auth_app1_dev"
    bound_account_ids               = []
    bound_ami_ids                   = []
    bound_ec2_instance_ids          = []
    bound_iam_instance_profile_arns = []
    bound_iam_principal_arns        = []
    bound_iam_role_arns             = []
    bound_regions                   = []
    bound_subnet_ids                = []
    bound_vpc_ids                   = []
    disallow_reauthentication       = false
    id                              = "auth/aws_auth_app1_dev/role/app1_cms_dev"
    resolve_aws_unique_ids          = true
    role                            = "app1_cms_dev"
    role_id                         = "d674ea67-9d6b-7f6a-87a6-434886feb6c8"
    token_bound_cidrs               = []
    token_explicit_max_ttl          = 0
    token_max_ttl                   = 0
    token_no_default_policy         = false
    token_num_uses                  = 0
    token_period                    = 0
    token_policies                  = [
        "app1_cms_pki_dev_rw",
    ]
    token_ttl                       = 0
    token_type                      = "default"
}

Removes the desired custom policy, puts the standard ones back

PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> terraform apply
<snip>
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following
symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.standard_customers.vault_aws_auth_backend_role.aws_auth_reader["app1_cms_dev"] will be updated in-place
  ~ resource "vault_aws_auth_backend_role" "aws_auth_reader" {
        id                              = "auth/aws_auth_app1_dev/role/app1_cms_dev"
      ~ token_policies                  = [
          - "app1_cms_pki_dev_rw",
          + "app1_cms_pki_dev_r",
          + "app1_cms_pki_shared_r",
        ]
        # (24 unchanged attributes hidden)
    }

  # module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer will be updated in-place
  ~ resource "vault_aws_auth_backend_role" "app1_cms_dev_pki_issuer" {
      ~ bound_iam_principal_arns        = [
          - "arn:aws:iam::accesskey:role/AmazonEKSNodesRole",
        ]
        id                              = "auth/aws_auth_app1_dev/role/app1_cms_dev"
        # (24 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.


2023-08-08T12:42:26.389-0700 [ERROR] AttachSchemaTransformer: No provider config schema available for provider["terraform.io/builtin/terraform"]
module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer: Modifying... [id=auth/aws_auth_app1_dev/role/app1_cms_dev]
module.standard_customers.vault_aws_auth_backend_role.aws_auth_reader["app1_cms_dev"]: Modifying... [id=auth/aws_auth_app1_dev/role/app1_cms_dev]
module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer: Modifications complete after 0s [id=auth/aws_auth_app1_dev/role/app1_cms_dev]
module.standard_customers.vault_aws_auth_backend_role.aws_auth_reader["app1_cms_dev"]: Modifications complete after 0s [id=auth/aws_auth_app1_dev/role/app1_cms_dev]

Apply complete! Resources: 0 added, 2 changed, 0 destroyed.

Outputs:

infra = <sensitive>
readers = <sensitive>
PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> vault read auth/aws_auth_app1_dev/role/app1_cms_dev
Key                               Value
---                               -----
allow_instance_migration          false
auth_type                         iam
bound_account_id                  []
bound_ami_id                      []
bound_ec2_instance_id             <nil>
bound_iam_instance_profile_arn    []
bound_iam_principal_arn           [arn:aws:iam::accesskey:role/AmazonEKSNodesRole]
bound_iam_principal_id            [iam_id_removed]
bound_iam_role_arn                []
bound_region                      []
bound_subnet_id                   []
bound_vpc_id                      []
disallow_reauthentication         false
inferred_aws_region               n/a
inferred_entity_type              n/a
resolve_aws_unique_ids            true
role_id                           d674ea67-9d6b-7f6a-87a6-434886feb6c8
role_tag                          n/a
token_bound_cidrs                 []
token_explicit_max_ttl            0s
token_max_ttl                     0s
token_no_default_policy           false
token_num_uses                    0
token_period                      0s
token_policies                    [app1_cms_pki_dev_r app1_cms_pki_shared_r]
token_ttl                         0s
token_type                        default
PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> terraform state show 'module.standard_customers.vault_aws_auth_backend_role.aws_auth_reader[\"app1_cms_dev\"]'
# module.standard_customers.vault_aws_auth_backend_role.aws_auth_reader["app1_cms_dev"]:
resource "vault_aws_auth_backend_role" "aws_auth_reader" {
    allow_instance_migration        = false
    auth_type                       = "iam"
    backend                         = "aws_auth_app1_dev"
    bound_account_ids               = []
    bound_ami_ids                   = []
    bound_ec2_instance_ids          = []
    bound_iam_instance_profile_arns = []
    bound_iam_principal_arns        = [
        "arn:aws:iam::accesskey:role/AmazonEKSNodesRole",
    ]
    bound_iam_role_arns             = []
    bound_regions                   = []
    bound_subnet_ids                = []
    bound_vpc_ids                   = []
    disallow_reauthentication       = false
    id                              = "auth/aws_auth_app1_dev/role/app1_cms_dev"
    resolve_aws_unique_ids          = true
    role                            = "app1_cms_dev"
    role_id                         = "d674ea67-9d6b-7f6a-87a6-434886feb6c8"
    token_bound_cidrs               = []
    token_explicit_max_ttl          = 0
    token_max_ttl                   = 0
    token_no_default_policy         = false
    token_num_uses                  = 0
    token_period                    = 0
    token_policies                  = [
        "app1_cms_pki_dev_r",
        "app1_cms_pki_shared_r",
    ]
    token_ttl                       = 0
    token_type                      = "default"
}
PS D:\git\company\HCPVault\HCPVault\environments\company-hcp-sre-dev\customers> terraform state show 'module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer'
# module.standard_customers.vault_aws_auth_backend_role.app1_cms_dev_pki_issuer:
resource "vault_aws_auth_backend_role" "app1_cms_dev_pki_issuer" {
    allow_instance_migration        = false
    auth_type                       = "iam"
    backend                         = "aws_auth_app1_dev"
    bound_account_ids               = []
    bound_ami_ids                   = []
    bound_ec2_instance_ids          = []
    bound_iam_instance_profile_arns = []
    bound_iam_principal_arns        = []
    bound_iam_role_arns             = []
    bound_regions                   = []
    bound_subnet_ids                = []
    bound_vpc_ids                   = []
    disallow_reauthentication       = false
    id                              = "auth/aws_auth_app1_dev/role/app1_cms_dev"
    resolve_aws_unique_ids          = true
    role                            = "app1_cms_dev"
    role_id                         = "d674ea67-9d6b-7f6a-87a6-434886feb6c8"
    token_bound_cidrs               = []
    token_explicit_max_ttl          = 0
    token_max_ttl                   = 0
    token_no_default_policy         = false
    token_num_uses                  = 0
    token_period                    = 0
    token_policies                  = [
        "app1_cms_pki_dev_rw",
    ]
    token_ttl                       = 0
    token_type                      = "default"
}

A quick side-note before we get onto the main topic… this syntax might be a misunderstanding, as if each.values.policies is always a list, the [*] is redundant and can be removed. The only way the [*] would be doing something useful here, is if some of your each.values.policies values are nulls or plain strings, which need to be transformed into lists at this point.


An analogy: this is similar to giving a robot of limited intelligence two commands: “Paint the wall blue if it is not already blue”, and “Paint the wall red if it is not already red”, and watching it enter an infinite loop trying to satisfy both commands simultaneously.

You absolutely do need to accomplish setting the correct value within the original resource block.

Fixing up the input data would be the best way of doing this. But, if that’s just too complicated to tackle right now, another option would be to change the

line to use a conditional expression.

  token_policies = try({
      "some_key" = ["some", "other", "policies"]
  }[each.key], each.value.policies[*])

would allow for selectively ignoring the large data structure and inserting ad-hoc overrides for particular elements of the for_each.

Thanks, I did end up fixing the underlying data structure to allow for an override value.
The logic that was creating the local.aws_auth_roles was originally doing this:
policies = lookup(local.all_reader_policies, "${app.app_name}_${boundary}")
I added a custom_reader_policy to my data structure and changed my logic to this

 policies = "${auth.custom_reader_policy}" != "" ? ["${auth.custom_reader_policy}"] : (
          lookup(local.all_reader_policies, "${app.app_name}_${boundary}")) // If there's no custom policy, use the calculated ones

Your comment about the wall-painting robots is spot on. I was hoping that adding depends_on would force it to work that way but alas.