Empty merge after partial destroy

Hi all,

while trying to reproduce a different merge issue, I found the following behaviour: when merging the content of two resources, the final merge output is null when one of the resources is destroyed.

Terraform v0.13.2
+ provider registry.terraform.io/hashicorp/tls v2.2.0

Here is a rather simple example:

locals {
  tls_keylist1 = [
    { name = "test1-1", algorithm = "ECDSA" }
  ]
  tls_keylist2 = [
    { name = "test2-1", algorithm = "ECDSA" }
  ]
}

resource "tls_private_key" "tls_keys1" {
  for_each = { for key in local.tls_keylist1 : key.name => key }

  algorithm = each.value.algorithm
}

resource "tls_private_key" "tls_keys2" {
  for_each = { for key in local.tls_keylist2 : key.name => key }

  algorithm = each.value.algorithm
}

output "tls_keys" {
  description = "A list of all TLS key objects."
  value = [
    merge(tls_private_key.tls_keys1, tls_private_key.tls_keys2)
  ]
}

The first apply works as expected:

Terraform will perform the following actions:

  # tls_private_key.tls_keys1["test1-1"] will be created
  + resource "tls_private_key" "tls_keys1" {
      + algorithm                  = "ECDSA"
      + ecdsa_curve                = "P224"
      + id                         = (known after apply)
      + private_key_pem            = (sensitive value)
      + public_key_fingerprint_md5 = (known after apply)
      + public_key_openssh         = (known after apply)
      + public_key_pem             = (known after apply)
      + rsa_bits                   = 2048
    }

  # tls_private_key.tls_keys2["test2-1"] will be created
  + resource "tls_private_key" "tls_keys2" {
      + algorithm                  = "ECDSA"
      + ecdsa_curve                = "P224"
      + id                         = (known after apply)
      + private_key_pem            = (sensitive value)
      + public_key_fingerprint_md5 = (known after apply)
      + public_key_openssh         = (known after apply)
      + public_key_pem             = (known after apply)
      + rsa_bits                   = 2048
    }

Plan: 2 to add, 0 to change, 0 to destroy.

tls_private_key.tls_keys1["test1-1"]: Creating...
tls_private_key.tls_keys2["test2-1"]: Creating...
tls_private_key.tls_keys1["test1-1"]: Creation complete after 0s [id=470906a55ca3315736fbf0c2b354b48bce650f77]
tls_private_key.tls_keys2["test2-1"]: Creation complete after 0s [id=4796ce8160e990f2869376629260639a1855aab9]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

tls_keys = [
  {
    "test1-1" = {
      "algorithm" = "ECDSA"
      "ecdsa_curve" = "P224"
      "id" = "470906a55ca3315736fbf0c2b354b48bce650f77"
      "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHLbGY1zx+go8IQ8aHLRj5dtXY7xRzVZVTSI0wxagBwYFK4EEACGhPAM6\nAARZq3dayO4PrDVqfGNdWCW8lSsZ6tvBVURwj973mWZNB0yNoJzrfVdoWXOZ+nbf\nRMoHAbUTXeZLxg==\n-----END EC PRIVATE KEY-----\n"
      "public_key_fingerprint_md5" = ""
      "public_key_openssh" = ""
      "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEWat3WsjuD6w1anxjXVglvJUrGerbwVVE\ncI/e95lmTQdMjaCc631XaFlzmfp230TKBwG1E13mS8Y=\n-----END PUBLIC KEY-----\n"
      "rsa_bits" = 2048
    }
    "test2-1" = {
      "algorithm" = "ECDSA"
      "ecdsa_curve" = "P224"
      "id" = "4796ce8160e990f2869376629260639a1855aab9"
      "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHNnVi0rGsWDRZqF2WNg4afc0ebsdrTMewdB8A1igBwYFK4EEACGhPAM6\nAASSs8DsyAGJ9LmNAw51fKfYHlKt+GfbuhWbZfx37kZSGUUU1cNEwK485nZ3sRp1\nqZVdVjY7zRrxiA==\n-----END EC PRIVATE KEY-----\n"
      "public_key_fingerprint_md5" = ""
      "public_key_openssh" = ""
      "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEkrPA7MgBifS5jQMOdXyn2B5Srfhn27oV\nm2X8d+5GUhlFFNXDRMCuPOZ2d7EadamVXVY2O80a8Yg=\n-----END PUBLIC KEY-----\n"
      "rsa_bits" = 2048
    }
  },
]

But now one of the keys is removed:

locals {
  tls_keylist1 = [
#   { name = "test1-1", algorithm = "ECDSA" }
  ]
  tls_keylist2 = [
    { name = "test2-1", algorithm = "ECDSA" }
  ]
}

And then this happens:

Terraform will perform the following actions:

  # tls_private_key.tls_keys1["test1-1"] will be destroyed
  - resource "tls_private_key" "tls_keys1" {
      - algorithm       = "ECDSA" -> null
      - ecdsa_curve     = "P224" -> null
      - id              = "470906a55ca3315736fbf0c2b354b48bce650f77" -> null
      - private_key_pem = (sensitive value)
      - public_key_pem  = <<~EOT
            -----BEGIN PUBLIC KEY-----
            ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEWat3WsjuD6w1anxjXVglvJUrGerbwVVE
            cI/e95lmTQdMjaCc631XaFlzmfp230TKBwG1E13mS8Y=
            -----END PUBLIC KEY-----
        EOT -> null
      - rsa_bits        = 2048 -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  ~ tls_keys = [
      ~ {
          - test1-1 = {
              - algorithm                  = "ECDSA"
              - ecdsa_curve                = "P224"
              - id                         = "470906a55ca3315736fbf0c2b354b48bce650f77"
              - private_key_pem            = <<~EOT
                    -----BEGIN EC PRIVATE KEY-----
                    MGgCAQEEHLbGY1zx+go8IQ8aHLRj5dtXY7xRzVZVTSI0wxagBwYFK4EEACGhPAM6
                    AARZq3dayO4PrDVqfGNdWCW8lSsZ6tvBVURwj973mWZNB0yNoJzrfVdoWXOZ+nbf
                    RMoHAbUTXeZLxg==
                    -----END EC PRIVATE KEY-----
                EOT
              - public_key_fingerprint_md5 = ""
              - public_key_openssh         = ""
              - public_key_pem             = <<~EOT
                    -----BEGIN PUBLIC KEY-----
                    ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEWat3WsjuD6w1anxjXVglvJUrGerbwVVE
                    cI/e95lmTQdMjaCc631XaFlzmfp230TKBwG1E13mS8Y=
                    -----END PUBLIC KEY-----
                EOT
              - rsa_bits                   = 2048
            } -> null
            test2-1 = {
                algorithm                  = "ECDSA"
                ecdsa_curve                = "P224"
                id                         = "4796ce8160e990f2869376629260639a1855aab9"
                private_key_pem            = <<~EOT
                    -----BEGIN EC PRIVATE KEY-----
                    MGgCAQEEHNnVi0rGsWDRZqF2WNg4afc0ebsdrTMewdB8A1igBwYFK4EEACGhPAM6
                    AASSs8DsyAGJ9LmNAw51fKfYHlKt+GfbuhWbZfx37kZSGUUU1cNEwK485nZ3sRp1
                    qZVdVjY7zRrxiA==
                    -----END EC PRIVATE KEY-----
                EOT
                public_key_fingerprint_md5 = ""
                public_key_openssh         = ""
                public_key_pem             = <<~EOT
                    -----BEGIN PUBLIC KEY-----
                    ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEkrPA7MgBifS5jQMOdXyn2B5Srfhn27oV
                    m2X8d+5GUhlFFNXDRMCuPOZ2d7EadamVXVY2O80a8Yg=
                    -----END PUBLIC KEY-----
                EOT
                rsa_bits                   = 2048
            }
        },
    ]

tls_private_key.tls_keys1["test1-1"]: Destroying... [id=470906a55ca3315736fbf0c2b354b48bce650f77]
tls_private_key.tls_keys1["test1-1"]: Destruction complete after 0s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Outputs:

tls_keys = [
  null,
]

Key test2-1 still exists in the Terraform state, but the output is empty, until the next apply:

Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  ~ tls_keys = [
      - null,
      + {
          + test2-1 = {
              + algorithm                  = "ECDSA"
              + ecdsa_curve                = "P224"
              + id                         = "4796ce8160e990f2869376629260639a1855aab9"
              + private_key_pem            = <<~EOT
                    -----BEGIN EC PRIVATE KEY-----
                    MGgCAQEEHNnVi0rGsWDRZqF2WNg4afc0ebsdrTMewdB8A1igBwYFK4EEACGhPAM6
                    AASSs8DsyAGJ9LmNAw51fKfYHlKt+GfbuhWbZfx37kZSGUUU1cNEwK485nZ3sRp1
                    qZVdVjY7zRrxiA==
                    -----END EC PRIVATE KEY-----
                EOT
              + public_key_fingerprint_md5 = ""
              + public_key_openssh         = ""
              + public_key_pem             = <<~EOT
                    -----BEGIN PUBLIC KEY-----
                    ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEkrPA7MgBifS5jQMOdXyn2B5Srfhn27oV
                    m2X8d+5GUhlFFNXDRMCuPOZ2d7EadamVXVY2O80a8Yg=
                    -----END PUBLIC KEY-----
                EOT
              + rsa_bits                   = 2048
            }
        },
    ]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

tls_keys = [
  {
    "test2-1" = {
      "algorithm" = "ECDSA"
      "ecdsa_curve" = "P224"
      "id" = "4796ce8160e990f2869376629260639a1855aab9"
      "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHNnVi0rGsWDRZqF2WNg4afc0ebsdrTMewdB8A1igBwYFK4EEACGhPAM6\nAASSs8DsyAGJ9LmNAw51fKfYHlKt+GfbuhWbZfx37kZSGUUU1cNEwK485nZ3sRp1\nqZVdVjY7zRrxiA==\n-----END EC PRIVATE KEY-----\n"
      "public_key_fingerprint_md5" = ""
      "public_key_openssh" = ""
      "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEkrPA7MgBifS5jQMOdXyn2B5Srfhn27oV\nm2X8d+5GUhlFFNXDRMCuPOZ2d7EadamVXVY2O80a8Yg=\n-----END PUBLIC KEY-----\n"
      "rsa_bits" = 2048
    }
  },
]

Has anybody else seen this behaviour?

Here is a slightly modified example:

locals {
  tls_keylist1 = {
    "test1-1" = { algorithm = "ECDSA" }
  }
  tls_keylist2 = {
    "test2-1" = { algorithm = "ECDSA" }
  }
}

resource "tls_private_key" "tls_keys1" {
  for_each = local.tls_keylist1

  algorithm = each.value.algorithm
}

resource "tls_private_key" "tls_keys2" {
  for_each = local.tls_keylist2

  algorithm = each.value.algorithm
}

output "tls_keys" {
  description = "A list of all TLS key objects."
  value = [
    for tls_key in merge(
      { for name, key in tls_private_key.tls_keys1 : name => merge({ name = name }, key) },
      { for name, key in tls_private_key.tls_keys2 : name => merge({ name = name }, key) }
    ) : tls_key
  ]
}

Initial create:

Terraform will perform the following actions:

  # tls_private_key.tls_keys1["test1-1"] will be created
  + resource "tls_private_key" "tls_keys1" {
      + algorithm                  = "ECDSA"
      + ecdsa_curve                = "P224"
      + id                         = (known after apply)
      + private_key_pem            = (sensitive value)
      + public_key_fingerprint_md5 = (known after apply)
      + public_key_openssh         = (known after apply)
      + public_key_pem             = (known after apply)
      + rsa_bits                   = 2048
    }

  # tls_private_key.tls_keys2["test2-1"] will be created
  + resource "tls_private_key" "tls_keys2" {
      + algorithm                  = "ECDSA"
      + ecdsa_curve                = "P224"
      + id                         = (known after apply)
      + private_key_pem            = (sensitive value)
      + public_key_fingerprint_md5 = (known after apply)
      + public_key_openssh         = (known after apply)
      + public_key_pem             = (known after apply)
      + rsa_bits                   = 2048
    }

Plan: 2 to add, 0 to change, 0 to destroy.

tls_private_key.tls_keys1["test1-1"]: Creating...
tls_private_key.tls_keys2["test2-1"]: Creating...
tls_private_key.tls_keys1["test1-1"]: Creation complete after 0s [id=60f23246e6fafd9cc30738af759ac2faf1ad81a2]
tls_private_key.tls_keys2["test2-1"]: Creation complete after 0s [id=1b8b095e87c8dff3da1d14435ea8ab1de9a3f896]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

tls_keys = [
  {
    "algorithm" = "ECDSA"
    "ecdsa_curve" = "P224"
    "id" = "60f23246e6fafd9cc30738af759ac2faf1ad81a2"
    "name" = "test1-1"
    "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHFtK2da/oWF9JPmDMuYipT919M6UT+PoblG7tFWgBwYFK4EEACGhPAM6\nAAQXrGqNJEqjmBIJhY+Tzo9djOy9qr57HFVNVptXPuyXQx1l5u3k/X4XEAhif99K\nz9wJWJzEV7lxbw==\n-----END EC PRIVATE KEY-----\n"
    "public_key_fingerprint_md5" = ""
    "public_key_openssh" = ""
    "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEF6xqjSRKo5gSCYWPk86PXYzsvaq+exxV\nTVabVz7sl0MdZebt5P1+FxAIYn/fSs/cCVicxFe5cW8=\n-----END PUBLIC KEY-----\n"
    "rsa_bits" = 2048
  },
  {
    "algorithm" = "ECDSA"
    "ecdsa_curve" = "P224"
    "id" = "1b8b095e87c8dff3da1d14435ea8ab1de9a3f896"
    "name" = "test2-1"
    "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHP7u0NLwuCEBktmRoAjnI/nYK3si7e51+VtXdbmgBwYFK4EEACGhPAM6\nAAQq33Rgqa4y0k3wN1MJsuQIgCUFUO/wVh5pNwNy1F7hsaFuvmO2+i/6geoc1Yyj\nySigQMrNz4X8gQ==\n-----END EC PRIVATE KEY-----\n"
    "public_key_fingerprint_md5" = ""
    "public_key_openssh" = ""
    "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEKt90YKmuMtJN8DdTCbLkCIAlBVDv8FYe\naTcDctRe4bGhbr5jtvov+oHqHNWMo8kooEDKzc+F/IE=\n-----END PUBLIC KEY-----\n"
    "rsa_bits" = 2048
  },
]

Removing one key:

Terraform will perform the following actions:

  # tls_private_key.tls_keys1["test1-1"] will be destroyed
  - resource "tls_private_key" "tls_keys1" {
      - algorithm       = "ECDSA" -> null
      - ecdsa_curve     = "P224" -> null
      - id              = "60f23246e6fafd9cc30738af759ac2faf1ad81a2" -> null
      - private_key_pem = (sensitive value)
      - public_key_pem  = <<~EOT
            -----BEGIN PUBLIC KEY-----
            ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEF6xqjSRKo5gSCYWPk86PXYzsvaq+exxV
            TVabVz7sl0MdZebt5P1+FxAIYn/fSs/cCVicxFe5cW8=
            -----END PUBLIC KEY-----
        EOT -> null
      - rsa_bits        = 2048 -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  ~ tls_keys = [
      - {
          - algorithm                  = "ECDSA"
          - ecdsa_curve                = "P224"
          - id                         = "60f23246e6fafd9cc30738af759ac2faf1ad81a2"
          - name                       = "test1-1"
          - private_key_pem            = <<~EOT
                -----BEGIN EC PRIVATE KEY-----
                MGgCAQEEHFtK2da/oWF9JPmDMuYipT919M6UT+PoblG7tFWgBwYFK4EEACGhPAM6
                AAQXrGqNJEqjmBIJhY+Tzo9djOy9qr57HFVNVptXPuyXQx1l5u3k/X4XEAhif99K
                z9wJWJzEV7lxbw==
                -----END EC PRIVATE KEY-----
            EOT
          - public_key_fingerprint_md5 = ""
          - public_key_openssh         = ""
          - public_key_pem             = <<~EOT
                -----BEGIN PUBLIC KEY-----
                ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEF6xqjSRKo5gSCYWPk86PXYzsvaq+exxV
                TVabVz7sl0MdZebt5P1+FxAIYn/fSs/cCVicxFe5cW8=
                -----END PUBLIC KEY-----
            EOT
          - rsa_bits                   = 2048
        },
        {
            algorithm                  = "ECDSA"
            ecdsa_curve                = "P224"
            id                         = "1b8b095e87c8dff3da1d14435ea8ab1de9a3f896"
            name                       = "test2-1"
            private_key_pem            = <<~EOT
                -----BEGIN EC PRIVATE KEY-----
                MGgCAQEEHP7u0NLwuCEBktmRoAjnI/nYK3si7e51+VtXdbmgBwYFK4EEACGhPAM6
                AAQq33Rgqa4y0k3wN1MJsuQIgCUFUO/wVh5pNwNy1F7hsaFuvmO2+i/6geoc1Yyj
                ySigQMrNz4X8gQ==
                -----END EC PRIVATE KEY-----
            EOT
            public_key_fingerprint_md5 = ""
            public_key_openssh         = ""
            public_key_pem             = <<~EOT
                -----BEGIN PUBLIC KEY-----
                ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEKt90YKmuMtJN8DdTCbLkCIAlBVDv8FYe
                aTcDctRe4bGhbr5jtvov+oHqHNWMo8kooEDKzc+F/IE=
                -----END PUBLIC KEY-----
            EOT
            rsa_bits                   = 2048
        },
    ]

tls_private_key.tls_keys1["test1-1"]: Destroying... [id=60f23246e6fafd9cc30738af759ac2faf1ad81a2]
tls_private_key.tls_keys1["test1-1"]: Destruction complete after 0s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Note that there is no output anymore - Terraform simply forgets about it, also gone from the Terraform state:

{
  "version": 4,
  "terraform_version": "0.13.2",
  ...
  "outputs": {},
  ...
}

Only at the next apply, it appears again:

Terraform will perform the following actions:

Plan: 0 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + tls_keys = [
      + {
          + algorithm                  = "ECDSA"
          + ecdsa_curve                = "P224"
          + id                         = "1b8b095e87c8dff3da1d14435ea8ab1de9a3f896"
          + name                       = "test2-1"
          + private_key_pem            = <<~EOT
                -----BEGIN EC PRIVATE KEY-----
                MGgCAQEEHP7u0NLwuCEBktmRoAjnI/nYK3si7e51+VtXdbmgBwYFK4EEACGhPAM6
                AAQq33Rgqa4y0k3wN1MJsuQIgCUFUO/wVh5pNwNy1F7hsaFuvmO2+i/6geoc1Yyj
                ySigQMrNz4X8gQ==
                -----END EC PRIVATE KEY-----
            EOT
          + public_key_fingerprint_md5 = ""
          + public_key_openssh         = ""
          + public_key_pem             = <<~EOT
                -----BEGIN PUBLIC KEY-----
                ME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEKt90YKmuMtJN8DdTCbLkCIAlBVDv8FYe
                aTcDctRe4bGhbr5jtvov+oHqHNWMo8kooEDKzc+F/IE=
                -----END PUBLIC KEY-----
            EOT
          + rsa_bits                   = 2048
        },
    ]

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

tls_keys = [
  {
    "algorithm" = "ECDSA"
    "ecdsa_curve" = "P224"
    "id" = "1b8b095e87c8dff3da1d14435ea8ab1de9a3f896"
    "name" = "test2-1"
    "private_key_pem" = "-----BEGIN EC PRIVATE KEY-----\nMGgCAQEEHP7u0NLwuCEBktmRoAjnI/nYK3si7e51+VtXdbmgBwYFK4EEACGhPAM6\nAAQq33Rgqa4y0k3wN1MJsuQIgCUFUO/wVh5pNwNy1F7hsaFuvmO2+i/6geoc1Yyj\nySigQMrNz4X8gQ==\n-----END EC PRIVATE KEY-----\n"
    "public_key_fingerprint_md5" = ""
    "public_key_openssh" = ""
    "public_key_pem" = "-----BEGIN PUBLIC KEY-----\nME4wEAYHKoZIzj0CAQYFK4EEACEDOgAEKt90YKmuMtJN8DdTCbLkCIAlBVDv8FYe\naTcDctRe4bGhbr5jtvov+oHqHNWMo8kooEDKzc+F/IE=\n-----END PUBLIC KEY-----\n"
    "rsa_bits" = 2048
  },
]

Another indication that merge is the problem here:

In the real code the merge happens on a cloud resource, like shown:

resource "hcloud_ssh_key" "ssh_keys" {
  for_each = merge(
    { for name, data in tls_private_key.ssh_keys : name => data.public_key_openssh },
    { for name, data in data.local_file.ssh_public_keys : name => data.content }
  )

  name       = each.key
  public_key = each.value
}

And whenever some of the underlying resources are destroyed, Terraform fails with:

Error: each.value cannot be used in this context

  on main.tf line 87, in resource "hcloud_ssh_key" "ssh_keys":
  87:   public_key = each.value

A reference to "each.value" has been used in a context in which it
unavailable, such as when the configuration no longer contains the value in
its "for_each" expression. Remove this reference to each.value in your
configuration to work around this error.

Interestingly, this error never occurs with create_before_destroy = true.

@peterpramb hi :slight_smile: Indeed, that is very strange and funny in the same time. I found one more way to make output disappear:

remove square brackets from output value - make it look like this
value = merge(tls_private_key.tls_keys1, tls_private_key.tls_keys2)

then make tls_keylist1 an empty list (just as you commented the string) and see what happens to the output

Actually pretty much the same – it will completely gone from stdout and the state :slight_smile:

Hi,

thanks for confirmation.

You’re right, for reproduction this is far simpler - I’m just used to flat lists which can be flexibly converted to maps as needed. :wink:

Time for an issue: #26146

Thanks for reporting this issue over on GitHub! As I replied over there, I think it should have been fixed with 0.13.3.