How "safe" is it to edit the tfstate file to remove sensitive values that have been `terraform import`-ed?

Terraform
1

With full understanding (and taking the responsibility) of the the warning from Manipulating State - Terraform CLI - Terraform by HashiCorp

Important: Modifying state data outside a normal plan or apply can
cause Terraform to lose track of managed resources, which might
waste money, annoy your colleagues, or even compromise the
security of your operations. Make sure to keep backups of your
state data when modifying state out-of-band.

and in relation to `terraform import` does not respect `ignore_changes` · Issue #20375 · hashicorp/terraform-provider-aws · GitHub,

I’ve got some resources I want to import, but those resources contain an ignorable element.

During the import, the ignorable element is imported, and so it now exists in the .tfstate file.

With full understanding that manipulation of the .tfstate file can lead to all sorts of issues, is it “safe” to CAREFULLY remove the values from the .tfstate file and put it back into the backend store?

Are there checksums or any sort of tfstate file integrity checks that would reveal that the file is inconsistent in some way?

The change really would be changing from (example matching the GitHub issue):

    {
      "mode": "managed",
      "type": "aws_ssm_parameter",
      "name": "parameter",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "index_key": "/PRIVATE/KEY",
          "schema_version": 0,
          "attributes": {
            "allowed_pattern": "",
            "arn": "arn:aws:ssm:eu-west-1:123456789012:parameter/PRIVATE/KEY",
            "data_type": "text",
            "description": "A private key whose value should not be present in the repo or TFstate.",
            "id": "/PRIVATE/KEY",
            "key_id": "",
            "name": "/PRIVATE/KEY",
            "overwrite": null,
            "tags": {},
            "tags_all": {},
            "tier": "Standard",
            "type": "String",
            "value": "dummy",
            "version": 1
          },
          "sensitive_attributes": [],
          "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjAifQ=="
        }
      ]
    },

to

    {
      "mode": "managed",
      "type": "aws_ssm_parameter",
      "name": "parameter",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "index_key": "/PRIVATE/KEY",
          "schema_version": 0,
          "attributes": {
            "allowed_pattern": "",
            "arn": "arn:aws:ssm:eu-west-1:123456789012:parameter/PRIVATE/KEY",
            "data_type": "text",
            "description": "A private key whose value should not be present in the repo or TFstate.",
            "id": "/PRIVATE/KEY",
            "key_id": "",
            "name": "/PRIVATE/KEY",
            "overwrite": null,
            "tags": {},
            "tags_all": {},
            "tier": "Standard",
            "type": "String",
            "value": "",
            "version": 1
          },
          "sensitive_attributes": [],
          "private": "eyJzY2hlbWFfdmVyc2lvbiI6IjAifQ=="
        }
      ]
    },

That is, just removing the dummy value (which is what I am using for this example).

An additional complexity is that the tfstate file is in a versioning backend (S3).

To truly hide the recording of the values, I think I’d need to change the backend to a local file, run the terraform import to import the manually created resources, edit the now local .tfstate file, and then migrate to a backend on S3.

Which does, in itself, seems as potentially fraught with issues as editing the .tfstate file in the first place!

Are there any other, “saner” ways to not get the sensitive data into the .tfstate file for a pre-existing resource when importing?

2

Hi @rquadling,

It’s hard to say exactly what might happen without knowing how the resource implementation handles the data internally. You mentioned that this was an imported resource, so it’s most likely that the value will be immediately written back into state next time the resource is read during a plan. Also, if the original value was derived from the configuration, the provider may attempt to overwrite the existing value with the new empty string.

As long as you have a backup of the state you can test the change with a plan and revert if necessary, but you cannot remove a value from the state that is required to be there by the provider.

3

The value is expected to be ignored from Terraform’s perspective. I just want Terraform to make sure that the resource exists, with 1 of the values NOT being in the tfstate file.

resource "aws_ssm_parameter" "parameter" {
  for_each    = local.parameter_store_keys
  name        = each.key
  description = each.value.description
  type        = "String"
  value       = ""

  lifecycle {
    ignore_changes = [value]
  }

  tags = merge(local.base_tags, {
    "Usage" = each.value.usage
  })
}

So, I have (generated as outputs, all the terraform commands I need to run the imports), I just want the value of the value property to be ignored. In the event that the resource is deleted manually, then a replacement will be created with the value of "" which is fine.

I’ve raised an issue with the AWS Provider on GitHub (`terraform import` does not respect `ignore_changes` · Issue #20375 · hashicorp/terraform-provider-aws · GitHub). This is a duplicate of a closed issue. Closed as it seems there was no feedback.

4

My current solution is to generate all the required aws ssm put-parameter statements, drop the Parameters, apply terraform to create clean/empty placeholders, and then run the aws ssm put-parameter statements.

As these parameters are only actually pulled in for us during a new EC2 launch … and the healthcheck will bounce the EC2 if not got all the right parameters, … feels a good enough approach.

Certainly better than editing tfstate files directly.

If anyone has any additional comments or thoughts, then I’d love to have them. I do feel the terraform import should ignore the values on import, and if it did, that would have solved the problem entirely.

5

Well. All for nothing.

Having deleted the parameter store entries (keeping a copy of the values), told Terraform to create them with dummy as the value, then updating the actual values via aws cli (aws console would also be OK), the next plan/apply cycle has pulled the values into the tfstate file, even though the .tf file explicitly says ignore changes to valueand version.

The output before the plan shows

Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the
last "terraform apply":
  # aws_ssm_parameter.parameter["/PRIVATE/KEY"] has been changed
  ~ resource "aws_ssm_parameter" "parameter" {
        id          = "/PRIVATE/KEY"
        name        = "/PRIVATE/KEY"
        tags        = {
            "BuildFrom"   = "Compute"
            "Environment" = "test"
            "Terraform"   = "true"
            "Usage"       = "General"
        }
      ~ value       = (sensitive value)
      ~ version     = 1 -> 2
        # (6 unchanged attributes hidden)
    }

The externally changed values are imported into the tfstate.

Oh well. Live and learn.

6

Unfortunately this is what I meant that you cannot remove a value form the state which is required by the provider. If the provider stores a value, terraform must return that same value later on.

To clarify the ignore_changes issue, ignore_changes is used to ignore differences between the most recent state and the configuration, preventing terraform from planning unwanted changes in the remote resource. It is not meant to prevent the provider from changing values in the state. The provider protocol does not have any way to allow terraform to ignore values returned from the provider.

1 Like
7

Thank you all for your comments.

We have enough security in place (limited access to .tfstate bucket and relevant credentials and AWS Parameter Store) to mitigate the potential exposure of credentials until we move to a different model for this.