How to show sensitive values

I can see how sensitive values need to be protected in CI/CD, but at my terminal I would like to see them. In some cases, there is a workaround, eg for outputs that are marked sensitive you can use json output. BUT this trick does not work for terraform apply: sensitive values are masked, and if the value is say a json blob (created from a template), the whole blob is masked. There is then no way for me to know what actually caused the change in that json/value.

Is there a way to tell terraform “just ignore sensitivity in all outputs for this command”?

Hi @schollii,

The intended way to reveal the sensitive values when you’re at a terminal is to run a command like terraform output -json, or any other variant that produces machine-readable output. It sounds like you’ve been using that already.

I think the situation you’ve described here is that you see Terraform proposing a change in the plan output but you can’t see what exactly is being proposed, because you’ve used a sensitive value as part of it.

While I would be the first to accept that it’s not a super convenient answer, I think the current answer is the same for plans as it is for output values: use one of the machine-readable output mechanisms to inspect the value. In the case of plan output, you can get there like this:

terraform plan -out=tfplan
terraform show -json tfplan
# then, if you decide that the planned change is acceptable
terraform apply tfplan

This sort of workflow is of course only reasonable for rare situations, because it’s inconvenient. If you’ll be doing this routinely (e.g. in automation) then I might consider a design that includes an additional output value where you’ve somehow masked out the sensitive parts of the big JSON string and then passed the result through nonsensitive to assure Terraform that you’ve removed all of the sensitive parts. Then that slightly-transformed version should be visible in the “changes to outputs” section of the plan so humans can inspect the shape of the data structure without also disclosing the sensitive portion(s).

1 Like

Thanks @apparentlymart, this workaround works.

But it is, to say the least, less than ideal.

Is there a reason there couldn’t be an -insecure mode to both plan and apply? This could be handy in combination with -target. Is it something that could be contributed by the community (I might give it a shot).

2 Likes

Hi @schollii,

I must admit I was not directly involved in the research for this feature but I believe that one of the key requirements was that human-readable output should never include sensitive values. This mechanism to get at the raw values by programmatic means was intended to allow you to programmatically inspect the sensitive results in order to automatically make decisions without a human ever seeing the sensitive value.

For now I would expect that we would not add a new option for this because new options of this sort add another dimension to the matrix of possibilities we need to account for in future improvements and fixes, and sensitive value handling is a cross-cutting concern because such an option would need to be handled correctly in every separate situation where Terraform renders a value. The risk of a bug accidentally exposing a sensitive value would therefore be increased.

I understand that the result is inconvenient in your case because you have a different sense of the requirements that motivated this design: you consider your own personal terminal to be “secure enough” to display those settings, whereas CI is not. However, we have limited resources and so unfortunately we often must make design tradeoffs that prioritize one use-case over another, and this is an example of that.

As I mentioned before, my hope would be that for most situations it would be preferable to design the system so that there’s never a need for a human to directly inspect sensitive values, because the configuration is designed to give the human operator all of the information they need to make a decision. That may not be straightforward in all cases, but I expect it will be possible in most cases now that we have the nonsensitive function to allow for selectively disclosing non-sensitive values derived from sensitive ones.

1 Like

Thanks @apparentlymart your thoughtful answers are always appreciated.

I realized since this discussion that it’s not so much seeing sensitive values, it’s seeing the proposed changes in a sensitive value that has non-sensitive info that is missing.

The whole point of a plan is to show what will change if the plan is applied. Indeed you don’t usually care what the new value of a sensitive value is, EXCEPT if that value is a mixture of sensitive and non-sensitive info. If a blob was generated from a template (whether it is json, yaml, plain text, csv, etc), and at least one of the variables used to render the template is sensitive, then it makes sense for the blob as a whole to be sensitive.

HOWEVER, that blob should have 2 representations: one that is for terraform, and one for humans. The human one would just replace the sensitive fields with eg **********, and mark that representation non-sensitive; the tf value would be marked sensitive. Then in a plan, it is feasible to only print the lines that have changed in the human-friendly representation.

Example:

# input.yaml file used as template
value: 
   child1: ${nonsensitive_var}
   child2: ${sensitive_var}

# .tf file
resource "xyz_type" "xyz_name" {
   something = templatefile("input.yaml", { 
      nonsensitive_var = var.nonsensitive_var
      sensitive_var = var.sensitive_var
  })
}

Then terraform apply would show this, if the value of var.nonsensitive_var changed since last apply:

  # module.... will be updated in-place
  ~ resource "xyz_type" "xyz_name" {
      ~ something = (sensitive; changed lines:
               value: 
             ~   child1: new_value_of_nonsensistive_var
                 child2: (sensitive -- unchanged value redacted)
          )
      ...
    }
1 Like

Hi @schollii,

I think what you’re suggesting here boils down to it being possible for a string to be partially sensitive. That’s a considerably more complex proposition than whole values being sensitive, because Terraform’s idea of strings is an atomic leaf value, not containing any divisible parts.

Given that, I don’t think there’s any practical path to Terraform doing anything like this automatically in the foreseeable future, and so I’d say this is just an unfortunate consequence of provider designs that involve packing together a bunch of structured information into a string attribute. That’s unfortunately more common than it ought to have been because the pre-v0.12 Terraform type system wasn’t rich enough for complex data structures and most providers were written under those constraints. In future though, providers will hopefully migrate to making more use of complex data structures rather than packed JSON/YAML strings (where the provider itself would then presumably encode the data in whatever format the remote API is expecting), at which point Terraform could properly track the sensitivity individual parts.

There will always be some edge cases where a templated string really is the best representation of something, in which case indeed including sensitive values will make the plan less helpful, but I expect there are fewer examples of that then there are of provider designs using JSON/YAML strings purely to work around the limitations of classic Terraform’s type system, which can therefore be addressed by a design change using the newer capabilities.

I think my only option then is to use nonsensitive() around the template values that go into the json. It’s not great but honestly, it’s not bad either: the fact that the AWS Opsworks custom json contains sensitive values at all is the weakest link (and not my doing!).

The terraform state is stored in s3 with SSE so this is the least of our security concerns.

And having the custom json show up in logs is not a concern. Let’s hope that by the time we gitops this, we’ll have moved away from (godawful) opsworks.

It’s funny that the OpsWorks resource types would be the ones you’re actually using here because by coincidence it was me in a previous life (my old employer) who originally implemented aws_opswork_stack and the various layer types.[1]

And so I can, at least, confirm that I did originally want custom_json to be custom_data which accepts any value and have the provider itself JSON-encode it, and so this is one of the things in the category of “providers written under those constraints” which could possibly get superseded by a non-string version in a future release of the provider, once the provider team is ready to adopt the new SDK.

Though I don’t work on the AWS provider anymore, so that’s of course not for me to commit to either way; I know there are various other examples that are probably more commonly used than OpsWorks. And indeed, by then perhaps you won’t need the OpsWorks resource types anymore anyway.

[1] I got a nice reminder of a funny bit of history here, where implementing this was when I first encountered the very limitation I was describing above, and at that time made a prooposal to address it. Although that particular proposal wasn’t accepted (for good reasons, in retrospect), we did finally get to address it in a different way as part of Terraform v0.12. :sweat_smile: Hopefully it won’t be long now for providers to start making fuller use of it.

1 Like

The following works well enough for my needs

To speed up the plan, you can use -target=foo.bar to generate a smaller plan file

tf plan -target=vault_generic_secret.foobar -out=/tmp/tfplan
tf show -json /tmp/tfplan  | jq -r ".resource_changes[0].change.before.data , .resource_changes[0].change.after.data"
3 Likes

I ran across this post while searching for how to show sensitive values, but my use case is different: I’d like to move us to using terraform for creating AWS IAM users for service accounts, rather than the GUI. This means we need to be able to get the secret key when first creating an account, so we can store it in our corporate password manager and share it with the appropriate people.

Unfortunately, the workaround of looking at terraform plan output doesn’t work in this case, because the secret key is not known until you actually apply the aws_iam_user and aws_iam_access_key resources.

Instead, the workaround I found is to apply using a local state file, then run terraform output --json to see the two outputs from that state file, then delete the local state file.

What I’d much prefer is a way to do this such that:

  • The access key and secret key are echo’ed to stdout when the apply finishes
  • … and they are NOT stored in the state file. Or at least, not the secret key.

Anyone know if that is doable?

Anything you load into terraform via an input variable, local variable, or environment variable, will end up in the state in some form. All outputs are stored in state file too. So you cannot have both.

However, json format of outputs does not hide sensitive data (I only tried in terraform 1.1.7). So once the IAM user has been created with terraform apply, you can do terraform output -json | jq <path-to-sensitive-value> to output sensitive data.

The terraform output command can also directly select a single value if you include a positional argument, and -raw mode can be helpful to avoid the need for piping through jq for simpler cases.

But indeed, it’s intentional that sensitivity is all about hiding things in the normal UI, so you can always get the underlying data by poking at it through entry points intended for consumption by other software, like this raw output mode. The idea here is that the normal Terraform output that’s probably captured in your automation system’s log buffer won’t contain the secrets but the pipeline code itself can still internally get the value to automatically pass it to some subsequent workflow step, outside of Terraform.

With that said though, I would suggest not using Terraform to actually issue IAM credentials. A typical compromise is to use Terraform to manage the roles/users/etc themselves but some external process to provision real credentials for those, so that the external process can ensure that it doesn’t take any record of the credentials itself, or ideally somehow avoid even intermediating the credentials so they go directly to the user who will use them. You may also wish to issue only time-limited credentials to force users to come back and auth to your issuing system periodically. But the details of all of that are outside of the scope of Terraform.

I can no longer see proposed changes to any resources of a certain type because one of them (once) had a sensitive value. This is frustrating and it’s forcing me to “fly blind.”

I need a way to see proposed changes to resources. I’ve seen people suggest looking at the tfplan as JSON, but my current one is 25k lines long!

Since the community isn’t up in arms, complaining about flying blind, they must know something that I don’t.

What am I missing?

@jamiejaxon_hashicorp, I’d love it help but I’d need to see the resource block that is causing that, and the plan you get.

If you just copy that block to a new main.tf and hardcode values, and use sensitive() to mark the same attribute that is causing issue in plan, and such that your problem is reproduced, then I can run locally and might have a trick for you.

Thanks for jumping in. I’ll have to think of a good repro case for you because it’s not quite that simple:

When one resource type (in case, one of an aws_cloudfront_distribution’s origin blocks) gets flagged as sensitive, terraform flags them all as sensitive, even if the others aren’t, so it masks changes to the non-sensitive blocks, too. I only have one origin that has a sensitive value, but the one I’m changing (non-sensitive) gets masked.

For instance, the masked origin, below, that I’m changing contains no sensitive information:

Terraform will perform the following actions:

  # aws_cloudfront_distribution.dev_auth_elb_distribution will be updated in-place
  ~ resource "aws_cloudfront_distribution" "dev_auth_elb_distribution" {
        id                             = "REDACTED"
        tags                           = {
            "Environment"   = "dev"
            "TerraformRepo" = "infrastructure-core"
        }
        # (21 unchanged attributes hidden)

      - custom_error_response {
          - error_caching_min_ttl = 10 -> null
          - error_code            = 403 -> null
          - response_code         = 404 -> null
          - response_page_path    = "/legacy_media_share/deployment_root/wwwroot/sites/onecpd/assets/404.htm" -> null
        }
      + custom_error_response {
          + error_caching_min_ttl = 10
          + error_code            = 403
          + response_code         = 404
          + response_page_path    = "/sites/onecpd/assets/404.htm"
        }
      - custom_error_response {
          - error_caching_min_ttl = 10 -> null
          - error_code            = 404 -> null
          - response_code         = 404 -> null
          - response_page_path    = "/legacy_media_share/deployment_root/wwwroot/sites/onecpd/assets/404.htm" -> null
        }
      + custom_error_response {
          + error_caching_min_ttl = 10
          + error_code            = 404
          + response_code         = 404
          + response_page_path    = "/sites/onecpd/assets/404.htm"
        }

      ~ ordered_cache_behavior {
          ~ path_pattern               = "/legacy_media_share/deployment_root/wwwroot/sites/onecpd/assets/404.htm" -> "/sites/onecpd/assets/404.htm"
            # (13 unchanged attributes hidden)
        }

        origin {
          # At least one attribute in this block is (or was) sensitive,
          # so its contents will not be displayed.
        }

        # (21 unchanged blocks hidden)
    }

@jamie_jackson that code can definitely be simplified, eg all the custom error response can be removed, and you only need 2 origins, one with a sensitive value and one without.

is there a property in that particular origin that was sensitive but is no longer sensitive? Then to reproduce this particular case you should just need one origin not 2, you first point one of its properties to a random_password value and then change to a non-sensitive local value and see if plan does this. You could then try using the nonsensitive() function on the property that used to be sensitive to see if tf plan still hides it.

I didn’t see your response before finishing my repro case but this should work for you:

@jamie_jackson I confirmed some behaviors and understand the reason, but I cannot think of a workaround other than what was mentioned in previous posts. Based on my analysis, it might be worth posting on the aws provider github, in case there is a simple fix that could be made on the aws provider to set the sensitivity on a specific origin or even better, on a specific property of a specific origin.

Details:

I verified that when none of the origins have sensitive data, the plan shows origin diffs, moreover it does so EVEN if there was previously sensitive data (ie if no sensitive data, apply, then add sensitive data, apply, then remove sensitivity of data, apply, then plans resume showing mods – I wasn’t sure of that.

I found that the reason that adding one sensitive property to one origin blocks makes all origins of that cloudfront resource sensitive is that the cloudfront resource “origin” attribute is modelled as one object: a list of origins, even though in the HCL it is represented as a sequence of blocks. The following gets added in the terraform state file in the cloudfront instance:

          "sensitive_attributes": [
            [
              {
                "type": "get_attr",
                "value": "origin"
              }
            ]
          ],

which does not discriminate between different origins.

However, that setting only affects that particular cloudfront instance, as can be seen from the state file. So while it is accurate that having sensitive data in one origin block taints all origins of a cloudfront instance, this does not propagate to other cloudfront instances; changes to these will remain visible in the plan (I confirmed this).

This seems like a limitation of how the provider modelled a cloudfront distro IMO. It might be worth posting on the aws provider github, maybe there is a simple fix that could be made on the provider (eg maybe there are other allowed types in the tfstate sensitive_attributes block, one that would allow specific items of origins list to be referenced instead of whole list).