Download and Cat a file from URL and store the output in aws parameter store

Terraform
1

Hello there!

I am looking at the options to achieve the below.

  1. Curl/Wget a public.pem file from the URL
  2. Cat the pem file
  3. Store the keys in the AWS SSM parameter store.

Could any one let me know how to achieve this?

Regards,
Krishna Mohan

2

One possibility would be to create a null_resource with a local-exec provisioner.

You can then use aws_ssm_parameter (doc) to read contents from the downloaded file.

3

Thank you for the response.

I tried with null_resource

`resource "null_resource" "ca_cert_download" {
    provisioner "local-exec" {
    command = "curl -sS -o /tmp/ap-southeast-2-bundle.pem https://truststore.pki.rds.amazonaws.com/ap-southeast-2/ap-southeast-2-bundle.pem"
        interpreter = ["bash", "-c"]
     }
 }`

I am not quite sure what to pass in the value as null_resource cannot provide the output.

resource "aws_ssm_parameter" "ca_cert_upload" {
    name  = "/${var.env}/${var.name}/rds/cert"
    tier  = "Advanced"
    type  = "SecureString"
    value = "???"
  }

I tried this https://github.com/matti/terraform-shell-resource, but the problem is unable to find the file as it is a run time.

I want to cat the pem file and copy the output in the SSM parameter

4

The aws_ssm_parameter resource needs to run after the null_resource one. That can be achieved with depends_on meta-argument.

The value argument should be something like file("/tmp/ap-southeast-2-bundle.pem")

5

As suggested, I tried it and got the below error.

resource "aws_ssm_parameter" "ca_cert_upload" {
    name  = "/${var.env}/${var.name}/rds/cert"
    tier  = "Advanced"
    type  = "SecureString"
    value = file("/tmp/ap-southeast-2-bundle.pem")
    tags  = var.tags 
    depends_on = [
      null_resource.ca_cert_download 
    ]
  }

Error message: terraform plan

╷
│ Error: Invalid function argument
│ 
│   on ../../modules/rds/main.tf line 204, in resource "aws_ssm_parameter" "ca_cert_upload":
│  204:     value = file("/tmp/ap-southeast-2-bundle.pem")
│     ├────────────────
│     │ while calling file(path)
│ 
│ Invalid value for "path" parameter: no file exists at "/tmp/ap-southeast-2-bundle.pem"; this function works only with files that are distributed as part of the configuration source code, so if this file
│ will be created by a resource in this configuration you must instead obtain this result from an attribute of that resource.
6

This function can be used only with files that already exist on disk at the beginning of a Terraform run. Functions do not participate in the dependency graph, so this function cannot be used with files that are generated dynamically during a Terraform operation. We do not recommend using dynamic local files in Terraform configurations, but in rare situations where this is necessary you can use the local_file data source to read files while respecting resource dependencies.

7

Thank you for the response and inputs.

I tried local_files data source options with depends_on but no luck.
So i came to forum and started digging into any solution and could not find any. So i raised one topic.

We will see any options come up in the replies. Thank you again.

8
  1. Is there a reason you absolutely need to download the certificate file at run time?
  2. It should have worked with local_file data source. What error did you get?
9
  1. Is there a reason you absolutely need to download the certificate file at run time? Yes, whenever it is going to expire, it will go and get the key and update. Which will use for my application and it will never fail
  2. It should have worked with the local_file data source. What error did you get?

Test 1: resource

resource "local_file" "ca_download" {
  filename = "tmp/ap-southeast-2-bundle.pem"
  content = <<-EOT
      curl -sS -o tmp/ap-southeast-2-bundle.pem https://truststore.pki.rds.amazonaws.com/ap-southeast-2/ap-southeast-2-bundle.pem
    EOT
}


resource "aws_ssm_parameter" "ca_cert_upload" {
    name  = "/${var.env}/${var.name}/rds/cert"
    tier  = "Advanced"
    type  = "SecureString"
    value = "${resource.local_file.ca_download.content}"
    tags  = var.tags 
    depends_on = [
      resource.local_file.ca_download
    ]
  }

It will write the same content in the local_file in SSM parameter.

Test 2: Data

data "local_file" "ca_download" {
  filename = "tmp/ap-southeast-2-bundle.pem"
    depends_on = [ 
      null_resource.ca_cert_download 
  ]
}

resource "aws_ssm_parameter" "ca_cert_upload" {
    name  = "/${var.env}/${var.name}/rds/cert"
    tier  = "Advanced"
    type  = "SecureString"
    value = "data.local_file.ca_download.content"
    tags  = var.tags 
    depends_on = [
      data.local_file.ca_download
    ]
  }

ERROR:

│ The file at given path cannot be read.
│ 
│ +Original Error: open /tmp/ap-southeast-2-bundle.pem: no such file or directory
10

null_resource is still needed


resource "null_resource" "ca_cert_download" { 
   # your definition goes here
}

data "local_file" "ca_download" {
  filename = "/tmp/ap-southeast-2-bundle.pem" # point this to the file downloaded by the null_resource

  depends_on = [
    null_resource.ca_cert_download 
  ]
}

resource "aws_ssm_parameter" "ca_cert_upload" {
    name  = "/${var.env}/${var.name}/rds/cert"
    tier  = "Advanced"
    type  = "SecureString"
    value = data.local_file.ca_download.content
}
11

The data worked pretty well for the first run and in the second run as expected it failed.

Error: Read local file data source error
│ 
│   with module.rds_cluster.data.local_file.ca_download,
│   on ../../modules/rds/main.tf line 201, in data "local_file" "ca_download":
│  201: data "local_file" "ca_download" {
│ 
│ The file at given path cannot be read.
│ 
│ +Original Error: open /tmp/ap-southeast-2-bundle.pem: no such file or
│ directory
╵
12

Are you removing the file at the end though (or is it run via CI)?

If so, your null_resource needs a trigger that will tell it to run every single time (e.g. something random or a timestamp kinda string)

13

I am running in CI pipeline. Every time it should run as part of pipeline.

14

Try

  triggers = {
    always_run = "${timestamp()}"
  }
15

Thank you so much @macmiranda. It worked. Much appreciated.

Successful Code:

resource "null_resource" "ca_cert_download" {
    provisioner "local-exec" {
    command = "curl -sS -o /tmp/ap-southeast-2-bundle.pem https://truststore.pki.rds.amazonaws.com/ap-southeast-2/ap-southeast-2-bundle.pem"
        interpreter = ["bash", "-c"]
     }

    triggers = {
      always_run = "${timestamp()}"
    }
 }

data "local_file" "ca_download" {
  filename = "/tmp/ap-southeast-2-bundle.pem"  
  depends_on = [
    null_resource.ca_cert_download 
  ]
}

resource "aws_ssm_parameter" "ca_cert_upload" {
    name  = "/${var.env}/${var.name}/rds/cert"
    tier  = "Advanced"
    type  = "SecureString"
    value = data.local_file.ca_download.content
    tags  = var.tags 
}

Note: AWS SSM parameter has Standard and Advanced tier
The standard tier has a restriction to use only 4096 characters. if you are using the same approach request to use Advanced tier(charges apply).*

1 Like