Updating `aws_s3_object` resource without deleting previous files

Terraform Providers AWS
1

Hi! I’m thinking of using Terraform for provisioning Lambda/API gateway, but also for uploading my local Lambda source code in zipfile + uploading it to S3.

The workflow I am considering is the following:

  • Developer modifies and builds JS source code
  • Deleloper runs terraform apply when they want to release the new code
  • Terraform zips source code + pushes code to a particular unique path in S3 (based on code hash)
  • Terraform re-provisions Lambda function to point to new source code path in S3.

This works fine, but I also notice that Terraform (correctly) deletes the previous S3 object every time a new S3 object is uploaded. This makes sense since there should only be 1 resource, but how can I stop Terrraform from deleting the existing S3 object (so that I can keep it as a backup) but create another one with the new code only (paths should never conflict)?

Below is my Terraform code

data "archive_file" "data_folder" {
  type        = "zip"
  source_dir  = "${path.module}/../../"
  output_path = "/tmp/data.zip"
  excludes = [
    "provisioning",
    "modules",
    ".terraform",
  ]
}

resource "aws_s3_object" "parser_code" {
  bucket = var.lambda_bucket.id
  key    = "${data.archive_file.data_folder.output_sha}/application.zip" # 👈 Should always be unique
  source = data.archive_file.data_folder.output_path
  etag   = filemd5(data.archive_file.data_folder.output_path)
}

resource "aws_lambda_function" "rich_transaction_parser" {
  s3_bucket     = aws_s3_object.parser_code.bucket
  s3_key        = aws_s3_object.parser_code.key
  # ...
}

Any thoughts on how I can simply tell Terraform to “ignore” the delete of S3 objects that it previously created?

2

That’s not how Terraform is designed to work. The code you create describes the state that you are wanting, with terrafirm apply used to update any resources to match that code.

In your case you are asking Terraform to create a single in the s3 bucket , so there will only ever be one

3

Yup that makes sense, am I using terraform incorrectly as part of AWS Lambda? Should Terraform be used to handle both packaging code as well as provisioning infrastructure that runs that code, or should it only be the latter? this tutorial makes it seem like Terraform can be used for both

4

Hi @PirosB3,

An earlier version of that tutorial showed a more realistic process where the build step would be handled by some other tool that would be responsible for building the package and recording it in S3 for the Terraform configuration to then use only by reference, but I believe the current version takes a different approach so that it can focus on Terraform and avoid the manual placeholder steps from the earlier version.

For real use I would suggest following what the old version of the tutorial suggests, though: design a build process that’s outside of Terraform, probably managed by a typical build automation or CI system, and then parameterize the Terraform configuration by the artifact ID (e.g. version number) you intend to use, so that your overall build/deploy pipeline can feed the result of the build step into the Terraform-powered deploy step as an input variable.

While you can use Terraform to create and manage the artifacts too, it’s not designed for that situation and as you’ve seen it has the drawback of assuming that the artifacts are typical long-lived singular infrastructure objects that Terraform typically deals with, rather than being a series of separate objects that you switch between and might roll back through in some situations.