Packer getting AWS authentication error after converting json to hcl2

Hi,

I am starting using packer to build aws ami images with ansible. I have created a sample json project that uses the AWS enviroment variables AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY.

Everything works fine with json. Following the recommendations I converted the packer script to hcl but now I am getting an AWS aunthentication error:

PACKER_LOG=1 packer build packer-build.json.pkr.hcl
.....
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin: 2024/08/08 16:23:34 Using AMI Filters {
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:   Filters: [{
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:       Name: "name",
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:       Values: ["al2023-ami-2023.5.20240805.0-kernel-6.1-arm64"]
>2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:     },{
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:       Name: "root-device-type",
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:       Values: ["ebs"]
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:     },{
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:       Name: "virtualization-type",
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:       Values: ["hvm"]
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:     }],
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:   IncludeDeprecated: false,
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin:   Owners: ["amzon"]
2024/08/08 16:23:34 packer-plugin-amazon_v1.3.2_x5.0_darwin_arm64 plugin: }
2024/08/08 16:23:34 [INFO] (telemetry) ending amazon-ami
        status code: 401, request id: 73b36562-8bd6-43d3-b24c-1f376c5dda54

  on packer-build.json.pkr.hcl line 100:
  (source code not available)

2024/08/08 16:23:34 [INFO] (telemetry) Finalizing.
Error: Datasource.Execute failed: Error querying AMI: AuthFailure: AWS was not able to validate the provided access credentials
        status code: 401, request id: 73b36562-8bd6-43d3-b24c-1f376c5dda54

  on packer-build.json.pkr.hcl line 100:
  (source code not available)


2024/08/08 16:23:35 waiting for all plugin processes to complete...

I have confirmed that the hcl file is valid with packer validate packer-build.json.pkr.hcl and even checked everything is correct with packer console and displayed the variables to confirm they are set. Also I have double checked that the original json file works.

My hcl file (after tweaking) as the following set:

....
variable "aws_region" {
  type    = string
  default = "********"
}

variable "aws_access_key" {
  type    = string
  default = env("AWS_ACCESS_KEY_ID")
  validation {
    condition     = length(var.aws_access_key) > 0
    error_message = <<EOF
The AWS_ACCESS_KEY_ID environment variable must be set.
EOF
  }

}

variable "aws_secret_key" {
  type    = string
  default = env("AWS_SECRET_ACCESS_KEY")
  validation {
    condition     = length(var.aws_secret_key) > 0
    error_message = <<EOF
The AWS_SECRET_ACCESS_KEY environment variable must be set.
EOF
  }
}

....

data "amazon-ami" "buildami" {
  region     = "${var.aws_region}"
  secret_key = "${var.aws_secret_key}"
  access_key = "${var.aws_access_key}"
  filters = {
    name                = "al2023-ami-2023.5.20240805.0-kernel-6.1-arm64"
    root-device-type    = "ebs"
    virtualization-type = "hvm"
  }
  most_recent = true
  owners      = ["amzon"]
}

...

source "amazon-ebs" "build-image" {
  access_key                  = "${var.aws_access_key}"
  ami_name                    = "${var.ami_name}"
  region                      = "${var.aws_region}"
...
source_ami           = "data.amazon-ami.buildami.id"
...

Any help or tips to help diagnosing this would be appreciated. I am about giving up on using hcl and continue using json.