Packer Build Not Working with AWS EBS Encryption using KMS(CMK)

tmohlatlole · June 15, 2023, 8:54am

Good morning

We have trying to resolve a compliance finding which states that EBS volume default encryption must be set ON. However, this breaks packer. After some research and trial and error, we eventually came up with the following code

{
    "variables": {
        "ami_id": "ami-id-here",
        "app_name": "test-server",
        "env": "dev",
        "vpc_id" : "vpc-id-here",
        "subnet_id": "subnet-id-here",
	    "region": "region-of-deployment",
        "accounts": "accounts-ids-to-share-image-with",
        "root_volume_size_Gi": "size-of-volume",
        "version": "0.58",
        "dd_api_key": "datadog-agent-api-key",
        "dd_site": "datadoghq.eu",
        "dd_agent_version": "7",
        "compute_kms_key_id": "the-key-id-here",
        "ec2_instance_role_name": "test-role"
    },
    "builders": [
        {
            "type": "amazon-ebs",
            "access_key": "{{user `aws_access_key`}}",
            "secret_key": "{{user `aws_secret_key`}}",
            "vpc_id" : "{{user `vpc_id`}}",
            "subnet_id": "{{user `subnet_id`}}",
            "region": "{{user `region`}}",
            "source_ami": "{{user `ami_id`}}",
            "instance_type": "t3.large",
            "ssh_username": "ec2-user",
            "ami_name": "test-golden-{{user `app_name`}}-ami-{{user `version`}}",
            "iam_instance_profile": "{{user `ec2_instance_role_name`}}",
            "tags": {
                "Env" : "{{user `env`}}",
                "Name" : "test-{{user `app_name`}}"
            },
            "encrypt_boot": true,
            "ami_block_device_mappings": [ {
                "device_name": "/dev/sda1",
                "volume_size": "{{user `root_volume_size_Gi`}}",
                "volume_type": "gp2",
                "delete_on_termination": true,
                "encrypted": true
            }],
            "launch_block_device_mappings": [ {
                "device_name": "/dev/sda1",
                "volume_size": "{{user `root_volume_size_Gi`}}",
                "volume_type": "gp2",
                "delete_on_termination": true,
                "encrypted": true,
                "kms_key_id": "{{user `compute_kms_key_id`}}"
            }]
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "environment_vars": ["DD_API_KEY={{user `dd_api_key`}}",
                        "DD_SITE={{user `dd_site`}}",
                        "DD_AGENT_MAJOR_VERSION={{user `dd_agent_version`}}"],
            "script": "scripts/common.sh"
        }
    ]
}

The build spec is as follows

---
version: 0.2

phases:
  pre_build:
    commands:
      - echo "Installing additional YUM & Python packages..."
      - yum -y update && yum -y install zip unzip curl python3-pip ruby
      - pip3 install awscli
      - echo "Installing Packer..."
      - cd /usr/local/bin/ && curl -qL -o packer.zip https://releases.hashicorp.com/packer/1.7.6/packer_1.7.6_linux_amd64.zip && unzip packer.zip
      - echo "Validating packer JSON"
      - cd $CODEBUILD_SRC_DIR && packer version
      - cd packer-amis
      - packer validate test.json
      - env
      - echo $HOME
  build:
    commands:
      - echo "Build Phase"
      - cd $CODEBUILD_SRC_DIR
      - cd packer-amis
      - packer build  test.json
  post_build:
    commands:
      - echo "HashiCorp Packer build completed on `date`"

The logs are as follows:

==> amazon-ebs:   8150K .......... .......... .......... .......... .......... 82% 1.13M 0s
==> amazon-ebs:   8200K .......... .......... .......... .......... .......... 82% 26.8M 0s
==> amazon-ebs:   8250K .......... .......... .......... .......... .......... 83%  219M 0s
==> amazon-ebs:   8300K .......... .......... .......... .......... .......... 83%  162M 0s
==> amazon-ebs:   8350K .......... .......... .......... .......... .......... 84%  253M 0s
==> amazon-ebs:   8400K .......... .......... .......... .......... .......... 84%  240M 0s
==> amazon-ebs:   8450K .......... .......... .......... .......... .......... 85%  268M 0s
==> amazon-ebs:   8500K .......... .......... .......... .......... .......... 85%  225M 0s
==> amazon-ebs:   8550K .......... .......... .......... .......... .......... 86%  180M 0s
==> amazon-ebs:   8600K .......... .......... .......... .......... .......... 87%  198M 0s
==> amazon-ebs:   8650K .......... .......... .......... .......... .......... 87%  233M 0s
==> amazon-ebs:   8700K .......... .......... .......... .......... .......... 88%  144M 0s
==> amazon-ebs:   8750K .......... .......... .......... .......... .......... 88%  171M 0s
==> amazon-ebs:   8800K .......... .......... .......... .......... .......... 89%  197M 0s
==> amazon-ebs:   8850K .......... .......... .......... .......... .......... 89%  166M 0s
==> amazon-ebs:   8900K .......... .......... .......... .......... .......... 90%  204M 0s
==> amazon-ebs:   8950K .......... .......... .......... .......... .......... 90%  152M 0s
==> amazon-ebs:   9000K .......... .......... .......... .......... .......... 91%  183M 0s
==> amazon-ebs:   9050K .......... .......... .......... .......... .......... 91%  213M 0s
==> amazon-ebs:   9100K .......... .......... .......... .......... .......... 92%  153M 0s
==> amazon-ebs:   9150K .......... .......... .......... .......... .......... 92%  183M 0s
==> amazon-ebs:   9200K .......... .......... .......... .......... .......... 93%  156M 0s
==> amazon-ebs:   9250K .......... .......... .......... .......... .......... 93%  122M 0s
==> amazon-ebs:   9300K .......... .......... .......... .......... .......... 94%  190M 0s
==> amazon-ebs:   9350K .......... .......... .......... .......... .......... 94%  134M 0s
==> amazon-ebs:   9400K .......... .......... .......... .......... .......... 95%  135M 0s
==> amazon-ebs:   9450K .......... .......... .......... .......... .......... 95%  186M 0s
==> amazon-ebs:   9500K .......... .......... .......... .......... .......... 96%  190M 0s
==> amazon-ebs:   9550K .......... .......... .......... .......... .......... 96%  193M 0s
==> amazon-ebs:   9600K .......... .......... .......... .......... .......... 97%  165M 0s
==> amazon-ebs:   9650K .......... .......... .......... .......... .......... 97%  167M 0s
==> amazon-ebs:   9700K .......... .......... .......... .......... .......... 98%  543K 0s
==> amazon-ebs:   9750K .......... .......... .......... .......... .......... 98% 96.3M 0s
==> amazon-ebs:   9800K .......... .......... .......... .......... .......... 99% 44.8M 0s
==> amazon-ebs:   9850K .......... .......... .......... .......... .......... 99% 48.8M 0s
==> amazon-ebs:   9900K .......... .......... .......... .......... ..        100%  149M=1.0s
==> amazon-ebs:

==> amazon-ebs: 2023-06-14 17:21:22 (9.61 MB/s) - ‘node_exporter-1.5.0.linux-amd64.tar.gz’ saved [10181045/10181045]

==> amazon-ebs:
    amazon-ebs: node_exporter-1.5.0.linux-amd64/
    amazon-ebs: node_exporter-1.5.0.linux-amd64/LICENSE
    amazon-ebs: node_exporter-1.5.0.linux-amd64/NOTICE
    amazon-ebs: node_exporter-1.5.0.linux-amd64/node_exporter

==> amazon-ebs: [sss_cache] [sysdb_domain_cache_connect] (0x0010): DB version too old [0.22], expected [0.23] for domain implicit_files!
==> amazon-ebs: Higher version of database is expected!
==> amazon-ebs: In order to upgrade the database, you must run SSSD.
==> amazon-ebs: Removing cache files in /var/lib/sss/db should fix the issue, but note that removing cache files will also remove all of your cached credentials.
==> amazon-ebs: Could not open available domains
==> amazon-ebs: [sss_cache] [sysdb_domain_cache_connect] (0x0010): DB version too old [0.22], expected [0.23] for domain implicit_files!
==> amazon-ebs: Higher version of database is expected!
==> amazon-ebs: In order to upgrade the database, you must run SSSD.
==> amazon-ebs: Removing cache files in /var/lib/sss/db should fix the issue, but note that removing cache files will also remove all of your cached credentials.
==> amazon-ebs: Could not open available domains
    amazon-ebs: [Unit]
    amazon-ebs: Description=Node Exporter
    amazon-ebs: After=network.target
    amazon-ebs:
    amazon-ebs: [Service]
    amazon-ebs: User=node_exporter
    amazon-ebs: Group=node_exporter
    amazon-ebs: Type=simple
    amazon-ebs: ExecStart=/usr/local/bin/node_exporter
    amazon-ebs:
    amazon-ebs: [Install]
    amazon-ebs: WantedBy=multi-user.target
==> amazon-ebs: Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service → /etc/systemd/system/node_exporter.service.
    amazon-ebs: ● node_exporter.service - Node Exporter
    amazon-ebs:    Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; vendor preset: disabled)
    amazon-ebs:    Active: active (running) since Wed 2023-06-14 17:21:23 SAST; 259ms ago
    amazon-ebs:  Main PID: 97269 (node_exporter)
    amazon-ebs:     Tasks: 4 (limit: 48718)
    amazon-ebs:    Memory: 5.2M
    amazon-ebs:    CGroup: /system.slice/node_exporter.service
    amazon-ebs:            └─97269 /usr/local/bin/node_exporter
    amazon-ebs:
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=thermal_zone
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=time
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=timex
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=udp_queues
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=uname
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=vmstat
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=xfs
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=node_exporter.go:117 level=info collector=zfs
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100
    amazon-ebs: Jun 14 17:21:23 ip-10-6-33-126.af-south-1.compute.internal node_exporter[97269]: ts=2023-06-14T15:21:23.063Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100

==> amazon-ebs: Stopping the source instance...
    amazon-ebs: Stopping instance
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating AMI qWTWdoF from instance i-1111222223333
    amazon-ebs: AMI: ami-123456789210
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Copying/Encrypting AMI (ami-123456789210) to other regions...
    amazon-ebs: Copying to: af-south-1
    amazon-ebs: Waiting for all copies to complete...

It seems as though the image build is successful, but the whole process fails right at the end when Packer is waiting for the ec2 instance to shut down for reasons we are not sure of yet. Kindly assist as we do not know what could be causing the process to fail right at the end.

Thank you in advance, looking forward to hearing from you.

Kind Regards
Tshepo

safenetwork-CMTY · June 18, 2023, 9:00pm

I have no knowledge of AWS EBS,
so I can only ask stupid questions and draw from my experience
from my many failings to create a VM using packer myself.

From my experience, failing to shut down = failing to make an ssh connection.
In my case, I use an ssh private key,
which means I need to make an ssh connection to the iso first and later the box,
and that also means that in my scripts I need to add the private key to both the iso the box.
Have you done something similar to that?

Is ssh used as it’s communicator?
I see an ssh_username, where’s the password?
I see access_key and secret key variables, are those used for the ssh connection?

tmohlatlole · June 21, 2023, 7:08am

Hi, thank you for the prompt response.

We do not think the issue is a failure to establish ssh connection because the provisioners’ block would not execute if this were the case, i.e. the shell script cannot run without the build server being able to ssh onto the image server to execute the scripts.

the provisioners’ block I am referring to is:

"provisioners": [
        {
            "type": "shell",
            "environment_vars": ["DD_API_KEY={{user `dd_api_key`}}",
                        "DD_SITE={{user `dd_site`}}",
                        "DD_AGENT_MAJOR_VERSION={{user `dd_agent_version`}}"],
            "script": "scripts/common.sh"
        }
    ]

Furthermore, all packages are installed by the script which means the connection is definitely established.

We are still puzzled and any assistance will be appreciated.

safenetwork-CMTY · June 21, 2023, 7:48am

We do not think the issue is a failure to establish ssh connection because the provisioners’ block would not execute if this were the case, i.e. the shell script cannot run without the build server being able to ssh onto the image server to execute the scripts.

In my experience packer makes at least two ssh connections, after the boot script with (what I call) the liveVM and once the provisioners are done, with (what I call) the buildVM.
Though again, I’m not familiar with AWS.

On my setup, I turn on SSH on the liveVM during boot script and I turn on SSH on the buildVM in the provisioner scripts.

Topic		Replies	Views
Amazon-ebs builder: issues with copying amis to different regions when ebs encryption by default is enabled Packer	0	1303	November 15, 2019
Inputting Access_Key and Secret Key variables for AMI build using Packer container does not work with AWS Packer	5	3135	May 26, 2021
Packer builder amazon-ebs authentication. Is it encrypted? Packer	0	272	May 2, 2022
How do I correctly set 'AMIRegionKMSKeyIDs'? Packer	1	340	November 24, 2023
We are trying to use packer latest version 1.9.1 or 1.8.x in our pipelines and running into below issue Packer	1	261	July 27, 2023

Packer Build Not Working with AWS EBS Encryption using KMS(CMK)

Related topics