Packer InSpec provisioner fails with "Error opening/processing"

carsoni · May 13, 2020, 3:43am

Hi Guys

I am using the Packer InSpec provisioner to run a hardening profile against AWS EC2 images which Packer is building.

I am storing the hardening profile in a self-hosted Gitlab instance but am having difficulty getting InSpec to correctly download the profile from Gitlab.

SSL connectivity was my initial problem but that has been solved by concatenating my local Root CA certificate to the cacert.pem in /opt/inspec/embedded/ssl/certs on the virtualbox runner I am using.

I can successfully download and run the dev-sec profile at “https://github.com/dev-sec/linux-baseline” but when I try to use the same URL structure to download from gitlab InSpec errors out.

Here is the debug run of the InSpec provisioner

 ==> hardening-paradigm-base-partitioned-server: Provisioning with Inspec...
 ==> hardening-paradigm-base-partitioned-server: Executing Inspec: inspec exec https://gitlab01.paradigm.local/paradigmsuite/PFMHardening --backend ssh --host 127.0.0.1 --key-files /tmp/packer-provisioner-inspec.596799465.key --user paradigmuser --port 41839 --input-file /tmp/packer-provisioner-inspec.197024820.yml -l debug
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:46.490098 #2176] DEBUG -- : [SSH] opening connection to paradigmuser@127.0.0.1
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:46.490167 #2176] DEBUG -- : [SSH] using options {:user_known_hosts_file=>"/dev/null", :port=>41839, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :auth_methods=>["none", "publickey"], :keys_only=>true, :keys=>["/tmp/packer-provisioner-inspec.596799465.key"], :password=>"<hidden>", :forward_agent=>nil, :non_interactive=>false, :verify_host_key=>false}
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:46.667158 #2176] DEBUG -- : [SSH] paradigmuser@127.0.0.1 cmd = cmd.exe /c ver
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:46.775177 #2176] DEBUG -- : [SSH] paradigmuser@127.0.0.1 cmd = Get-WmiObject Win32_OperatingSystem | Select Caption,Version | ConvertTo-Json
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:46.893801 #2176] DEBUG -- : [SSH] paradigmuser@127.0.0.1 cmd = uname -s
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:46.991013 #2176] DEBUG -- : [SSH] paradigmuser@127.0.0.1 cmd = uname -m
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:47.196134 #2176] DEBUG -- : [SSH] paradigmuser@127.0.0.1 cmd = test -f /etc/debian_version
 hardening-paradigm-base-partitioned-server: D, [2020-05-13T12:32:47.301559 #2176] DEBUG -- : [SSH] paradigmuser@127.0.0.1 cmd = test -f /etc/lsb-release && cat /etc/lsb-release
 hardening-paradigm-base-partitioned-server: [2020-05-13T12:32:47+10:00] DEBUG: Option backend_cache is enabled
 hardening-paradigm-base-partitioned-server: [2020-05-13T12:32:47+10:00] DEBUG: Resolve https://gitlab01.paradigm.local/paradigmsuite/PFMHardening into cache /home/paradigmuser/.inspec/cache
 hardening-paradigm-base-partitioned-server: [2020-05-13T12:32:47+10:00] DEBUG: Fetching URL: https://gitlab01.paradigm.local/paradigmsuite/PFMHardening
 hardening-paradigm-base-partitioned-server: [2020-05-13T12:32:48+10:00] WARN: Unrecognized content type: text/html; charset=utf-8. Assuming tar.gz
 hardening-paradigm-base-partitioned-server: [2020-05-13T12:32:48+10:00] DEBUG: Archive stored at temporary location: /tmp/inspec-dl-20200513-2176-1271vgv.tar.gz
 hardening-paradigm-base-partitioned-server: Error opening/processing /home/paradigmuser/.inspec/cache/caf35a9cd1b7f534ac7aef70b0d027fd3d0cc6bd8bbca9e1cc0eb8cb217901bf.tar.gz: not in gzip format
 hardening-paradigm-base-partitioned-server: [2020-05-13T12:32:48+10:00] DEBUG: Dependency does not exist in the cache https://gitlab01.paradigm.local/paradigmsuite/PFMHardening
 hardening-paradigm-base-partitioned-server: [2020-05-13T12:32:48+10:00] DEBUG: Fetched archive moved to: /home/paradigmuser/.inspec/cache/caf35a9cd1b7f534ac7aef70b0d027fd3d0cc6bd8bbca9e1cc0eb8cb217901bf.tar.gz

I noticed the “assuming .tar.gz” so based on that I provided a link to a .tar.gz file of the repository and then a .zip file but both produced the same error.

Do any of you know what the correct form of the profile address should or is there something that needs to be implemented in the profile to make it an acceptable download target?

Many thanks in advance
Regards
Ian Carson

carsoni · May 18, 2020, 7:27am

Answering my own question at least partially.

I still do not know what form of address works with the InSpec provisioner to get the archive .tar.gz of the profile from Gitlab. However it is possible to point directly at a profile file which has already been downloaded and placed in the directory of the repo in which it is to be used.

My InSpec provisioner entry looks like this

	{
		"type": "inspec",
		"profile": "./shared/scripts/PFMHardening.tar.gz",
		"user": "ubuntu",
		"extra_arguments": ["-l", "debug"]
	}

Hopefully that will be useful to someone.

I am still interested in finding out the answer to my original question

Thanks & Regards
Ian Carson

carsoni · May 21, 2020, 11:45pm

Update to my answer.

For anyone using CI/CD (I’m using Gitlab) it’s worth including a script to download the required file to the correct location at run time. It saves on manual copying and ensures access to the latest profile at all times