If you create multiple images via a packer/terraform pipeline (ami,openstack, vmware) is it possible to scan the template file for potential os level vulns associated with the images that will be built? The alternative now is actually spinning up the images and scanning them dynamically with something like Nessus. Is there any tool that can shift this process to the left and scan the code files?
Hi @skalamaras ! Shifting left is an admirable goal, but there are as always many ways to skin that cat.
Your goal is to decide whether, based on a declaration of state, if that state exposes potential vulnerabilities or misconfigurations. What you really want is not to deploy a vulnerable state, but you don’t want to find out after the fact, hence the shift left. However, let’s remember that it’s the state we are interested in, not the declaration.
The state is a product of two things:
- Upstream state (ie, the source image)
Can we, just by looking at a Packer template, decide whether the upstream state is vulnerable? I don’t think anyone keeps a 100% up-to-date database of vulnerable images, this would be impossible, since images can be modified in arbitrary ways. The best you can do is to check the state for known vulnerabilities, by comparing it to relevant standards. For the installed packages, check them against the vulndb, compare sha’s, etc , for the filesystem, compare with a hardening compliance profile.
In short: No, you cannot tell from the declaration of the upstream image whether it is vulnerable. You have to look inside the image.
If the provisioner is declarative itself, such as the Ansible provisioner, then you can look at the statement and decide whether vulnerabilities are may be introduced. This is inspecting a component though, not the Packer template, and it should be indeed be done upstream, or even better, completely out of band. IE, you have a whole other pipeline which assures the quality of the components (Ansible playbooks, roles) rather than the product (the image made by Packer).
Provisioners change state
Remember that the provisioners change the state. So, if you want to shift left by looking at the upstream image and deciding whether to break the build, how can you be sure that your subsequent application of the provisioners won’t fix the problem? You now have a paradox:
- If (the image is vulnerable) and (the provisioners will fix it) but (you shift left and fail on image inspection) then
- The provisioner will never get executed
- You have wasted effort on writing something that never gets used.
However if you don’t make assertions on the upstream image, and the provisioner declares non vulnerable state then you can inspect the actual product (your final image that Packer has built) before you commit the image.
Use compliance as code as provisioners
So, we come to the conclusion: Yes, shift left, but in the packer template itself.
You cannot inspect only the Packer template and decide whether the final image will be defective, based on the arguments I’ve outlined. For one thing, there is nothing to compare upstream images to, for another, you will end up wasting engineering time.
Pipeline execution time is much less expensive than engineering time, so I typically add these compliance statements as provisioners in the Packer template itself. Typically I add one or all of:
- A trivy scan of the final image before it is committed via a
- An Inspec profile via the Inspec provisioner (although this is apparently not currently maintained)
- A TestInfra compliance profile via the shell provisioner
There are plenty of options !
The point however is you cannot tell just from looking at the declaration, you must inspect the final product, because provisioners change state.
Your pipeline must ensure the quality of both the product as well as the components.
Thanks for the suggestions. Can you provide an example code for Trivy and how it can be used via a shell provisione.
This is the simplest example I have at hand:
It runs the scanner on the image before committing it. In this case, it runs inside the container or image that is being built, and as such might leave a trace if not cleaned up properly – TrivyDB files, etc.
You can also decide to fail the build later, in the post-processing using a
In this case, it runs outside the built image, on the machine that is actually running packer.