Eschewing Packer's Parallel build feature on CI

Packer
1

Is Packer’s Parallel builds feature irrelevant, or even downright unhelpful in the context of CI? Packer’s parallel builds feature is a bit selling point, but I’m about to intentionally not use it. Here’s why.

I’ve created some elaborate packer projects on Jenkins CI in order to build a lot of different variants of the same thing, leveraging HCL2 and Packer’s native parallel build feature. I have organized/chained the jobs together so that my “base image” jobs build several variants, and downstream jobs copy their base image and build upon that. It’s pretty neat.

What I’m finding, though, is that Packer isn’t very sophisticated in its ability to manage resources when building 6 or 7 virtualbox-iso builders concurrently, and the system becomes disproportionally crippled with the load… at least it appears much less sophisticated than, say, a build system where I used either:

  1. a dedicated parallel pipeline stage per packer builder, or
  2. one pipeline build per packer builder, then trigger a bunch of them in parallel from a “super-job” - using the multi-job approach would have the benefit of allowing access to build results immediately after a single “variant” is completed, instead of having to wait (sometimes a considerable time) for all to complete before the artifacts can be downloaded.

…and use Jenkins’ native tools and configuration to manage resources and adapt to busy situations.

Just wondering if anyone that has done Packer on CI system has run into this general concern and adapted somehow to it, perhaps in the ways I’m suggesting, perhaps in other ways… Thanks for your time.

1 Like
2

Hi!

We do something similar to what you are describing here. Not “build a lot of different variants of the same thing”, but rather build several tiers of an application in the same Packer template; I believe the end effect is more or less the same, and your post reminded me of some of the issues we had.

We have a similar setup - building and testing images with Jenkins, but we use the declarative pipeline format for job definitions. Using the matrix declaration, the pipeline is split into N parallel stages each with their own agent where Packer is instantiated from. In each parallel stage, we use Packer’s -only flag to build just the target we are interested in. Since we do not have any dependency between the builders, this can all be done in parallel. I suppose that you may not have this luxury, since you work from a common base image.

The pipeline is sped up by the number of available Jenkins executors and we can still test the resulting images independently, registering the test results with a junit in the Jenkinsfile.

As a general note, we have also seen that launching several builders on the same agent, which also execute on that agent, is a good way to slow things to a halt. Instead, if you’re building images which use remote builders, we’ve never run into problems. Of course this works for things like cloud providers, but not building local ISOs. For that, we’ve found it best to use a dedicated agent per builder.

I hope this anecdote is of some value,
Bruce

2 Likes
3

Thanks @brucellino1. Glad to hear that someone else has the same issue with concurrent, local packer builders.

I see, your “targets” are stages or environments (testing/dev/staging/prod). I’ve seen this done with Matrix Projects in Jenkins, but I’m not using them currently. I currently handle the parameterization of environments using ansible inventory…

In my case, the thing we’re building with packer is not an application server, but developer infrastructure (jenkins itself). So I have to be careful about which environments I can let packer recreate when the jenkins job runs. (I assume in your case you’re deploying these “tiers” in your matrix project in a way that you’ve designed so that the jenkins job can overwrite the target machine(s) every time, by design)

Not sure if this is interesting to you, but I’m working on a scheme by which I:

  1. Create a base image / template in vSphere
  2. Create an entire jenkins-testing cluster from this template using ansible inventory/variables (this part is done by the CI system, which assumes it can overwrite *-testing machines every time. The CI system uses stage=testing.)
  3. Create an entire jenkins-staging cluster from this template using ansible inventory/variables (this is done by a manual run of the ansible project, specifying stage=staging)
  4. Promote staging to production using ansible. This is also manual and involves extra steps that restore the backup of production jenkins cluster (basically the job history).

FWIW, we use a combination of declarative and scripted pipeline (typically declarative Jenkinsfiles, with scripted where necessary, and also groovy/scripted in shared libraries).

1 Like