I have come across similar frustration and confusion. Although I haven’t yet found this particular Tao, a few realisations did help to illuminate.
The first is that there are different kinds of “Declarative Infrastructure as Code”. Ansible, Packer and Terraform are all declarative IaC, but Packer is not idempotent.
- a Terraform statement is statement of reality. You are telling the world to be in a particular state. If it is not in such a state, Terraform will try to manipulate whatever resources it has at its disposal to make the world be so; but if it is, Terraform’s plan will consist of zero actions, and all will be well with the world.
- An Ansible playbook is a statement of state. You are telling a given artifact in the world to be in a particular state. If it is not, Ansible will manipulate whatever resources it has at its disposal to make the artifact be so.
- a Packer template is a recipe. Start with this, use these tools and produce the following. Using the same ingredients and recipe will always make cakes that taste the same, but they will be different cakes – if that renders the idea. This is not the same as “if there is already a cake, don’t make it again”, ie Packer is not idempotent.
Realisation 1: Ensure that Terraform and Ansible act on different levels of the world. Don’t use Ansible to change things that are in Terraform’s state, e.g. If you can separate out what Ansible is repsonsible for from what Terraform is responsible for, they are essentially blind to each other and won’t argue over the nature of reality. IE they won’t see a difference in their perception of reality.
Realisation 2: Trick Terraform into thinking that it’s resources are immutable. If your Terraform state includes an AMI, but doesn’t make statements about the content of that AMI, then Terraform won’t try to change it.
This led me to the following workflow:
- Create AMI with Packer + Ansible playbook
- Plan/Apply with Terraform
- Deployed state s with image i in state i_s
- Make changes to playbook
- Run playbook against image state i_s image state i_s’
- Ansible has made changes to the image state, but Terraform thinks it’s the same image!
- Commit changes to playbook
- BACK TO STEP 1 ABOVE.
What you can see now are two processes - the outer loop (Packer + Ansible + Terraform) which produces versioned, well-known desired states, and the inner loop where changes to a single artifact’s state.
The inner loop is where you get your fast feedback.
The outer loop is something you can codify into a delivery pipeline, since it’s triggered by commits to your playbook (or other changes to the declaration of state).
Hope that helps