Proper process for plan in CI and apply in CD?

I have been trying to wrap my head around the correct process for using plan in CI and apply in CD.

The intended process is:

  1. PR open
  2. CI process runs tf plan
  3. CI saves the plan file, data dir (TF_DATA_DIR) and dependency lock file into an artifact
  4. Someone sees the plan file, “lgtm”, approves, merges
  5. CD sees merge, retrieves plan file, data dir, deplock file
  6. CD runs tf apply on that plan file

I think that is the right way to do it, but I am not sure. Much of the reference, e.g. this, of the deplock file assume some user is running it and can check it into VCS, which is not the case if no user ever can run plan or apply outside of a pipeline.

Question 1: Is this process correct? Are we missing something? Should we be removing something?

Question 2: Why would we get errors like "Error: Inconsistent dependency lock file"?

Thanks in advance as always.

Hi @deitch,

The dependency lock file is generated by either terraform init or terraform providers lock. The terraform plan and terraform apply commands only read the file, and then expect to find all of the selected providers already installed in the working directory and ready to use.

Therefore I would expect someone to have run one of the two commands I mentioned before they opened the PR, and thus the dependency lock file would be included in version control. The set of providers you depend on is an important part of how the configuration will be understood by Terraform, and so the dependency lock file should be in VCS along with the configuration.

Ultimately Terraform can’t actually tell whether you put the file in VCS or not as long as it’s present on disk by the time you are running Terraform CLI commands, but of course you’ll need to make sure your automation behaves equivalently to if that file had been included in version control, meaning that e.g. it must exactly match between plan and apply.

Hi @apparentlymart

I think you are saying that an Inconsistent dependency lock file error means, "I found a deplock and compared it with actual providers and there is a mismatch. Is that correct?

If so, then tf init must have run earlier, it saved both the deplock and the providers, and when the plan or apply ran (I think it is the apply stage that gets the error, but I need to double-check), it compared the existing providers to the deplock and found a mismatch.

The providers will be in TF_DATA_DIR, and the deplock is in the current working directory?

Bringing it back up to the higher level, what is the correct fully automated pipeline process? Obviously not going to run tf init and check it into VCS, as all of those providers shouldn’t be in there.

Digging in, I have a bit more detail. It definitely is from apply from an existing plan file.

Also, it is 2 specific errors:

74| Error: Inconsistent dependency lock file
75│ 
76│ The following dependency selections recorded in the lock file are
77│ inconsistent with the configuration in the saved plan:
78│   - provider registry.terraform.io/hashicorp/aws: required by this configuration but no version is selected
79│   - provider registry.terraform.io/hashicorp/time: required by this configuration but no version is selected
80│ 
81│ A saved plan can be applied only to the same configuration it was created
82│ from. Create a new plan from the updated configuration.

Our pipeline specifically does:

  1. CI: init
  2. CI: plan
  3. CI: save the following as artifacts: plan file; contents of TF_DATA_DIR; deplock file
  4. PR is approved and merged
  5. CD: retrieve the artifacts and installs them
  6. CD: apply the plan file. error occurs

Should we be doing something different?

The “but no version is selected” part of this error message suggests that the lock file doesn’t contain a block for each of these providers at all, or possibly that Terraform cannot find the dependency lock file and so it’s treating all providers as having no selections.

I can’t explain what would cause that to be true with the information available. Your step 5 of “retrieve all the artifacts” is presumably what’s responsible for giving Terraform the illusion that the apply is running in exactly the same directory as plan was, even though that’s not really true. My first instinct then would be that it isn’t quite rebuilding an identical working directory for some reason.

Is there some way you can compare the working directory at the completion of step 2 with the working directory at the completion of step 5? If this process is implemented correctly, the two should be identical.

If this process is implemented correctly, the two should be identical.

Yeah, that is exactly what threw me off. It should be, and only when we made sure to copy the lockfile did we start running into these problems.

It will take some time, but I will try it again, maybe with TF_LOG=TRACE enabled.

But it might take a few days at least. I will post back here.