Once a plan is finished what other tasks does terraform cloud perform? i have some simple workspaces that take 3 mins for the plan to return changes then it takes another 5 to 7 minutes where it stays in planning before returning and allowing the apply.
I can confirm this, I’m also seeing 5min+ delays once a plan has ‘finished’ before returning and allowing for apply.
This is a rather critical issue for me as I’m using a provider which expires credentials after 15mins and as a result plans are failing to apply cleanly due to these post plan delays.
Any feedback or guidance on how to avoid this would be very much appreciated.
Interesting. How is the
plan triggered? Any webhooks or integrated VCS? Are you using the free tier?
we were using the free tier as a POC and triggered plans via VCS PR’s. We did not get a solution so gave up and have moved to an internal jenkins based solution. Disappointing as apart from this it was a pretty good solution
The workspaces I’ve observed this on are connected to and triggered by GitHub VCS.
I primarily use the free tier but I’ve also observed this when on the trial plan.
- TF Version 0.13.5 (but I have also observed this on series 0.12)
- Environment Variable
TFE_PARALLELISMis set to
- I’m using plugins as placed in
terraform.d/plugins/..., which are ~90MB in size.
Are you able to access the web page describing the run while it’s in this situation of not returning even though the plan is apparently concluded? I wonder if the web UI will give some more details about what Terraform Cloud is doing at that time.
For me the WebUI displays the Plan panel, with the logs ending with the plan synopsis, e.g.:
Terraform will perform the following actions: ... Plan: 0 to add, 2 to change, 0 to destroy.
During this delay there is no other feedback indicating why the plan is still processing, with the Apply panel displaying as
Apply pending, and no Confirm button being available.
This most likely has to do with the size of your repository or the resulting size of the Terraform folder, after it has downloaded all the modules and providers.
The Terraform Cloud worker always start on a clean VM, therefore at the end of the plan phase, they pack and upload the entire folder, so it’s available for the apply. This happens after Terraform shows the proposed actions and is probably why this delay happens.
In order to additionally troubleshoot this, can you do the following:
- start in an empty folder on your computer or in a VM
- clone the repository from your VCS
- check the size of the repository with something like
du -s -h(make sure it’s the entire repository and not just the Terraform working directory if you use this option)
- check the number of files with something like
find . -type f | wc -l
- if you use working directory change into it
- perform “terraform init”
- again check the size of the entire repository
- again check the number of files
- last, but not least - share the results here
Additionally, for Terraform Cloud related issues, you can always open a ticket with support by emailing
@baconsi, just out of curiosity, did you open a ticket with support about this issue ?
Thanks @qwerty1979bg that makes perfect sense.
Size and file count after clone:
$ du -s -h 361M $ find . -type f | wc -l 190
Size and file count after
$ du -s -h 493M $ find . -type f | wc -l 200
Queuing a plan for this workspace on TF Cloud results in a ~7min delay between when the plan result is first displayed and the
confirm & apply button being available.
I’m going to whittle down this repo size and will post new numbers thereafter, thanks for the assistance!