Terraform detailed-exitcode causes PLAN to fail, when exit code = 2

[darrens280]

Hi. I have a Terraform plan running as part of Azure DevOps CI/CD pipeline YAML template that runs with -detailed-exitcode CLI switch. This template has been used by many other pipelines without issue. However now I have come across some strange behaviour.

Hoping someone can assist with why this might be occurring…

The PLAN runs ok, however the pipeline fails/errors with exit code 1 as shown below.

Plan: 2 to add, 0 to change, 1 to destroy.


─────────────────────────────────────────────────────────────────────────────

Saved the plan to: myplanfile.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "myplanfile.tfplan"
Terraform plan exit code = 2

##[error]Script failed with exit code: 1

• Exit code 2 would be expected since there are changes. I have added simple Write-Host "Terraform plan exit code = $LASTEXITCODE" at the end of Terraform Plan task in the pipeline, and this confirms exit code 2, however the pipeline still fails as per extract above.
• I have enabled TF_LOG=debug, and also TF_LOG=trace to further investigate, however these do not show any errors in the entire log.
• Weirdly, if I add terraform show -json $(planfile) after the plan runs, then, my write-output correctly shows exit code 2, and the plan file is displayed in json, and the entire pipeline then succeeds! Very strange.
• Also if I remove the -detailed-exitcode switch, then all works as expected, however we wish to leave this in for drift detection, so removing this is not viable.

My Terraform template looks like this:

# TERRAFORM PLAN TASK

parameters:
  - name: azureDevOpsServiceConnectionName
    type: string

steps:
  - task: AzureCLI@2
    displayName: 'Terraform PLAN'
    inputs:
      azureSubscription: ${{ parameters.azureDevOpsServiceConnectionName }}
      scriptType: pscore
      scriptLocation: inlineScript
      workingDirectory: $(workingDirectory)
      inlineScript: |

        terraform plan '--var-file myvars.tfvars -out $(planfile) -detailed-exitcode'

        Write-Host "Terraform plan exit code = $LASTEXITCODE"

    env:
      ARM_USE_OIDC: true
      ARM_ADO_PIPELINE_SERVICE_CONNECTION_ID: $(service-connection-id)

Hoping someone will be able to advise/assist. Thanks

[darrens280]

Small update - I found in the HashiCorp documentation, that using PowerShell on Windows (ie. my Azure DevOps self-hosted agents) to run Terraform PLAN is not recommended, and that command prompt is preferred (see here, about half way down).

I therefore tested by amending my YAML template, AzureCLI@2 task to use, scriptType: batch (instead of pscore) - now the pipeline gives the correct exit code of 2 (whereas pscore gives 1), however the pipeline still fails, when according to HashiCorp documentation regarding terraform plan and -detailed-exitcode parameter, an error code of 2 actually means success, but with changes found….. so why is my pipeline failing?

Plan: 2 to add, 0 to change, 1 to destroy.


─────────────────────────────────────────────────────────────────────────────

Saved the plan to: myplanfile.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "myplanfile.tfplan"

##[error]Script failed with exit code: 2

[apparentlymart]

Hi @darrens280,

The usual convention for command line tools like Terraform is that only exit code zero represents success and any nonzero status code represents failure.

Terraform breaks that convention when you use -detailed-exitcode by using a nonzero exit code to represent a special kind of success. The reason Terraform only does this when you opt in is because it’s nonstandard, and so likely to cause problems for scripts that aren’t written to handle it.

If you want to use -detailed-exitcode in your automation then you need to make sure your script includes code to somehow make either PowerShell or the Windows Command Interpreter not treat the nonzero exit code as a failure.

I’m not familiar enough with Azure DevOps, PowerShell, or the Windows Command Interpreter to make a confident suggestion, but your example script was able to print that terraform plan’s exit code was 2 and so it does seem like PowerShell was able to continue executing the script, but perhaps it’s “remembering” that terraform plan returned a nonzero exit code and so is reporting to Azure Devops that the entire script failed.

The documentation for $LASTEXITCODE says that PowerShell only sets it after running a “native program” like Terraform, so I might try something like this but I’m just guessing based on the documentation and haven’t actually tried it:

terraform plan '--var-file myvars.tfvars -out $(planfile) -detailed-exitcode'

if ($LastExitCode -eq 2) {
  # -detailed-exitcode makes Terraform use exit status 2
  # to represent successful execution with changes pending,
  # so this script should still succeed in that case.
  exit 0
}

My hope is that exit 0 will make PowerShell itself report a successful exit code to Azure DevOps, and so your job should be treated as successful, overriding PowerShell’s assumption that Terraform’s exit status 2 represents an error.

---

By the way: the warnings about using PowerShell to run Terraform are about some bugs in how PowerShell handles command line parsing for native commands.

I don’t think your specific usage here is encountering any of those problems because Terraform appears to be succeeding, so I don’t think you need to worry about that warning here. The way those problems would appear is Terraform reporting that the command line arguments are invalid, which would cause Terraform to fail with an error much earlier in its work before it even tries to create a plan.

[darrens280]

Thank you for taking the time to respond.

I was hoping to avoid a direct side-step of the error code, by saying if it’s 2, then make it 0. That just feels dirty.

And thank you for your feedback regarding scriptType. I have since changed the task back to using pscore.

---

UPDATE:

I have managed to get the pipeline, and exit codes to function correctly with their expected outputs, by adding the following input to my pipeline task AzureCLI@2… that runs the Terraform PLAN.

powerShellIgnoreLASTEXITCODE: true

When testing with an additional Write-Output to show me the the actual value of the exit code, I correctly received exit code 2, (which was expected, since TF found changes) and the pipeline completed successfully. Previously, without the above input, I was getting exit code 2, but pipeline would fail.

I’ve never had to use this input before, in combination with -detailed-exitcode so am still not entirely sure why it’s required now.

The official Microsoft documentation has this to say about this specific input switch:

powerShellIgnoreLASTEXITCODE - Ignore $LASTEXITCODE
boolean. Optional. Use when scriptType = ps || scriptType = pscore. Default value: false.

If this input is false, the line if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE } is appended to the end of your script. This will propagate the last exit code from an external command as the exit code of PowerShell. Otherwise, the line is not appended to the end of your script.

Thanks again @apparentlymart

Hope this may help someone in future