I have used terraform taint a number of times for various things. Today I read that terraform apply -replace should be used instead. However, that does not work in a Terraform Cloud VCS-controlled workspace. It produces this error: “Error: Apply not allowed for workspaces with a VCS connection” Is there another option? Should terraform taint continue to be used in this situation?
Hi @steve_schram,
If terraform taint is currently working for you then there is no harm in continuing to use it. It still behaves as documented.
terraform apply -replace=... is preferable for many people because it performs the whole process in a single run, rather than there being a brief period where the object is marked as damaged (which is what “tainted” represents) which risks someone else pushing a new configuration in the meantime and the two operations ending up merged together.
However, you’re right that in the VCS workflow you do not run terraform apply directly and so that exact approach will not work there. In Terraform Cloud this “replace” idea is one of the Run Options, so another way to activate it would be to create a run using the Terraform Cloud API with the replace-addrs property set to contain the addresses of the objects you want to replace. This should be possible because in the underlying API (unlike in Terraform CLI) the act of uploading a new configuration and the act of starting a new run are separated, so it’s possible to create a new run without uploading a new configuration.
(Uploading a new configuration is the part of terraform apply that doesn’t make sense for a VCS-integrated Workspace, because for those workspaces you always upload new configurations indirectly via your version control system.)
Unfortunately the UI for manually starting runs does not currently have support for setting replace-addrs; it supports only a subset of the available Run Options. If triggering this through the web UI would be an acceptable solution for you, I suggest sending your use-case to HashiCorp Support who can then pass the feedback on to the Terraform Cloud product teams.
Thank you for the detailed reply, @apparentlymart
I am content with terraform taint but, because of the deprication notice, I became concerned that it will be removed. Of course, it would be better to have the full capability of Terraform in the cloud environment, and I’ve often wondered why it is so limited. This is just one more case where we are constrained to a subset of full scope of Terraform.
By the way, I only see small subset of the documented run types on Terraform Cloud:
.
Is that a limitation of our subscription type?
Hi @steve_schram,
When using Terraform Cloud’s VCS integration, you effectively replace the ability to arbitrarily upload configuration to Terraform Cloud with the requirement that all configuration changes must be made through version control. I understand that this can seem like you have less functionality in that mode; for other customers those restrictions are a key requirement of the product, because they wish to enforce a particular workflow in VCS and to prevent operators from using the terraform apply command to circumvent that workflow.
If you want to retain the flexibility of arbitrary runs of terraform apply from the command line then one way to achieve that would be to disable Terraform Cloud’s opinionated VCS integration and instead run Terraform CLI from an automation system offered by your Git platform, such as GitHub Actions if you are using GitHub. This will then give you the option to bypass version control when you need/want to, and just run terraform apply directly in such cases.
You can still use Terraform Cloud’s remote execution environment and various other features in that case. You would just be disabling Terraform Cloud’s automatic import of configuration from version control, and with it the requirement for all configuration changes to come from version control.
The web UI for starting runs is far less frequently used than other workflows, so the Terraform team focused on implementing the most commonly-used options first and have not yet implemented support for more complex run options such as forced replacement. This is not a matter of account level, but rather of implementation prioritization.
That makes a lot of sense about keeping the VCS integration locked down, but not all terraform apply commands would violate VCS as the single source of truth. I think you identified the issue in the last paragraph: “the Terraform team focused on implementing the most commonly-used options first”. And I think that marking “taint” as deprecated is getting things out of order. It shouldn’t be deprecated before the “forced replacement” option is available on the UI. Can you see about getting that deprecation marker removed until the UI option for forced replacement is available? Should I submit a PR against the documentation?
Right now, yes they would, because terraform apply always runs locally.
It would be a lovely feature improvement, though, if there was a CLI command for triggering a remote apply operation.
terraform apply, when running in remote operations mode, does two main actions:
- Upload the content of the current working directory as a new configuration version in Terraform Cloud.
- Start a new run to plan and apply that new configuration version.
The VCS integration feature blocks step 1, because that would allow introducing some code that isn’t guaranteed to be in the relevant version control branch
Step 2 is allowed for a VCS-integrated workspace, and is where the “replace” option would be handled, but because this command combines both of these steps together it isn’t possible to reach step 2 without seeing the error in step 1.
Terraform CLI remote operations are designed to mimic the behavior of Terraform local operations as much as possible, and so there currently isn’t any command for “create a new run with whatever configuration you already have”; in local mode there would be no “configuration you already have”, so it just always uses the one in the current working directory, and this behavior allows people to adopt Terraform Cloud remote operations without significantly changing their existing use of Terraform CLI.
Step 2 is exposed as part of Terraform Cloud’s API, and that’s what the web UI is ultimately doing when you start a run there, but since replace is used relatively rarely there hasn’t been enough interest so far for the team which builds the web UI to add support for the “replace addrs” option in particular. However, if you contact HashiCorp Support about your use-case then they can log the feature request with the relevant team. (Terraform Cloud teams expect to get feedback primarily through the support team, rather than in this community forum that’s mainly for peer-to-peer help.)
Thanks for the very good explanation of the operation of terraform apply. Unfortunately, I have been too vague in my comments. A fault I need to work on, for sure. My thoughts were more along the lines of the theoretical operation of “Apply”, not the specific terraform apply CLI command. For instance, if every type of “Apply” could be done in the Cloud, no CLI terraform apply would need to be done apart from preview plans to exercise code changes before a VCS commit.
So when you talk about contacting HashiCorp support, is that the answer to my question about removing the deprecation notice?
Can you see about getting that deprecation marker removed until the UI option for forced replacement is available? Should I submit a PR against the documentation?
I don’t think it’s likely that this command will be undeprecated because it is deprecated due to a flaw in its design – that it modifies the state to mark the object as a separate step from actually replacing it, therefore creating risk of colliding with changes made by others – not because the replace functionality exists. The taint concept would still have that design flaw even if the replace option did not exist.
However, I was suggesting that you contact the support team to note that there is currently missing functionality from your perspective – that the -replace feature is not conveniently accessible when you are in VCS mode, because there’s no web UI for it – so that the teams responsible for the Terraform Cloud web UI can see that there is demand for using the replace option with the VCS workflow enabled and therefore adjust their prioritization of that accordingly.
In the meantime though, you can use the Terraform Cloud API to access this functionality, although I acknowledge that is far less convenient than using the web UI or Terraform CLI to do it.
I understand. I guess I have become accustomed to tools providing a new option well in advance of a deprecation, so that I just assumed that is how it always works. I will plan to do two things: report this to the Hashicorp support team, and add the functionality to our CLI tool tfc-ops.