I too would like to have transient resource, in my case it would be giving an access right to one provider profile while doing the deployment and remove it once everything is deployed.
This comes from this situation:
profile A has access to a S3 bucket that contains the CI generated artifacts
profiles B, C, D… create Lambda functions that are isolated from resources in profile A but still needs to access the CI generated artifacts that serve as a source for those lambda.
Right now, I use a local-exec provisioner to retrieve the zip files locally and use a filename reference on the lambda functions. But I believe it would induce less data transfer if profile B could be granted a temporary ListObject/GetObject ACL while it creates the lambda functions.
Why are we trying to use Terraform as a general automation platform? I don’t understand. Transient resources, step by step processes, if/then/else is not what Terraform is intended to do or expected to excel at.
“Build a server, create an AMI from it, remove the server” is absolutely a process that Terraform can handle:
Create a .tf file with the server built, run it to create the server.
Go grab a cup of coffee and pet your dog.
Create another .tf file with the AMI created, run it once the server exists to create the AMI
Go pet the dog again (they like that).
Create a third .tf file (or modify the first) with the server removed, run it after the AMI is created to remove the server.
That’s the terraform way of handling this. Why are we trying to turn a “descriptive state” tool into an “infrastructure automation” tool?
Terraform isn’t an automation tool. It doesn’t act as and isn’t intended to act as a “execute these tasks line by line till you get to the right ending.” It is a state tool…ephemerality doesn’t play into a state tool because there is no concept of an “ephemeral permanent state”. If you want terraform to create a route 53 record for you, create the record. If you want it to remove a record for you, remove the record. Those are two distinct states of the system. If you want the record to be created for you, do some things and then remove it, that isn’t a state, that’s an automated process…use a different tool.
Toolboxes aren’t filled with hammers. They have a hammer, a couple different screwdrivers, pliers, cutters and wrenches. No one tries to cut wood with a hammer, no one tries to unscrew a bolt with a screwdriver…that is insane…
Yet technical people try to “cut wood with a hammer” all the time…it baffles the mind. Go look at Ansible, Bash, Python, Jenkins/groovy just to name a few…you’ll find it very very easy to grant Profile B temporary access to a thing, do other things, and then revoke that access.
And your employer will appreciate you for not wasting your time trying to fit square pegs into round holes.
TF is an automation tool: it provides a declarative language to describe an end state and then automates generating and executing the sequential actions of getting there. It also provides a few declarative hooks for inserting custom sequential actions in a few different way and places. The big question is how much of that type of customization should be explicitly supported within and by TF?
The problem isn’t finding a solution for problem A, and then finding another solution for problem B, and then … Finding a hundred different solution for each of a hundred different problems is easy. The hard thing is that the complexity of systems tends to grow superlinearly with the number of different solutions used. Keeping that number small is critical to manageability. My rule of thumb; if your team doesn’t have at least 3 people who can, from memory, describe how to deploy your set up (e.g. “check out the config from $URL, run tf apply from $DIR until it quits changing things”) then have fun the next time someone what to take a few weeks off for vacation.
As applied to TF, the issue is that TF provides 99% of what a lot of people want, so hacking that last 1% into TF by working around it’s assumptions adds less overall complexity than partition that last 1% into some disconnected solution. A set of one or more TF projects, where applying them all repeatedly in any order until none of them want to change anything is assured to converge to a correct state, is (from the system operators view) a reasonably simple and robust solution even if it’s not how the devs designed TF ot be used. A hodgepodge of TF projects, 3rd party web UIs and other tools that need manual application, in the one right order, with the correct flags at each state, with manual edits to files at specific points? … If you ask me to deal with that, I’ll start by checking if the NDA allows me to post it (anonymized) to thedailywtf.com
BTW: what should I do when my boss hands me a square peg and points me at a round hole? A non-trivial chunk of my career can be classed as being knowingly assigned to do just that. (The cases where things “just fit” tend to not use up much time.) The experience to know when to use a lathe or broach, or just grab a big sledge, is why people get paid above minimum wadge to do this job.
Thanks @douglasw0 and @bcsgh! Your responses on this thread perfectly illustrate the two sides of the debate on provisioners in Terraform.
We have correctly identified that your question @obones is outside of the intended scope of Terraform, and so any solution that uses Terraform to solve this problem will be more of a work-around than a solution. The ultimate costs of that work-around may be higher than implementing a script or tool to work in conjunction with Terraform to get the job done; you’ll have to make a pragmatic decision here.
Unfortunately I do not have a good example for how to get this done with Terraform alone. As an example of the tools-based approach, check out this great post: Blue/Green workspace layout - #4 by apparentlymart. I hope this helps spark a solution to your problem.
Thank you for chastising me, it’s always a pleasure to get such answers when all I’m trying is to work around a (perceived) shortcoming of a tool.
If the aws_lambda_function and aws_lambda_layer_version had allowed me to indicate a different provider for their S3 source, I would not even have had to find workarounds and would not have wasted anyone’s time.
I just made a feature request in the provider’s project, but in the meantime, I’m left with a hacky solution that is a bit brittle…