Hi, sorry if this is in the docs or is a dupe, I’ve been surfing around and so far have not found anything that works the way we’d like it to.
We have a module that has all the configuration in it to deploy Jenkins to a Kubernetes cluster.
We then have around 40 end user modules that use the above module as their source.
We’ve been using Terragrunt to keep things DRY as I first wrote this a couple years back when Terraform had fewer capabilities and now I’m trying to figure out if we can get rid of Terragrunt and just use built in Terraform capabilities to do what we are trying to do.
Repositories and Variables
- A source repo called
terraform-jenkins-configuration(or TJC for short). This repo contains all the.tffiles to install Jenkins including variables, many of which have default values. - An end user repository that has the root Terraform module called
jenkins-latest. This repo has a source line pointing to theterraform-jenkins-configurationrepository. - The TJC repo has a variable defined called
clusterthat holds connection information including secrets like a token to the Kubernetes cluster we are talking to.
First Scenario
For most of the end user repositories, we use the cluster variable as is, just consuming its default properties, but for the jenkins-latest repository we need to pass in different values for the cluster variable. I know we can do that in the end user repo main.tf file, but this means we would have to have the cluster secret checked into the end users repositors and we don’t want our end users to be able to see that secret.
What I’d like to do is something like this in the end user jenkins-latest repository
main.tf
module "jenkins" {
jenkinsRepo = "jenkins-latest"
source = "../terraform-jenkins-configuration"
cluster = var.clusters.latest-cluster
}
And have the variable clusters be defined in the TJC source module. However, Terraform does not allow this. That var would have to be defined in the end user repository, thus defeating our purpose of not giving them the secret.
Yes, I know, we should be using vault or something similar, but we don’t have vault and aren’t going to for some time (if ever).
I’ve gotten this working by creating latest-cluster.tfvars in the source repo and having the data for the latest cluster stored there
i.e.
cluster = {
id = "<redacted>"
url = "<redacted>"
apiUrl = "<redacted>"
token = "<redacted>"
subDomain = "<redacted>"
jenkinsDeployerRole = "<redacted>"
projectOwnerSelRole = "<redacted>"
jenkinsProjectOwnerRole = "<redacted>"
jenkinsProjectMemberRole = "<redacted>"
}
and then passing it when I run terraform plan
i.e.
terraform plan -var-file="../terraform-jenkins-configuration/latest-cluster.tfvars"
But this then requires that I also have a cluster variable defined in the root module and assign cluster = var.cluster in the root module.
This works but is super clunky. I don’t like that we have to remember to pass
-var-file="../terraform-jenkins-configuration/latest-cluster.tfvars"
to terraform plan and terraform apply and it’s going to take some rework in our deployment system.
I’d love a way to do the entire definition inside of the root module’s maint.tf without using Terragrunt to auto generate the variable and tfvars into the root module structure at init.
Any ideas on how to accomplish this better?
Second Scenario
This is a bit different but is the other “last” thing we need to figure out to remove our dependance on Terragrunt
Right now we have one user account for all 40 of the Terraform states we store in Artifactory. Because of this we don’t want to store the remote_state file in the end users root module repository. If we did that, the end user admins could see the auth, hop into Artifactory and look at all the other users Terraform state files, which also have all of their secrets.
Yep, still know we need Vault, don’t get me started . . .
One way we are thinking of fixing this is to pre-create a bunch of service accounts and create a new state store for each instance. This is doable but is a huge PITA.
Right now we use Terragrunt to generate the remote_state file during the terragrunt init phase so it is never in the end users repository. This works fine but if there is a different better way of handling this now, it would be super cool.
For example if there was a way to tell the root module to get the remote_state from the source that would be awesome.