I am trying to debug a provider binary using delve with terraform 0.13.4. I invoke the binary as follows:
$ dlv exec .local/share/terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit -- -debuggable
Following the instructions from the flag’s log message I then invoke terraform as:
$ TF_REATTACH_PROVIDERS='{"terraform.resinstack.io/resinstack/linuxkit":{"Protocol":"netrpc","Pid":2199,"Test":true,"Addr":{"Network":"unix","String":"/tmp/plugin714804218"}}}' TF_LOG=debug terraform apply
However, none of my breakpoints are hit and delve doesn’t show the supervised process doing anything. I am reasonably confident that based on the logs that terraform isn’t reattaching to the address as provided:
2020-10-29T11:35:52.448-0700 [INFO] plugin: configuring client automatic mTLS
2020-10-29T11:35:52.469-0700 [DEBUG] plugin: starting plugin: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit args=[.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit]
2020-10-29T11:35:52.469-0700 [DEBUG] plugin: plugin started: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit pid=4817
2020-10-29T11:35:52.469-0700 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit
2020-10-29T11:35:52.478-0700 [INFO] plugin.terraform-provider-linuxkit: configuring server automatic mTLS: timestamp=2020-10-29T11:35:52.478-0700
2020-10-29T11:35:52.500-0700 [DEBUG] plugin.terraform-provider-linuxkit: plugin address: address=/tmp/plugin783949786 network=unix timestamp=2020-10-29T11:35:52.499-0700
2020-10-29T11:35:52.500-0700 [DEBUG] plugin: using plugin: version=5
2020-10-29T11:35:52.544-0700 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-10-29T11:35:52.546-0700 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit pid=4817
2020-10-29T11:35:52.546-0700 [DEBUG] plugin: plugin exited
My question is simple: why doesn’t this work. I’m trying to get to the point where I can single step through a provider that is exhibiting strange behavior. Are there other (undocumented) flags or variables that need to be set to enable terraform to attach to an unsupervised plugin?
1 Like
Will try my best - GO is outside my wheelhouse, but I’ve been facing a similar situation - wth is happening with my custom provider? - and sort of got it working. A better way than what I’ve done must exist. AFAIK, none of this appears in the Terraform docs or hashicups example, or on any one site. I collected it from several different places on the interweb.
The provider itself needs to be set up for debugging. I modified my provider’s main() to have a debug mode, like so:
func main() {
var debugMode bool
flag.BoolVar(&debugMode, "debuggable", true, "set to true to run the provider with support for debuggers like delve")
flag.Parse()
if debugMode {
err := plugin.Debug(context.Background(), "mycorp.com/mycorp/coffee",
&plugin.ServeOpts{
ProviderFunc: func() *schema.Provider {
return coffee.Provider()
},
})
if err != nil {
log.Println(err.Error())
}
} else {
plugin.Serve(&plugin.ServeOpts{
ProviderFunc: func() *schema.Provider {
return coffee.Provider()
},
})
}
}
I built my provider binary with a couple of extra compiler flags:
-o terraform-provider-coffee -gcflags '-N -l'
This disables compiler optimizations, making it easier to see/understand what the code is doing.
Fire up the resulting binary via delve:
$ dlv exec terraform-provider-coffee -- -debuggable
(dlv) c to continue
That gets you your TF_REATTACH_PROVIDERS string.
HTH. Unfortunately, I don’t have any more information about delve - once I sort of figured out this process, I was able to set up GOLAND and am using its breakpoints/debugger.
I am on windows trying to debug a provider. I build the provider binary without any optimizations, adding debug opts in main.go
I tried debugging through vs code and delve using code (dlv debug . -- --debug), debugging the executable from the same git repo, adding my provider to dev_overrides, tried debugging binary from all the locations – %AppData%/terraform.d folder and from .terraform folder that is created after running terraform init.
Delve starts as usual, am able to add breakpoints, it outputs the TF_REATTACH_PROVIDERS string, I am setting it as env variable, able to echo it as well, but when running terraform apply it does not reattach the provider. Tried with cmd, powershell, git bash and on WSL. Same problem.
Also, one more thing to mention is that even tho I am using dev_overrides, I have to run terraform init, can’t run terraform apply directly (I read on the documentation that if using dev_overrides no need to do terraform init)
For the sanity of future me, posting a follow up here since this is where I landed while wasting several hours trying to troubleshoot why TF_REATTACH_PROVIDERS was seemingly being ignored by terraform.
It turns out that the second argument to plugin.Debug() needs to be a string that matches the source declared in the terraform terraform { ... required_providers{} } block.
For example, trying to track down a bug in the onelogin TF provider, the Terraform should read as:
required_providers {
onelogin = {
source = "local/onelogin" # <== this source
version = "0.2.0"
}
}
}
needs to match in the golang terraform-provider-onelogin provider code
plugin.Debug(context.Background(), "local/onelogin", opts)
I’m sure this is documented somewhere and I just missed it, and/or can’t read the output to put 2 and 2 together.