TF_REATTACH_PROVIDERS appears ignored

I am trying to debug a provider binary using delve with terraform 0.13.4. I invoke the binary as follows:

$ dlv exec .local/share/terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit -- -debuggable

Following the instructions from the flag’s log message I then invoke terraform as:

$ TF_REATTACH_PROVIDERS='{"terraform.resinstack.io/resinstack/linuxkit":{"Protocol":"netrpc","Pid":2199,"Test":true,"Addr":{"Network":"unix","String":"/tmp/plugin714804218"}}}' TF_LOG=debug terraform apply

However, none of my breakpoints are hit and delve doesn’t show the supervised process doing anything. I am reasonably confident that based on the logs that terraform isn’t reattaching to the address as provided:

2020-10-29T11:35:52.448-0700 [INFO]  plugin: configuring client automatic mTLS
2020-10-29T11:35:52.469-0700 [DEBUG] plugin: starting plugin: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit args=[.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit]
2020-10-29T11:35:52.469-0700 [DEBUG] plugin: plugin started: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit pid=4817
2020-10-29T11:35:52.469-0700 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit
2020-10-29T11:35:52.478-0700 [INFO]  plugin.terraform-provider-linuxkit: configuring server automatic mTLS: timestamp=2020-10-29T11:35:52.478-0700
2020-10-29T11:35:52.500-0700 [DEBUG] plugin.terraform-provider-linuxkit: plugin address: address=/tmp/plugin783949786 network=unix timestamp=2020-10-29T11:35:52.499-0700
2020-10-29T11:35:52.500-0700 [DEBUG] plugin: using plugin: version=5
2020-10-29T11:35:52.544-0700 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-10-29T11:35:52.546-0700 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/terraform.resinstack.io/resinstack/linuxkit/1.0.0/linux_amd64/terraform-provider-linuxkit pid=4817
2020-10-29T11:35:52.546-0700 [DEBUG] plugin: plugin exited

My question is simple: why doesn’t this work. I’m trying to get to the point where I can single step through a provider that is exhibiting strange behavior. Are there other (undocumented) flags or variables that need to be set to enable terraform to attach to an unsupervised plugin?

Will try my best - GO is outside my wheelhouse, but I’ve been facing a similar situation - wth is happening with my custom provider? - and sort of got it working. A better way than what I’ve done must exist. AFAIK, none of this appears in the Terraform docs or hashicups example, or on any one site. I collected it from several different places on the interweb.

The provider itself needs to be set up for debugging. I modified my provider’s main() to have a debug mode, like so:

func main() {

	var debugMode bool

	flag.BoolVar(&debugMode, "debuggable", true, "set to true to run the provider with support for debuggers like delve")
	flag.Parse()

	if debugMode {
		err := plugin.Debug(context.Background(), "mycorp.com/mycorp/coffee",
			&plugin.ServeOpts{
				ProviderFunc: func() *schema.Provider {
					return coffee.Provider()
				},
			})
		if err != nil {
			log.Println(err.Error())
		}
	} else {
		plugin.Serve(&plugin.ServeOpts{
			ProviderFunc: func() *schema.Provider {
				return coffee.Provider()
			},
		})
	}
}

I built my provider binary with a couple of extra compiler flags:

-o terraform-provider-coffee -gcflags '-N -l'

This disables compiler optimizations, making it easier to see/understand what the code is doing.

Fire up the resulting binary via delve:

$ dlv exec terraform-provider-coffee -- -debuggable

(dlv) c to continue

That gets you your TF_REATTACH_PROVIDERS string.

HTH. Unfortunately, I don’t have any more information about delve - once I sort of figured out this process, I was able to set up GOLAND and am using its breakpoints/debugger.