Terraform AWS Provider Panic & Plugin did not respond

Terraform Providers AWS
1

Hello, I’m facing some plugin issues, maybe someone has already solved it:
I’ve tried many versions of AWS provider, but the result is same, and I’ve reverted plugin to default version that I’ve used:

Here is terraform provider setup

terraform              {
   required_version    = ">= 0.12.0, < 0.16.0"

   required_providers  { 
     aws               = ">= 3.0.0,  < 3.4.0"
  #  aws               = ">= 3.6.0,  < 4.20.0"
     local             = ">= 1.4.0,  < 4.20.0"
     template          = ">= 2.2.0,  < 4.20.0"
   }
}

Actual error
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may
│ contain more details.
╵

Stack trace from the terraform-provider-aws_v3.34.0_x5 plugin:

panic: interface conversion: interface {} is nil, not map[string]interface {}

goroutine 203 [running]:
github.com/terraform-providers/terraform-provider-aws/aws.buildLaunchTemplateData(0xc00127d780, 0x75d0a13, 0x4, 0x60218c0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_launch_template.go:1379 +0x2cbf
github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsLaunchTemplateCreate(0xc00127d780, 0x68c44a0, 0xc00205d600, 0xa, 0xffffffffffffffff)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-aws/aws/resource_aws_launch_template.go:661 +0xcc
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc0009b4e00, 0x80780a8, 0xc001267c80, 0xc00127d780, 0x68c44a0, 0xc00205d600, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:318 +0x1ee
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0009b4e00, 0x80780a8, 0xc001267c80, 0xc001263960, 0xc0010a4460, 0x68c44a0, 0xc00205d600, 0x0, 0x0, 0x0, ...)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:456 +0x67b
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc0000a8768, 0x80780a8, 0xc001267c80, 0xc0016c2f00, 0xc001267c80, 0x73e5740, 0xc001b49500)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/grpc_provider.go:955 +0x8ef
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ApplyResourceChange(0xc001ac31e0, 0x8078150, 0xc001267c80, 0xc001263810, 0xc001ac31e0, 0xc001b49560, 0xc000360ba0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:332 +0xb5
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler(0x73e5740, 0xc001ac31e0, 0x8078150, 0xc001b49560, 0xc00105cfc0, 0x0, 0x8078150, 0xc001b49560, 0xc0012c0000, 0x2849)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:380 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002ab500, 0x8097b98, 0xc00036c600, 0xc001d65900, 0xc00065cab0, 0xb4b0200, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0002ab500, 0x8097b98, 0xc00036c600, 0xc001d65900, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000de82c0, 0xc0002ab500, 0x8097b98, 0xc00036c600, 0xc001d65900)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:857 +0x1fd

Error: The terraform-provider-aws_v3.34.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
1 Like
2

After adjusting the version requirements did you run terraform init -upgrade or remove the plugins from the .terraform directory?

1 Like
3

Hi, @stuart-c

Yes, I’ve did everything you’ve mentioned
Plus I’ve completely remove the .terraform directory and .lock files
Additionally I’ve tested in clean directory with terraform init, plan, apply

The result is same

4

Facing exact same issue.

5

Facing a similar issue here after upgrading to 1.0.8 and yes I deleted the .terraform dir before redoing init.

│ Error: Request cancelled
│
│ with module.ds-bigquery.google_bigquery_table.management_zone_views[“view_name”],
│ on ds-bigquery/bq-management-views.tf line 7, in resource “google_bigquery_table” “management_zone_views”:
│ 7: resource “google_bigquery_table” “management_zone_views” {
│
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵

Stack trace from the terraform-provider-google_v3.61.0_x5 plugin:

panic: interface conversion: interface {} is string, not map[string]interface {}

goroutine 4466 [running]:
github.com/hashicorp/terraform-provider-google/google.bigQueryTablecheckNameExists(0xc000df9600, 0x1, 0x1, 0x0, 0x2d21b78)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-google/google/resource_bigquery_table.go:26 +0x164
github.com/hashicorp/terraform-provider-google/google.jsonCompareWithMapKeyOverride(0x2526520, 0xc000543940, 0x2526520, 0xc000543ae0, 0x2d22688, 0x40c201, 0xc000df9f30, 0x10)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-google/google/resource_bigquery_table.go:47 +0x63f
github.com/hashicorp/terraform-provider-google/google.jsonCompareWithMapKeyOverride(0x275dac0, 0xc000c5ec00, 0x275dac0, 0xc000c5f140, 0x2d22688, 0x1, 0xc000f90120, 0xc000fd1000)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-google/google/resource_bigquery_table.go:87 +0x3db
github.com/hashicorp/terraform-provider-google/google.jsonCompareWithMapKeyOverride(0x275dac0, 0xc000c5ebd0, 0x275dac0, 0xc000c5f110, 0x2d22688, 0xc000df9410, 0x194, 0x2526520)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-google/google/resource_bigquery_table.go:87 +0x3db
github.com/hashicorp/terraform-provider-google/google.jsonCompareWithMapKeyOverride(0x2526520, 0xc0005439a0, 0x2526520, 0xc000543b40, 0x2d22688, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-google/google/resource_bigquery_table.go:56 +0x73f
github.com/hashicorp/terraform-provider-google/google.bigQueryTableSchemaDiffSuppress(0x2bf5c3a, 0x6, 0xc000e2c900, 0x88c, 0xc000e2db00, 0x88a, 0xc0012ee200, 0xc000a17500)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-google/google/resource_bigquery_table.go:147 +0x253
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.diff(0xc000936870, 0x2bf5c3a, 0x6, 0xc000946280, 0xc0005426e0, 0x31f89e0, 0xc0012ee200, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.4/helper/schema/schema.go:972 +0x37c
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.schemaMap.Diff(0xc000936870, 0x31c7be0, 0xc000f6e380, 0xc00124a0e0, 0xc001264c30, 0xc00071fec0, 0x2bcafa0, 0xc0000b9200, 0x430fe00, 0x2561960, ...)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.4/helper/schema/schema.go:522 +0x215
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).SimpleDiff(0xc00093b2c0, 0x31c7be0, 0xc000f6e380, 0xc00124a0e0, 0xc001264c30, 0x2bcafa0, 0xc0000b9200, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.4/helper/schema/resource.go:446 +0x9f
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).PlanResourceChange(0xc0000d2960, 0x31c7be0, 0xc000f6e380, 0xc000319540, 0xc000f6e380, 0xc0012ea500, 0x320b3c0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.4.4/helper/schema/grpc_provider.go:693 +0x792
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).PlanResourceChange(0xc000dad120, 0x31c7be0, 0xc000f6e380, 0xc00124a000, 0xc000dad120, 0xc000e6c480, 0xc000847ba0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:315 +0xac
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x2b1ee60, 0xc000dad120, 0x31c7ca0, 0xc000e6c480, 0xc0012ea5a0, 0x0, 0x31c7ca0, 0xc000e6c480, 0xc000ee2000, 0x28b5)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:362 +0x217
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001bc700, 0x3202040, 0xc000702c00, 0xc000fbc700, 0xc00080c960, 0x42d0ec8, 0x0, 0x0, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.36.0/server.go:1217 +0x50a
google.golang.org/grpc.(*Server).handleStream(0xc0001bc700, 0x3202040, 0xc000702c00, 0xc000fbc700, 0x0)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.36.0/server.go:1540 +0xcfd
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000721c40, 0xc0001bc700, 0x3202040, 0xc000702c00, 0xc000fbc700)
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.36.0/server.go:878 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.36.0/server.go:876 +0x204

Error: The terraform-provider-google_v3.61.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
6

For me the issue followed after upgrading provider versions and terraform cli. Errors seemed to disappear after purging the plugin cache and performing terraform init -upgrade.

However providers/plugins were still failing…

Turning on DEBUG (export TF_LOG=DEBUG;) led me to discover that it was actually due to running on an Apple M1: terraform destroy command is not working and seems to be hanging on Mac OS Big Sur · Issue #27350 · hashicorp/terraform · GitHub

setting export GODEBUG=asyncpreemptoff=1; has now resolved the issue.

2 Likes
7

In #27350 this was diagnosed in Dec 2020. It’s now April 2022 and Terraform still has this problem. There’s no official documentation of the workaround and no attempt by Hashicorp to fix this.

Is Hashicorp Intel only? Should Mac users be looking for different solutions?

8

Hi guys - still no solution???

 Error: Request cancelled
│ 
│   with aws_s3_object.artifact,
│   on main.tf line 14, in resource "aws_s3_object" "artifact":
│   14: resource "aws_s3_object" "artifact" {
│ 
│ The plugin.(*GRPCProvider).ApplyResourceChange request was cancelled.
╵
ERROR: Job failed: command terminated with exit code 1
9

I tried this, deleting .terraform file and .lock file, then ran terraform init again and it was fine.

10

I was getting this with AWS OpenSearch Serverless and while this error is super misleading it was actually just the creds for the role I was assuming didn’t have permissions to do anything with OpenSearch. Once I fixed that it was fine.
Extremely frustrating figuring that out as the error doesn’t help you at all. Hopefully this helps someone else going down this rabbit hole.