Terraform cloud runtime out of memory

One of my workspaces grew in terms of resources to handling 4 aws regions, 4 EKS clusters and accompanying resources

Now when i try to add another module, it starts failing with random plugin failures such as follows

e[0me[0m
e[1me[31mError: e[0me[0me[1mUnrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.e[0m

e[0me[0me[0m
e[31m
e[1me[31mError: e[0me[0me[1mUnrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.e[0m

and when looking in the log (trace level)
iā€™m seeing indicators that the terraform cloud plan runtime (remote mode) is having resources issues such as follows

2020/05/12 16:50:08 [ERROR] module.eu-west-1-eks: eval: *terraform.EvalInitProvider, err: Unrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.
2020/05/12 16:50:08 [ERROR] module.eu-west-1-eks: eval: *terraform.EvalSequence, err: Unrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.
2020/05/12 16:50:08 [TRACE] [walkPlan] Exiting eval tree: module.eu-west-1-eks.provider.aws
2020/05/12 16:50:08 [TRACE] vertex "module.eu-west-1-eks.provider.aws": visit complete
2020/05/12 16:50:08 [TRACE] dag/walk: upstream of "module.eu-west-1-eks.aws_iam_policy.alb_ingress_controller" errored, so skipping
2020-05-12T16:50:08.754Z [DEBUG] plugin.terraform-provider-aws_v2.61.0_x4: fatal error: runtime: out of memory
2020-05-12T16:50:08.754Z [DEBUG] plugin.terraform-provider-aws_v2.61.0_x4: 
2020-05-12T16:50:08.754Z [DEBUG] plugin.terraform-provider-aws_v2.61.0_x4: runtime stack:
2020-05-12T16:50:08.754Z [DEBUG] plugin.terraform-provider-aws_v2.61.0_x4: runtime.throw(0x5987320, 0x16)
2020-05-12T16:50:08.754Z [DEBUG] plugin.terraform-provider-aws_v2.61.0_x4: 	/opt/goenv/versions/1.13.7/src/runtime/panic.go:774 +0x72 fp=0x7ffd6295a4f8 sp=0x7ffd6295a4c8 pc=0x42dc62

please advise

Hi,

There is a new Terraform - v0.12.29 that addresses a problem with memory usage when the Terraform code has many resources defined.

The same fix is available in the v0.13 branch, namely v0.13.0-rc1.

As time permits, could you try if this fixes the issue ?

Regards,
Filip Dimitrov

1 Like