Terraform aws plugin timeout issue with newer versions

Terraform
1

I am currently trying to upgrade from Terraform 0.11.x to 0.12.x but I am having issues with the current versions of the aws provider plugin.

I have gradually downgraded the versions until it works again, when I use terraform 0.11.14 and the provider version 1.38.0, the plan operation works, when I update the provider to the current 1.x version 1.60.0 I get a timeout

% terraform plan

Error: Error asking for user input: 1 error occurred:
        * provider.aws: timeout while waiting for plugin to start

is there any way I can debug that further?

I am using terraform in Windows 10 x64 1803

Terraform version 0.11.14 AWS version 1.38 (works)
Terraform version 0.11.14 AWS version 1.60 (timeout)
Terraform version 0.12.3 AWS version 2.17 (timeout)

2

Hi Alexander! I wanted to see if you had tried re-running terraform init prior to `terraform plan. Also, did you download the providers manually or did you let Terraform download them automatically? Thanks!

3

I reran terraform init before trying plan again. I downloaded the providers via the version parameter automatically.
(to be exact, I delete the provider file and reran init)

4

Ok, great. One thing that may help is to set up DEBUG logging and see if that gives any additional details on the output (more info on DEBUG logging is at https://www.terraform.io/docs/internals/debugging.html).

5

That doesn’t give much info, the lines around the ERROR log lines are like this:

2019/07/02 17:55:41 [DEBUG] Starting graph walk: walkInput
2019-07-02T17:55:41.409+0200 [DEBUG] plugin: waiting for RPC address: path=C:\Projects\terraform\ecs-terraform\.terraform\plugins\windows_amd64\terraform-provider-aws_v1.60.0_x4.exe
2019/07/02 17:56:41 [ERROR] root: eval: *terraform.EvalInitProvider, err: timeout while waiting for plugin to start
2019/07/02 17:56:41 [ERROR] root: eval: *terraform.EvalSequence, err: timeout while waiting for plugin to start
2019/07/02 17:56:41 [TRACE] [walkInput] Exiting eval tree: provider.aws
6

Thanks for the details, this is a bit puzzling. One other thing you can try is removing the .terraform directory and re-running terraform init. Let me know if that helps.

7

After moving the .terraform dir out of the way, the same error appears

8

I have tried to use the example two-tier project from the git repo and that gives the same error. I am beginning to suspect something like an issue with a security software like a virus scanner on the machine.

9

Similar issues. Even updated to 2.43 from 2.42 tonight. Same issues.

10

Hi @alexlehm,

While the difference in behavior between provider versions is a new variant, the most common explanation we’ve seen for timeouts when launching providers on Windows systems is interference from firewall or similar security software that can prevent the main Terraform process from connecting to the child process where the provider plugin is running. Could it be the case that the previous version had already been added to a firewall approval list, but the new version is not?

The log is interesting to me because it suggests that the timeout was while “waiting for RPC address”, and not while connecting to the RPC address as has been the common form of this problem before.

It could be informative to try running the plugin executables directly, tricking them into thinking they are running as child processes of Terraform, to see what they do:

set TF_PLUGIN_MAGIC_COOKIE=d602bf8f470bc67ca7faa0386276bbdd4330efaf76d1a219cb4d6991ca9872b2
C:\Projects\terraform\ecs-terraform\.terraform\plugins\windows_amd64\terraform-provider-aws_v1.60.0_x4.exe

That TF_PLUGIN_MAGIC_COOKIE environment variable is what the plugin programs use to detect whether they are running under Terraform’s supervision, so by setting it in the shell we can trick them into talking to the command prompt window as if it were Terraform.

If it’s working as expected, we should see this print out a line of text that specifies which provider protocol the plugin is expecting to talk, and on what address it has started up its RPC server. We might see the timeout message you described if instead the executable were to produce an unexpected error message or to exit before completing the handshake.

If you’d be willing, it would be interesting to see if this behaves differently for your working version 1.38 compared to the non-working versions 1.60 and 2.17. That might give a clue as to what to investigate next.

@mpechner it looks like you have a different problem where the AWS provider is starting up correctly but then it is failing to find credentials to interact with the AWS API. I’d suggest starting a new topic for that if you are still having problems, possibly in the AWS provider category. (Though from the GitHub issue it sounds like you’ve found a solution.)