Mlock, golang, docker, random and Ubuntu 20.04

Hi,

I wasn’t sure if this counted as a bug or not so I’m raising here rather than in Github.

I recently saw odd error messages when attempting terraform plan in a docker container on Ubuntu 20.04. Specifically from the random provider.

failed to retrieve schema from provider “random”: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: EOF

I’ve written up more details here https://blog.gripdev.xyz/2020/07/14/terraform-docker-ubuntu-20-04-go-1-14-and-memlock-down-the-rabbit-hole/

It looks like this relates to a workaround used in go 1.14 and it’s affect on the Ubuntu kernel

I’m not sure if there is a change that can be made, for example to the version of go used, which would mean this didn’t affect users.

One thing that did happen was I lost a lot of time thinking this was an issue with the GRPC link rather than the provider crashing. It would be great to get the crash highlighted in the error message more clearly as the cause.

1 Like

Thanks for this! The --ulimit memlock=-1 arg to docker run helped us tremendously.

This needs to be made more public! This saved me a LOONG time of debugging! --ulimit memlock=1 worked like a dream

Hi @lawrencegripper! Thanks for pointing this out and documenting the workaround.

From reviewing the Go issues you linked to, it seems like this problem has been mitigated for Go 1.15 and the problematic functionality will be removed altogether in Go 1.16.

Go 1.15’s release process is currently underway, so Terraform providers can’t adopt it yet due to a policy of only using stable Go releases. However, given that the random provider’s behavior is pretty straightforward and only uses a small part of the Go standard library I’d expect it wouldn’t be too difficult to review the 1.15 release notes once the release is final to see if there are changes to functionality this provider depends on, and thus unblock upgrading to 1.15 for the next release.

With that said, I’m hedging a bit here because I personally work primarily on Terraform Core and not on the official providers, and also it sounds like this issue is broad enough that it could potentially affect any provider built with Go 1.14, and so adopting Go 1.15 across all of them is likely to be a longer effort.

Terraform Core v0.13 is also due in a couple weeks and will be built with Go 1.14, so we’ll be on the lookout for symptoms that seem similar to this issue with the Terraform Core release.

Thanks again!