Terraform returns "i/o timeout" error when speaking to AWS services

When I execute any action that requires Terraform to speak to AWS services, I receive this kind of error:

Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::12345678910:role/terraform-role) cannot be assumed.
│
│ There are a number of possible causes of this - the most common are:
│   * The credentials used in order to assume the role are invalid
│   * The credentials do not have appropriate permission to assume the role
│   * The role ARN is not valid
│
│ Error: operation error STS: AssumeRole, https response error StatusCode: 0, RequestID: , request send failed, Post "https://sts.us-east-1.amazonaws.com/": dial tcp: lookup sts.us-east-1.amazonaws.com on 1.1.1.1:53: read udp 192.168.1.246:0->1.1.1.1:53: i/o timeout
│
│
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 13, in provider "aws":
│   13: provider "aws" {

What I’ve tested so far:

  1. Another Macbook connected to the same network and have the same software versions installed is working well.
  2. Switching to another network does not help.
  3. Tried with the Internet provider’s DNS server (192.168.1.1), with Google’s (8.8.8.8), with CloudFlare’s (1.1.1.1) - no difference.
  4. The command dig @1.1.1.1 sts.us-east-2.amazonaws.com works just fine.
  5. AWS CLI can talk to STS without any issues.
  6. If I run a Terraform within a container, and mount my AWS credentials as a volume, it works just fine. So the problem is only when I run it via the local terminal.
  7. Tried different versions of Terraform, namely 1.2.5, 1.0.0, and 1.0.3 - no difference.

I’m running out of ideas, any help please?

Versions:

macOS Monterey, version 12.5
Terraform v1.2.5
aws-cli/2.0.30 Python/3.7.4 Darwin/21.6.0 botocore/2.0.0dev34
Go runtime version: go1.18.1

Hi @krsh-off were you able to resolve the issue? I am facing the same issue you have described. Any pointers welcome.

Hi @anilkpc1, yes, I solved the issue.

The root cause was not the Terraform or Macbook itself, it was a Cisco AnyConnect client installed on my corporate Macbook. Whenever it was running (not even connected to VPN, just running), it was preventing Terraform from functioning normally, probably by tweaking the network configuration on the Macbook itself. I was able to restore normal operations by killing the Cisco AnyConnect process and uninstalling it. Later on, I just submitted a ticket to my company and they uninstalled it permanently.

So my best guess is either Cisco AnyConnect or any other VPN software on your laptop.