Terraform SSH connections for provisioners timeout

Terraform
1

I have two terraform modules that creates a jumphost VM and an app VM respectively. The app module runs a script over the corresponding machine using SSH provisioner. There IS NO explicit connect block for the jumphost VM but there IS one for the app VM with bastion host settings. I always get the following error:

Error: timeout - last error: ssh: rejected: connect failed (Connection timed out)

I tried connecting to the instances manually, and the connection works fine. How do I troubleshoot this?

Terraform v0.12.3
+ provider.google v2.10.0
+ provider.template v2.1.2

Here’s the app module :

resource "google_compute_instance" "app" {
  name                      = var.instance_name
  machine_type              = var.machine_type
  zone                      = var.launch_zone

  boot_disk {
    ..
  }

  network_interface {
    subnetwork = var.subnet_name
    network_ip = google_compute_address.app-internal.address

    access_config {
      nat_ip = google_compute_address.app-external.address
    }
  }

  metadata = {
    ssh-keys = "${var.ssh_user}:${var.ssh_metadata}"
  }

  connection {
    user        = var.ssh_user
    host        = google_compute_address.app-internal.address
    // private_key = var.private_key
    private_key = file("/home/eternaltyro/.ssh/id_ecdsa")

    bastion_host = var.bastion_host
    bastion_user = var.bastion_user
    agent = false
  }

  provisioner "remote-exec" {
    script = var.script_path
  }

}

I use google cloud provider here. I invoke the module like so:

module "gateway" {
  source = "../modules/gateway"

  instance_name = "sandbox-gateway"
  subnet_name   = module.vpc.gateway_subnet
  internal_ip   = "10.0.99.2"

  ssh_user          = "eternaltyro"
  ssh_metadata      = "ecdsa-sha2-nistp256 AAAAE2VjLONGSTRINGi/qH32h0+HJwmfc= e@c"
}

module "app" {
  source = "../modules/app"

  instance_name = "sandbox" # var.client_name
  network_tags  = ["app", "protected"]
  subnet_name = module.vpc.primary_subnet
  internal_ip = "10.0.3.4"
 
  ssh_user     = "eternaltyro"
  ssh_metadata = "ecdsa-sha2-nistp256 AAAAE2VjLONGSTRINGi/qH32h0+HJwmfc= eternaltyro@cockroach"
  script_path  = "app-init.sh"
  bastion_user = "eternaltyro"
  bastion_host = module.gateway.external_ip_address
  private_key  = file("~/.ssh/id_ecdsa")
}

Any troubleshooting advice would be helpful.

2

I’m dumb. Google compute engine Network security tags don’t work between two different VPC networks:

From doc: 添加网络标记  |  VPC  |  Google Cloud

The network tags that you assign to an instance only apply to the VPC network where the instance’s primary network interface is located. This is true even for VPC Network Peering because peered networks remain distinct networks. Thus, the network tags are still only meaningful in the network that contains the instance’s primary network interface.

False negative, y’all!