TCP SSH timeout while running remote_exec on AWS instance

Terraform
1

Terraform Version

hashicorp/terraform:0.12.23

Terraform Configuration Files

resource "aws_instance" "jellyfin_server" {
  key_name = aws_key_pair.jellyfin_keys.key_name
  ami = data.aws_ami.amazon_linux_2.id
  instance_type = var.EC2_INSTANCE_TYPE
  iam_instance_profile = aws_iam_instance_profile.jellyfin_instance_profile.name
  associate_public_ip_address = true
  root_block_device {
    volume_size = var.EBS_ROOT_VOLUME_SIZE
  }
  ebs_block_device {
    device_name = "/dev/sdf"
    volume_size = var.EBS_MEDIA_VOLUME_SIZE
    volume_type = var.EBS_MEDIA_VOLUME_TYPE
  }

  vpc_security_group_ids = [aws_security_group.jellyfin_server_sg.id]
  subnet_id = aws_subnet.jellyfin_a.id
  connection {
    type     = "ssh"
    user     = "ec2-user"
    private_key = file(".ssh/jellyfin-key")
    host     = self.public_ip
  }

  provisioner "remote-exec" {
    inline = [
      "mkdir ~/jellyfin",
      "mkdir ~/jellyfin/scripts",
      "mkdir ~/jellyfin/media",
      "mkdir ~/jellyfin/cache",
      "mkdir ~/jellyfin/config",
      "sudo amazon-linux-extras enable nginx1",
      "sudo yum install -y docker nginx dos2unix",
      "sudo usermod -a -G docker ec2-user",
      "sudo systemctl enable docker",
      "sudo systemctl enable nginx"
    ]
  }

  provisioner "remote-exec" {
    inline = [
      "EBS_DEVICE_NAME=$(lsblk | grep ${var.EBS_MEDIA_VOLUME_SIZE}G | awk '{print $1}')",
      "sudo mkfs -t xfs /dev/$${EBS_DEVICE_NAME}",
      "EBS_DEVICE_UUID=$(sudo blkid | grep $${EBS_DEVICE_NAME} | awk -F'\"' '{print $2}')",
      "sudo echo -e \"UUID=$${EBS_DEVICE_UUID} /home/ec2-user/jellyfin/media  xfs  defaults,nofail  0  2\" | sudo tee -a /etc/fstab",
      "sudo mount -a",
      "sudo chown -R ec2-user /home/ec2-user/jellyfin/media"
    ]
  }

  // Creates nginx conf file for jellyfin server based on ALB or Domain name
  provisioner "file" {
    content = templatefile("templates/server.conf.tmpl", { server_name = local.server_name })
    destination = "/tmp/instant-jellyfin.conf"
  }

  // Lays down cron for syncing files form s3 bucket to local storage
  provisioner "file" {
    content = file("files/start-sync.sh") 
    destination = "/tmp/start-s3sync.sh"
  }

  // Creates cron job script with bucket variable
  provisioner "file" {
    content = templatefile("templates/s3sync.sh.tmpl", { BUCKET = aws_s3_bucket.jellyfin_media.id })
    destination = "/tmp/s3sync.sh"
  }

  // Lays down docker script
  provisioner "file" {
    content = file("files/start-jellyfin.sh")
    destination = "/tmp/start-jellyfin.sh"
  }

  // Starts Nginx with jellyfin conf
  provisioner "file" {
    content = file("files/start-nginx.sh")
    destination = "/tmp/start-nginx.sh"
  }

  // Starts jellyfin backups to S3
  provisioner "file" {
    content = file("files/start-backup.sh")
    destination = "/tmp/start-backup.sh"
  }


  // Creates backup script for jellyfin config
  provisioner "file" {
    content = templatefile("templates/backup.sh.tmpl", { BUCKET = aws_s3_bucket.jellyfin_backup.id })
    destination = "/tmp/backup.sh"
  }

  // Creates restore script for jellyfin config
  provisioner "file" {
    content = templatefile("templates/restore.sh.tmpl", { BUCKET = aws_s3_bucket.jellyfin_backup.id })
    destination = "/tmp/restore.sh"
  }


   provisioner "remote-exec" {
    inline = [
      "sudo mv /tmp/instant-jellyfin.conf /etc/nginx/conf.d/instant-jellyfin.conf",
      "mv /tmp/start-s3sync.sh ~/jellyfin/scripts/start-s3sync.sh",
      "mv /tmp/s3sync.sh ~/jellyfin/scripts/s3sync.sh",
      "mv /tmp/start-jellyfin.sh ~/jellyfin/scripts/start-jellyfin.sh",
      "mv /tmp/start-nginx.sh ~/jellyfin/scripts/start-nginx.sh",
      "mv /tmp/start-backup.sh ~/jellyfin/scripts/start-backup.sh",
      "mv /tmp/backup.sh ~/jellyfin/scripts/backup.sh",
      "mv /tmp/restore.sh ~/jellyfin/scripts/restore.sh",
      "sudo dos2unix /etc/nginx/conf.d/instant-jellyfin.conf",
      "sudo sed -i '/^[ ]*http[ ]*{.*/a     server_names_hash_bucket_size  128;' /etc/nginx/nginx.conf",
      "dos2unix ~/jellyfin/scripts/start-s3sync.sh",
      "dos2unix ~/jellyfin/scripts/s3sync.sh",
      "dos2unix ~/jellyfin/scripts/start-jellyfin.sh",
      "dos2unix ~/jellyfin/scripts/start-nginx.sh",
      "dos2unix ~/jellyfin/scripts/start-backup.sh",
      "dos2unix ~/jellyfin/scripts/backup.sh",
      "dos2unix ~/jellyfin/scripts/restore.sh",
      "/bin/bash ~/jellyfin/scripts/start-s3sync.sh",
      "/bin/bash ~/jellyfin/scripts/start-jellyfin.sh",
      "/bin/bash ~/jellyfin/scripts/start-nginx.sh",
      "/bin/bash ~/jellyfin/scripts/start-backup.sh"
    ]
  } 

  tags = {
    Name        = "${var.ENVIRONMENT}-jellyfin-server"
    Environment = var.ENVIRONMENT
  }
}

Debug Output

Crash Output

Error: timeout - last error: dial tcp 54.191.141.95:22: i/o timeout

Expected Behavior

Should create files on the newly created aws instance

Actual Behavior

terraform_1 | aws_instance.jellyfin_server (remote-exec): Connecting to remote host via SSH…
terraform_1 | aws_instance.jellyfin_server (remote-exec): Host: 54.191.141.95
terraform_1 | aws_instance.jellyfin_server (remote-exec): User: ec2-user
terraform_1 | aws_instance.jellyfin_server (remote-exec): Password: false
terraform_1 | aws_instance.jellyfin_server (remote-exec): Private key: true
terraform_1 | aws_instance.jellyfin_server (remote-exec): Certificate: false
terraform_1 | aws_instance.jellyfin_server (remote-exec): SSH Agent: false
terraform_1 | aws_instance.jellyfin_server (remote-exec): Checking Host Key: false
terraform_1 | aws_instance.jellyfin_server: Still creating… [5m20s elapsed]
terraform_1 | aws_instance.jellyfin_server: Still creating… [5m30s elapsed]
terraform_1 |
terraform_1 |
terraform_1 | Error: timeout - last error: dial tcp 54.191.141.95:22: i/o timeout
terraform_1 |
terraform_1 |
instant-jellyfin_terraform_1 exited with code 1

Steps to Reproduce

Additional Context

References

2

Hi @sd999111 ,

Just wanted to clarify that this is not a “crash”, but just an error that Terraform is reporting to you.

It means that Terraform was unable to connect via SSH to the host (in this case 54.192.141.95 at port 22).

This can happen for various reasons, for example:

  • The server is not listening to port 22
  • A firewall is blocking access to the port

Please check if the AWS security group is allowing access to the aws_instance from the IP where you are running Terraform.

Additionally, maybe it takes time for the AWS instance to start serving requests and you need to introduce some wait before you start your provisioning steps.

Also, keep in mind that “provisioners are a last resort” and consider using other options:

  • cloud-init via the user-data attribute of the aws_instance resource
  • using HashiCorp’s Packer to create an AWS AMI that has all of your required software / applications already installed

Hope this helps you with your task.

Regards,
Filip

3

Make sure you can putty (using your ppk cert)
and add this to your connection,
timeout = “4m”
You’ll know through putty that it takes a few mins for the instance to allow connections