How to start a powershell script in a new process on an AWS EC2 windows instance

Terraform Providers AWS
1

I can’t figure out how to launch a powershell script in the background on an AWS EC2 windows instance using terraform “remote-exec”.

I am creating an AWS EC2 windows instance. Then, I am copying some powershell scripts to the instance (using the file provisioner) and I would like to then run these powershell scripts as new background processes (using the remote-exec provisioner). The scripts get copied over fine. The remote-exec provisioner seems to execute the command to start the scripts in the background. However, the scripts appear to never actually be running.

If I execute the exact same scripts using “remote-exec” in the foreground (not as background processes) then the scripts run on the remote machine. It is only when they are executed as background processes that they appear not to run. However, the scripts are long-running scripts (potentially running for hours) and terraform will pause until the script finishes running before continuing. This is unacceptable. I would like to build multiple machines in parallel (hence the need to launch the scripts in the background so that terraform can continue).

Here are two example powershell scripts that I want to run on a remote EC2 instance. These scripts simply loop and print a message to a log file every 5 seconds. They never exit.

# this is looper1.ps1
$looperLogFile = "C:\Users\Administrator\Looper1.log"
$sleepTime = 5
while ($true) {
    Add-Content -Path $looperLogFile -Value "Looper1. Waiting $sleepTime."
    Start-Sleep -s $sleepTime
}

# this is looper2.ps2
$looperLogFile = "C:\Users\Administrator\Looper2.log"
$sleepTime = 5
while ($true) {
    Add-Content -Path $looperLogFile -Value "Looper2. Waiting $sleepTime."
    Start-Sleep -s $sleepTime
}

and here is the terraform code that instantiates the EC2 instance, copies over the file and tries to execute them:

resource "aws_instance" "training_node" {
  instance_type          = "t3.small"
  ami = "ami-abcdefgxxxxx"
  key_name               = "trainingNode-keyPair"
  vpc_security_group_ids = [aws_security_group.training_sg.id]
  subnet_id              = aws_subnet.training_public_subnet.id
  source_dest_check      = false

  root_block_device {
    volume_size = 30
  }

  provisioner "file" {
    source = "./looper1.ps1"
    destination = "C:/Users/Administrator/looper1.ps1"
  }

  provisioner "file" {
    source = "./looper2.ps1"
    destination = "C:/Users/Administrator/looper2.ps1"
  }

  # Unblock all files so that you can execute them
  provisioner "remote-exec" {
    inline = [
      "powershell \"Get-Childitem C:\\Users\\Administrator -Recurse | Unblock-File\""
    ]
  }

  provisioner "remote-exec" {
    inline = [
      "start /b \"looper1\" \"powershell.exe\" \"C:\\Users\\Administrator\\looper1.ps1\"",
    ]
  }

  provisioner "remote-exec" {
    inline = [
      "powershell.exe Start-Process -FilePath \"powershell\" -ArgumentList C:\\Users\\Administrator\\looper2.ps1",
    ]
  }

  connection {
    type = "ssh"
    port = "22"
    target_platform = "windows"
    user = "Administrator"
    host = self.public_ip
    timeout = "5m"
    private_key = file("trainingNode-keyPair.pem")
  }

  tags = {
    Name = "training-node"
  }
}

This is what I see as the output from the “terraform apply” command (I did not turn the TRACE debugging on - it didn’t seem to show any more information than the regular output. However, if need be then I can turn on the TRACE debugging and attach it as a file here).

aws_instance.training_node: Still creating... [10s elapsed]
aws_instance.training_node: Provisioning with 'file'...
aws_instance.training_node: Still creating... [20s elapsed]
...
aws_instance.training_node: Still creating... [3m0s elapsed]
aws_instance.training_node: Still creating... [3m10s elapsed]
aws_instance.training_node: Provisioning with 'file'...
aws_instance.training_node: Provisioning with 'remote-exec'...
aws_instance.training_node (remote-exec): Connecting to remote host via SSH...
aws_instance.training_node (remote-exec):   Host: 35.182.145.242
aws_instance.training_node (remote-exec):   User: Administrator
aws_instance.training_node (remote-exec):   Password: false
aws_instance.training_node (remote-exec):   Private key: true
aws_instance.training_node (remote-exec):   Certificate: false
aws_instance.training_node (remote-exec):   SSH Agent: false
aws_instance.training_node (remote-exec):   Checking Host Key: false
aws_instance.training_node (remote-exec):   Target Platform: windows
aws_instance.training_node (remote-exec): Connected!

aws_instance.training_node (remote-exec): administrator@EC2AMAZ-ENNP0JE C:\Users\administrator>powershell "Get-Childitem 
C:\Users\Administrator -Recurse | Unblock-File"
aws_instance.training_node: Still creating... [3m20s elapsed]
aws_instance.training_node: Provisioning with 'remote-exec'...
aws_instance.training_node (remote-exec): Connecting to remote host via SSH...
aws_instance.training_node (remote-exec):   Host: 35.182.145.242
aws_instance.training_node (remote-exec):   User: Administrator
aws_instance.training_node (remote-exec):   Password: false
aws_instance.training_node (remote-exec):   Private key: true
aws_instance.training_node (remote-exec):   Certificate: false
aws_instance.training_node (remote-exec):   SSH Agent: false
aws_instance.training_node (remote-exec):   Checking Host Key: false
aws_instance.training_node (remote-exec):   Target Platform: windows
aws_instance.training_node (remote-exec): Connected!

aws_instance.training_node (remote-exec): administrator@EC2AMAZ-ENNP0JE C:\Users\administrator>start /b "looper1" "powershell.exe" "C:\Users\Administrator\looper1.ps1"
aws_instance.training_node: Provisioning with 'remote-exec'...
aws_instance.training_node (remote-exec): Connecting to remote host via SSH...
aws_instance.training_node (remote-exec):   Host: 35.182.145.242
aws_instance.training_node (remote-exec):   User: Administrator
aws_instance.training_node (remote-exec):   Password: false
aws_instance.training_node (remote-exec):   Private key: true
aws_instance.training_node (remote-exec):   Certificate: false
aws_instance.training_node (remote-exec):   SSH Agent: false
aws_instance.training_node (remote-exec):   Checking Host Key: false
aws_instance.training_node (remote-exec):   Target Platform: windows
aws_instance.training_node (remote-exec): Connected!

aws_instance.training_node (remote-exec): administrator@EC2AMAZ-ENNP0JE C:\Users\administrator>powershell.exe Start-Process -FilePath "powershell" -ArgumentList C:\Users\Administrator\looper2.ps1
aws_instance.training_node: Creation complete after 3m23s [id=i-0696603d3f0b1454e]

On the EC2 windows instance:

The files c:\Users\Administrator\looper1.ps1 and c:\Users\Administrator\looper2.ps1 are created.

However the log files c:\Users\Administrator\Looper1.log and c:\Users\Administrator\Looper2.log never appear (which means that the looper1.ps1 and looper2.ps1 scripts never ran).

From the terraform output it looks like terraform is using “remote-exec” to execute the commands:

start /b "looper1" "powershell.exe" "C:\Users\Administrator\looper1.ps1"
powershell.exe Start-Process -FilePath "powershell" -ArgumentList C:\Users\Administrator\looper2.ps1

I have verified that the shell used by the EC2 windows instance for ssh is “cmd”.

If I log in to the EC2 windows instance and open a “cmd” window and manually execute the command

start /b "looper1" "powershell.exe" "C:\Users\Administrator\looper1.ps1"

Then the looper1.ps1 scripts runs in the background and the Looper1.log file appears.

If, in the cmd window I manually execute the command:

powershell.exe Start-Process -FilePath "powershell" -ArgumentList C:\Users\Administrator\looper2.ps1

then the looper2.ps1 script runs in a new powershell window (in the background) and the Looper2.log file appears.

So, there seems to be some kind of difference between how terraform’s “remote-exec” is executing the commands and how they are executed manually by myself from a “cmd” window on the EC2 windows instance.

Further, if instead of copying the powershell script files over to the windows machine using terraform and executing them via terraform I create an AMI with the script files already resident in C:\Users\Administrator and I configure AWS EC2Launch on the windows machine to run the powershell scripts in the background at startup (by adding them to the C:\ProgramData\Amazon\Ec2Launch\config\agent-config.yml file like so:

version: "1.0"
config:
  - stage: boot
    tasks:
      - task: extendRootPartition
  - stage: preReady
    tasks:
      - task: activateWindows
        inputs:
          activation:
            type: amazon
      - task: setDnsSuffix
        inputs:
          suffixes:
            - $REGION.ec2-utilities.amazonaws.com
      - task: setAdminAccount
        inputs:
          name: Administrator
          password:
            type: doNothing
            data: ""
      - task: setWallpaper
        inputs:
          attributes:
            - hostName
            - instanceId
            - privateIpAddress
            - publicIpAddress
            - instanceSize
            - availabilityZone
            - architecture
            - memory
            - network
          path: C:\ProgramData\Amazon\EC2Launch\wallpaper\Ec2Wallpaper.jpg
      - task: enableOpenSsh
  - stage: postReady
    tasks:
      - task: startSsm
      - task: executeScript
        inputs:
          - content: '& C:\Users\Administrator\looper1.ps1'
            detach: true
            frequency: once
            runAs: admin
            type: powershell
          - content: '& C:\Users\Administrator\looper2.ps1'
            detach: true
            frequency: once
            runAs: admin
            type: powershell

then both looper1.ps1 and looper2.ps1 scripts successfully run in the background at machine startup (I see both Looper1.log and Looper2.log files are created).

So,

  • I can run the powershell scripts in the background manually from a cmd shell
  • and also via EC2Launch
  • and also in the foreground using “remote-exec”

However, the same commands do not run when executed to run in the background from terraform via “remote-exec”.

I can’t figure out what the differences are and I can’t figure out how to launch a powershell script in the background on an AWS EC2 windows instance using terraform “remote-exec”.

If anyone has any ideas or suggestions then they would be very much appreciated.