I am attempting to install/configure Waypoint in a k8s cluster entirely through Terraform. My goal with this project is to bring up a fully functional k8s cluster w/ supporting services through a single terraform apply. Here is the output (key redacted from the build command):
~/code/quilt/api rvm:(ruby-3.0.2) git:(waypoint)
➜ waypoint -vvv build -var "env=local"
2022-05-23T09:50:04.473-0400 [INFO] waypoint: waypoint version: full_string=v0.8.1 version=v0.8.1 prerelease="" metadata="" revision=""
2022-05-23T09:50:04.473-0400 [TRACE] waypoint: starting interrupt listener for context cancellation
2022-05-23T09:50:04.473-0400 [TRACE] waypoint: interrupt listener goroutine started
2022-05-23T09:50:04.473-0400 [DEBUG] waypoint: home configuration directory: path=/Users/ross/Library/Preferences/waypoint
2022-05-23T09:50:04.474-0400 [TRACE] waypoint: no API client provided, initializing connection if possible
2022-05-23T09:50:04.474-0400 [TRACE] waypoint.server: Local server may be created later - server credentials optional
2022-05-23T09:50:04.474-0400 [INFO] waypoint.server: attempting to source credentials and connect
2022-05-23T09:50:04.474-0400 [DEBUG] waypoint.serverclient: connection information: address=waypoint.demo.wequilt.io:9701 tls=true tls_skip_verify=true send_auth=true has_token=true
2022-05-23T09:50:08.751-0400 [DEBUG] waypoint.server: connection established with sourced credentials
2022-05-23T09:50:08.751-0400 [TRACE] waypoint: requesting version info from server
2022-05-23T09:50:09.952-0400 [INFO] waypoint: server version info: version=v0.8.2 api_min=1 api_current=1 entrypoint_min=1 entrypoint_current=1
2022-05-23T09:50:09.952-0400 [INFO] waypoint: negotiated api version: version=1
2022-05-23T09:50:09.952-0400 [DEBUG] waypoint: will operate on app: name=api
2022-05-23T09:50:09.952-0400 [DEBUG] waypoint.setupLocalJobSystem: determining if a local or remote runner should be used for this and future jobs
» Building api...2022-05-23T09:50:12.902-0400 [DEBUG] waypoint: Default runner profile exists - operation is possible remotely.
2022-05-23T09:50:12.902-0400 [DEBUG] waypoint.setupLocalJobSystem: Running this command: git -C /Users/ross/code/quilt/api rev-parse --show-toplevel
2022-05-23T09:50:12.916-0400 [WARN] waypoint.setupLocalJobSystem: failed to determine if local vcs is dirty: err="failed to diff repo at "/Users/ross/code/quilt/api" against remote with url "ssh://git@github.com/wequilt/api.git" ref "waypoint": Failed to get remote name for url ssh://git@github.com/wequilt/api.git: no remote with url matching "ssh://git@github.com/wequilt/api.git" found"
2022-05-23T09:50:12.916-0400 [DEBUG] waypoint.setupLocalJobSystem: result: isLocal=false
2022-05-23T09:50:12.916-0400 [DEBUG] waypoint: queueing job: operation=*gen.Job_Build
2022-05-23T09:50:14.006-0400 [DEBUG] waypoint: opening job stream: job_id=01G3RJXESXWHAZ17PR51W8HZ21
2022-05-23T09:50:14.722-0400 [WARN] waypoint: unknown stream event: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="&{job:{id:"01G3RJXESXWHAZ17PR51W8HZ21" depends_on:"01G3RJXESY5ECRZVHFKM1X0BHZ" application:{application:"api" project:"api"} workspace:{workspace:"default"} target_runner:{id:{id:"01G3RJXESXBZDTYRSMXBC0VF76"}} ondemand_runner:{id:"01g3jbvw9amcw8pht9b7mmgt82"} data_source:{git:{url:"ssh://git@github.com/wequilt/api.git" ref:"waypoint" ssh:{private_key_pem:"-----BEGIN OPENSSH PRIVATE KEY-----\n****\n-----END OPENSSH PRIVATE KEY-----\n"}}} variables:{name:"env" str:"local" cli:{}} build:{} state:QUEUED queue_time:{seconds:1653313813 nanos:310564222}}}"
2022-05-23T09:50:15.548-0400 [DEBUG] waypoint: Executing operation on an on-demand runner from profile with ID %q: job_id=01G3RJXESXWHAZ17PR51W8HZ21 EXTRA_VALUE_AT_END=01G3JBVW9AMCW8PHT9B7MMGT82
2022-05-23T09:50:16.220-0400 [WARN] waypoint: unknown stream event: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="&{job:{id:"01G3RJXESXWHAZ17PR51W8HZ21" depends_on:"01G3RJXESY5ECRZVHFKM1X0BHZ" application:{application:"api" project:"api"} workspace:{workspace:"default"} target_runner:{id:{id:"01G3RJXESXBZDTYRSMXBC0VF76"}} ondemand_runner:{id:"01g3jbvw9amcw8pht9b7mmgt82"} data_source:{git:{url:"ssh://git@github.com/wequilt/api.git" ref:"waypoint" ssh:{private_key_pem:"-----BEGIN OPENSSH PRIVATE KEY-----\n****\n-----END OPENSSH PRIVATE KEY-----\n"}}} variables:{name:"env" str:"local" cli:{}} build:{} state:WAITING assigned_runner:{id:"01G3RJXESXBZDTYRSMXBC0VF76"} queue_time:{seconds:1653313813 nanos:310564222} assign_time:{seconds:1653313814 nanos:604880226}}}"
2022-05-23T09:50:16.221-0400 [WARN] waypoint: unknown stream event: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="&{job:{id:"01G3RJXESXWHAZ17PR51W8HZ21" depends_on:"01G3RJXESY5ECRZVHFKM1X0BHZ" application:{application:"api" project:"api"} workspace:{workspace:"default"} target_runner:{id:{id:"01G3RJXESXBZDTYRSMXBC0VF76"}} ondemand_runner:{id:"01g3jbvw9amcw8pht9b7mmgt82"} data_source:{git:{url:"ssh://git@github.com/wequilt/api.git" ref:"waypoint" ssh:{private_key_pem:"-----BEGIN OPENSSH PRIVATE KEY-----\n****\n-----END OPENSSH PRIVATE KEY-----\n"}}} variables:{name:"env" str:"local" cli:{}} build:{} state:RUNNING assigned_runner:{id:"01G3RJXESXBZDTYRSMXBC0VF76"} queue_time:{seconds:1653313813 nanos:310564222} assign_time:{seconds:1653313814 nanos:604880226} ack_time:{seconds:1653313814 nanos:607217383}}}"
2022-05-23T09:50:16.222-0400 [TRACE] waypoint: job terminal output: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="line:{msg:"Cloning data from Git" style:"header"}"
2022-05-23T09:50:16.222-0400 [TRACE] waypoint: job terminal output: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="line:{msg:"URL: ssh://git@github.com/wequilt/api.git" style:"info"}"
2022-05-23T09:50:16.222-0400 [TRACE] waypoint: job terminal output: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="line:{msg:"Ref: waypoint" style:"info"}"
2022-05-23T09:50:16.222-0400 [TRACE] waypoint: job terminal output: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="line:{msg:"Auth: ssh" style:"info"}"
Performing operation on "kubernetes" with runner profile "01G3JBVW9AMCW8PHT9B7MMGT82"
» Cloning data from Git
URL: ssh://git@github.com/wequilt/api.git
Ref: waypoint
Auth: ssh2022-05-23T09:50:20.138-0400 [WARN] waypoint: unknown stream event: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="&{data_source_ref:{git:{commit:"12c62577e10162ad9b991c448f7439af2f6ca6f9" timestamp:{seconds:1652883934}}}}"
2022-05-23T09:50:20.138-0400 [WARN] waypoint: unknown stream event: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="&{job:{id:"01G3RJXESXWHAZ17PR51W8HZ21" depends_on:"01G3RJXESY5ECRZVHFKM1X0BHZ" application:{application:"api" project:"api"} workspace:{workspace:"default"} target_runner:{id:{id:"01G3RJXESXBZDTYRSMXBC0VF76"}} ondemand_runner:{id:"01g3jbvw9amcw8pht9b7mmgt82"} data_source:{git:{url:"ssh://git@github.com/wequilt/api.git" ref:"waypoint" ssh:{private_key_pem:"-----BEGIN OPENSSH PRIVATE KEY-----\n****\n-----END OPENSSH PRIVATE KEY-----\n"}}} variables:{name:"env" str:"local" cli:{}} build:{} state:RUNNING assigned_runner:{id:"01G3RJXESXBZDTYRSMXBC0VF76"} queue_time:{seconds:1653313813 nanos:310564222} assign_time:{seconds:1653313814 nanos:604880226} ack_time:{seconds:1653313814 nanos:607217383} data_source_ref:{git:{commit:"12c62577e10162ad9b991c448f7439af2f6ca6f9" timestamp:{seconds:1652883934}}}}}"
2022-05-23T09:50:20.138-0400 [WARN] waypoint: unknown stream event: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="&{job:{id:"01G3RJXESXWHAZ17PR51W8HZ21" depends_on:"01G3RJXESY5ECRZVHFKM1X0BHZ" application:{application:"api" project:"api"} workspace:{workspace:"default"} target_runner:{id:{id:"01G3RJXESXBZDTYRSMXBC0VF76"}} ondemand_runner:{id:"01g3jbvw9amcw8pht9b7mmgt82"} data_source:{git:{url:"ssh://git@github.com/wequilt/api.git" ref:"waypoint" ssh:{private_key_pem:"-----BEGIN OPENSSH PRIVATE KEY-----\n****\n-----END OPENSSH PRIVATE KEY-----\n"}}} variables:{name:"env" str:"local" cli:{}} build:{} state:RUNNING assigned_runner:{id:"01G3RJXESXBZDTYRSMXBC0VF76"} queue_time:{seconds:1653313813 nanos:310564222} assign_time:{seconds:1653313814 nanos:604880226} ack_time:{seconds:1653313814 nanos:607217383} data_source_ref:{git:{commit:"12c62577e10162ad9b991c448f7439af2f6ca6f9" timestamp:{seconds:1652883934}}} config:{source:FILE}}}"
2022-05-23T09:52:14.885-0400 [WARN] waypoint: unknown stream event: job_id=01G3RJXESXWHAZ17PR51W8HZ21 event="&{job:{id:"01G3RJXESXWHAZ17PR51W8HZ21" depends_on:"01G3RJXESY5ECRZVHFKM1X0BHZ" application:{application:"api" project:"api"} workspace:{workspace:"default"} target_runner:{id:{id:"01G3RJXESXBZDTYRSMXBC0VF76"}} ondemand_runner:{id:"01g3jbvw9amcw8pht9b7mmgt82"} data_source:{git:{url:"ssh://git@github.com/wequilt/api.git" ref:"waypoint" ssh:{private_key_pem:"-----BEGIN OPENSSH PRIVATE KEY-----\n****\n-----END OPENSSH PRIVATE KEY-----\n"}}} variables:{name:"env" str:"local" cli:{}} build:{} state:ERROR assigned_runner:{id:"01G3RJXESXBZDTYRSMXBC0VF76"} queue_time:{seconds:1653313813 nanos:310564222} assign_time:{seconds:1653313814 nanos:604880226} ack_time:{seconds:1653313814 nanos:607217383} data_source_ref:{git:{commit:"12c62577e10162ad9b991c448f7439af2f6ca6f9" timestamp:{seconds:1652883934}}} config:{source:FILE} error:{code:1 message:"canceled"} cancel_time:{seconds:1653313934 nanos:609639561}}}"
2022-05-23T09:52:14.886-0400 [WARN] waypoint: job failed: job_id=01G3RJXESXWHAZ17PR51W8HZ21 code=Canceled message=canceled
! operation canceled
2022-05-23T09:52:14.886-0400 [TRACE] waypoint: stopping signal listeners and cancelling the context
I created this configuration by copying resources from the official helm chart into native terraform. Using the helm_release Terraform resource wasn’t quite flexible enough for me to integrate Waypoint with existing Traefik, cert-manager, etc. infrastructure. By an large it seems to be working but seems like it maybe doesn’t create a task record before starting an on-demand runner for some reason? Maybe? It’s hard for me to tell what’s going on exactly. Happy to prove any other logs or config that would be helpful.