Unable to connect to target by ssh - Error dialing the worker: failed to WebSocket dial: failed to send handshake request

karthick-core · October 25, 2020, 2:46pm

I’m running boundary server on a single virtualbox vm in nat port forwarding mode.
The ubuntu vm has both controller and worker configured and running.

Also have these port forwarding rules set on host machine 9200->vm:9200, 9202->vm:9202

I can authenticate with the controller successfuly. However can’t connect to targets.

boundary targets read -id ttcp_yj50ER3Uq3 -token at_Zh5SfzpyYR_s1bWQSM96NgAY9UAy1W235eL3fiaui4QNBkhEkSnYy7ReY24ME2bYK3FtZGZWfX3C7NGPbPABNcgGusvRjQ1EFwc2AS7v1rq6Xxjv1fYnmwDbrLmV7UHdkP5y9ZUxFnN6BAtwRnTH82g2Q

Target information:
  Created Time:               Sun, 25 Oct 2020 18:18:47 IST
  Description:                Provides an initial target in Boundary
  ID:                         ttcp_yj50ER3Uq3
  Name:                       Generated target
  Session Connection Limit:   1
  Session Max Seconds:        28800
  Type:                       tcp
  Updated Time:               Sun, 25 Oct 2020 20:01:40 IST
  Version:                    3

  Scope:
    ID:                       p_kdHp9amf5d
    Name:                     Generated project scope
    Parent Scope ID:          o_xEbV3JxnWG
    Type:                     project

  Host Sets:
    Host Catalog ID:          hcst_YkiuiAkEq4
    ID:                       hsst_yYUwzxseOB

  Attributes:
    Default Port:             22

My controller hcl:

disable_mlock = true

# Controller configuration block
controller {
  # This name attr must be unique across all controller instances if running in HA mode
  name = "demo-controller-1"
  description = "A controller for a demo!"

  # Database URL for postgres. This can be a direct "postgres://"
  # URL, or it can be "file://" to read the contents of a file to
  # supply the url, or "env://" to name an environment variable
  # that contains the URL.
  database {
      url = "postgresql://postgres:password123@localhost:5432/boundary?sslmode=disable"
  }
}

# API listener configuration block
listener "tcp" {
  # Should be the address of the NIC that the controller server will be reached on
  address = "10.0.2.5"
  # The purpose of this listener block
  purpose = "api"

  tls_disable = true

  # Uncomment to enable CORS for the Admin UI. Be sure to set the allowed origin(s)
  # to appropriate values.
  #cors_enabled = true
  #cors_allowed_origins = ["yourcorp.yourdomain.com"]
}

# Data-plane listener configuration block (used for worker coordination)
listener "tcp" {
  # Should be the IP of the NIC that the worker will connect on
  address = "10.0.2.5"
  # The purpose of this listener
  purpose = "cluster"

  tls_disable = true
}

# Root KMS configuration block: this is the root key for Boundary
# Use a production KMS such as AWS KMS in production installs
kms "aead" {
  purpose = "root"
  aead_type = "aes-gcm"
  key = "sP1fnF5Xz85RrXyELHFeZg9Ad2qt4Z4bgNHVGtD6ung="
  key_id = "global_root"
}

# Worker authorization KMS
# Use a production KMS such as AWS KMS for production installs
# This key is the same key used in the worker configuration
kms "aead" {
  purpose = "worker-auth"
  aead_type = "aes-gcm"
  key = "8fZBjCUfN0TzjEGLQldGY4+iE9AkOvCfjh7+p0GtRBQ="
  key_id = "global_worker-auth"
}

# Recovery KMS block: configures the recovery key for Boundary
# Use a production KMS such as AWS KMS for production installs
kms "aead" {
  purpose = "recovery"
  aead_type = "aes-gcm"
  key = "8fZBjCUfN0TzjEGLQldGY4+iE9AkOvCfjh7+p0GtRBQ="
  key_id = "global_recovery"
}

My worker hcl:

listener "tcp" {
    purpose = "proxy"
    tls_disable = true
}

worker {
  # Name attr must be unique across workers
  name = "demo-worker-1"
  description = "A default worker created demonstration"

  # Workers must be able to reach controllers on :9202
  controllers = [
    "10.0.2.5"
  ]

#  public_addr = "127.0.0.1"
}

# must be same key as used on controller config
kms "aead" {
    purpose = "worker-auth"
    aead_type = "aes-gcm"
    key = "8fZBjCUfN0TzjEGLQldGY4+iE9AkOvCfjh7+p0GtRBQ="
    key_id = "global_worker-auth"
}

Systemd status:

kk@kk-ubvm:~/Desktop$ sudo systemctl status boundary-controller.service 
● boundary-controller.service - boundary controller
     Loaded: loaded (/etc/systemd/system/boundary-controller.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2020-10-25 19:49:15 IST; 29min ago
   Main PID: 4094 (boundary)
      Tasks: 10 (limit: 4652)
     Memory: 17.6M
     CGroup: /system.slice/boundary-controller.service
             ├─4094 /usr/local/bin/boundary server -config /etc/boundary-controller.hcl
             └─4102 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

Oct 25 19:49:15 kk-ubvm boundary[4094]:                Listener 2: tcp (addr: "10.0.2.5:9201", max_request_duration: "1m30s", purpose: "cluster")
Oct 25 19:49:15 kk-ubvm boundary[4094]:                 Log Level: info
Oct 25 19:49:15 kk-ubvm boundary[4094]:                     Mlock: supported: true, enabled: false
Oct 25 19:49:15 kk-ubvm boundary[4094]:                   Version: Boundary v0.1.1
Oct 25 19:49:15 kk-ubvm boundary[4094]:               Version Sha: eccd68d73c3edf14863ecfd31f9023063b809d5a
Oct 25 19:49:15 kk-ubvm boundary[4094]: ==> Boundary server started! Log data will stream in below:
Oct 25 19:49:15 kk-ubvm boundary[4094]: 2020-10-25T19:49:15.977+0530 [INFO]  controller: cluster address: addr=10.0.2.5:9201
Oct 25 19:50:01 kk-ubvm boundary[4094]: 2020-10-25T19:50:01.731+0530 [INFO]  controller: worker successfully authed: name=demo-worker-1
Oct 25 19:50:20 kk-ubvm boundary[4094]: 2020-10-25T19:50:20.691+0530 [INFO]  controller: worker successfully authed: name=demo-worker-1
Oct 25 19:52:05 kk-ubvm boundary[4094]: 2020-10-25T19:52:05.750+0530 [INFO]  controller: worker successfully authed: name=demo-worker-1
kk@kk-ubvm:~/Desktop$ sudo systemctl status boundary-worker.service 
● boundary-worker.service - boundary worker
     Loaded: loaded (/etc/systemd/system/boundary-worker.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2020-10-25 19:52:05 IST; 26min ago
   Main PID: 4300 (boundary)
      Tasks: 9 (limit: 4652)
     Memory: 107.6M
     CGroup: /system.slice/boundary-worker.service
             ├─4300 /usr/local/bin/boundary server -config /etc/boundary-worker.hcl
             └─4329 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session

Oct 25 19:52:05 kk-ubvm boundary[4300]:   [Worker-Auth] AEAD Type: aes-gcm
Oct 25 19:52:05 kk-ubvm boundary[4300]:                       Cgo: disabled
Oct 25 19:52:05 kk-ubvm boundary[4300]:                Listener 1: tcp (addr: "127.0.0.1:9202", max_request_duration: "1m30s", purpose: "proxy")
Oct 25 19:52:05 kk-ubvm boundary[4300]:                 Log Level: info
Oct 25 19:52:05 kk-ubvm boundary[4300]:                     Mlock: supported: true, enabled: true
Oct 25 19:52:05 kk-ubvm boundary[4300]:               Public Addr: 127.0.0.1:9202
Oct 25 19:52:05 kk-ubvm boundary[4300]:                   Version: Boundary v0.1.1
Oct 25 19:52:05 kk-ubvm boundary[4300]:               Version Sha: eccd68d73c3edf14863ecfd31f9023063b809d5a
Oct 25 19:52:05 kk-ubvm boundary[4300]: ==> Boundary server started! Log data will stream in below:
Oct 25 19:52:05 kk-ubvm boundary[4300]: 2020-10-25T19:52:05.748+0530 [INFO]  worker: connected to controller: address=10.0.2.5:9201

The error message while trying to connect to target

boundary connect ssh -target-id ttcp_yj50ER3Uq3 -token at_Zh5SfzpyYR_s1bWQSM96NgAY9UAy1W235eL3fiaui4QNBkhEkSnYy7ReY24ME2bYK3FtZGZWfX3C7NGPbPABNcgGusvRjQ1EFwc2AS7v1rq6Xxjv1fYnmwDbrLmV7UHdkP5y9ZUxFnN6BAtwRnTH82g2Q
Error dialing the worker: failed to WebSocket dial: failed to send handshake request: Get "https://127.0.0.1:9202/v1/proxy": read tcp 127.0.0.1:51644->127.0.0.1:9202: read: connection reset by peer
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 40185

malnick · October 25, 2020, 3:05pm

Thanks for trying out Boundary @karthick-core!

By default, the worker will advertise it’s addr as the bind address, in this case, it’s binding to localhost. Use the public_addr setting in your worker configuration to advertise the VM address that is bridged to your host machine. This will make it so the proxy address goes from localhost:9202 to <pub_addr>:9202 when the Boundary CLI connects to the worker.

Also thanks for sending such detailed info here!

karthick-core · October 25, 2020, 4:00pm

Thanks @malnick.
I updated the worker config as suggested after adding a bridge interface(host only net) to the VM.
Now my worker config looks like this

kk@kk-ubvm:~$ cat /etc/boundary-worker.hcl
listener "tcp" {
    purpose = "proxy"
    tls_disable = true
}

worker {
  # Name attr must be unique across workers
  name = "demo-worker-1"
  description = "A default worker created demonstration"

  # Workers must be able to reach controllers on :9202
  controllers = [
    "10.0.2.5"
  ]

  public_addr = "192.168.99.102"
}

# must be same key as used on controller config
kms "aead" {
    purpose = "worker-auth"
    aead_type = "aes-gcm"
    key = "8fZBjCUfN0TzjEGLQldGY4+iE9AkOvCfjh7+p0GtRBQ="
    key_id = "global_worker-auth"
}
kk@kk-ubvm:~$

However, the host still have problem connecting to the worker.

 ~ ▓▒░ boundary connect ssh -target-id ttcp_yj50ER3Uq3 -token at_Zh5SfzpyYR_s1bWQSM96NgAY9UAy1W235eL3fiaui4QNBkhEkSnYy7ReY24ME2bYK3FtZGZWfX3C7NGPbPABNcgGusvRjQ1EFwc2AS7v1rq6Xxjv1fYnmwDbrLmV7UHdkP5y9ZUxFnN6BAtwRnTH82g2Q
Unable to connect to worker at 192.168.99.102:9202
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 44229

 ~ ▓▒░ ssh kk@192.168.99.102 uname -a                                                                                                                                                   ░▒▓ 255 х  09:27:38 PM
kk@192.168.99.102's password:
Linux kk-ubvm 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

 ~ ▓▒░

NAT rules configured on virtualbox

malnick · October 25, 2020, 4:20pm

@karthick-core - in your first question you said that you’re “running boundary server on a single virtualbox vm” - should the IP be the same as the controller? I.e., 10.0.2.5?

karthick-core · October 25, 2020, 4:41pm

Let me try to clarify the network config of the boundary server VM.

Controller and Worker configured on a single Ubuntu VM.
The VM has 2 network interfaces

NAT network - 10.0.2.5 IP for the VM (controller is configured on this network)
Host Only network - 192.168.99.102 IP (I did not have this before this post, and it’s added just to directly connect the VM from host and add an entry for public_addr)

The whole idea is to reach the other virtualbox vms connected on the nat network(10.0.2.x) through the boundary server(10.0.2.5).

jeff · October 25, 2020, 7:35pm

By not defining a listener for the worker it will default to 127.0.0.1 (you can verify this looking at the info printed on startup). Nothing outside the host will therefore be able to connect to it.

You need to set a listening address for the worker. Note that if it’s the same as the public address you want (which it sounds like it should be, 10.0.2.5) then you don’t need to set public_addr. Also if it’s all on the same box you can combine the configs and just run it once.

karthick-core · October 26, 2020, 6:21am

Thanks @jeff. That helps. I did not have the listener address configured for the worker.
Adding address = 192.168.99.102 to the proxy listener does the job.
The documentation may need an update I guess, no references for this parameter in https://www.boundaryproject.io/docs/configuration/worker

malnick · October 26, 2020, 2:37pm

Thanks @karthick-core - I added https://github.com/hashicorp/boundary/pull/749 to address the missing address parameter for worker configuration.