Can controller and worker communicate through public ip with public cert

Boundary
1

hello everyone, so here is how I want to use boundary to secure access to my services.

I use terraform to provision boundary controller onto google cloud and then workers from my onprem datacenters will register to the controller so that I could ssh to and do some development and testing, and without exposing other ports than 80 and 443. on controller node I have haproxy at the front and using acme.sh to get ssl cert and same on my onprem datacenter which has haproxy node as well and getting cert from zerossl

currently I am able to use recovery key with terraform to provision boundary resources such as organizations, users, hosts and targets etc from my local that point to https://boundary.example.com and next trying to connect between controller and worker

here is my controller config

disable_mlock = true

listener "tcp" {
  address = "0.0.0.0:9200"
  purpose = "api"

  tls_disable   = true
}

listener "tcp" {
  address = "0.0.0.0:9201"
  purpose = "cluster"
}

controller {
  name = "boundary-controller-0"

  public_cluster_addr = "boundary.example.com:443"

  graceful_shutdown_wait_duration = "16s"

  database {
    url = "postgresql://boundary:boundary@localhost:5432/boundary"
  }
}

events {
  audit_enabled       = true
  sysevents_enabled   = true
  observations_enable = true
  sink "stderr" {
    name = "all-events"
    description = "All events sent to stderr"
    event_types = ["*"]
    format = "cloudevents-json"
  }
  sink {
    name = "file-sink"
    description = "All events sent to a file"
    event_types = ["*"]
    format = "cloudevents-json"
    file {
      path = "/var/log/boundary"
      file_name = "controller.log"
    }
    audit_config {
      audit_filter_overrides {
        sensitive = "redact"
        secret    = "redact"
      }
    }
  }
}

kms "aead" {
  purpose = "root"
  aead_type = "aes-gcm"
  key = "{{ root_key }}"
  key_id = "root"
}

kms "aead" {
  purpose    = "recovery"
  aead_type = "aes-gcm"
  key = "{{ recovery_key }}"
  key_id = "recovery"
}

kms "aead" {
  purpose    = "worker-auth"
  aead_type = "aes-gcm"
  key = "{{ worker_auth_key }}"
  key_id = "worker-auth"
}

my worker config

listener "tcp" {
  purpose = "proxy"
  address = "0.0.0.0:9212"
}

worker {
  name = "boundary-worker-0"
  public_addr = "service0.region0.example.com:443"
  initial_upstreams = [
    "boundary.example.com:443"
  ]
}

kms "aead" {
  purpose = "worker-auth"
  aead_type = "aes-gcm"
  key = "{{ worker_auth_key }}"
  key_id = "worker-auth"
}

here are some of the logs from controller when starting up

Sep 18 15:31:49 boundary-controller-0 boundary[27003]: ==> Boundary server configuration:
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:             [Recovery] AEAD Type: aes-gcm
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                 [Root] AEAD Type: aes-gcm
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:          [Worker-Auth] AEAD Type: aes-gcm
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                              Cgo: disabled
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:   Controller Public Cluster Addr: boundary.example.com:443
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                       Listener 1: tcp (addr: "0.0.0.0:9200", cors_allowed_headers: "[]", cors_allowed_origins: "[*]", cors_enabled: "true", max_request_duration: "1m30s", purpose: "api")
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                       Listener 2: tcp (addr: "0.0.0.0:9201", max_request_duration: "1m30s", purpose: "cluster")
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                        Log Level: info
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                            Mlock: supported: true, enabled: false
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                          Version: Boundary v0.13.1
Sep 18 15:31:49 boundary-controller-0 boundary[27003]:                      Version Sha: db01791662a7126fbf4ea0a27b23b70acd20b17b
Sep 18 15:31:49 boundary-controller-0 boundary[27003]: ==> Boundary server started! Log data will stream in below:
Sep 18 15:31:49 boundary-controller-0 boundary[27003]: {"id":"r9UYaEOK8h","source":"https://hashicorp.com/boundary/boundary-controller-0/controller","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"github.com/hashicorp/boundary/internal/observability/event.(*HclogLoggerAdapter).writeEvent","data":{"@original-log-level":"none","@original-log-name":"aws","msg":"configuring client automatic mTLS"}},"datacontentype":"application/cloudevents","time":"2023-09-18T15:31:49.763563777+07:00"}

when worker starting up(fail)

==> Boundary server configuration:

           [Worker-Auth] AEAD Type: aes-gcm
                               Cgo: disabled
                        Listener 1: tcp (addr: "0.0.0.0:9212", max_request_duration: "1m30s", purpose: "proxy")
                         Log Level: info
                             Mlock: supported: true, enabled: true
                           Version: Boundary v0.13.1
                       Version Sha: db01791662a7126fbf4ea0a27b23b70acd20b17b
        Worker Auth Current Key Id: scruffy-favoring-legend-dragster-freebee-tassel-qualm-overtime
  Worker Auth Registration Request: GzusqckarbczHoLGQ4UA25uSRACS1zBDorjx1xvwosGauYub4MfYQVcvmW69oSHweFa1eJ3g9CzibGNZyFv8346yFbs6wjhH7BkbPEWKFaegNsbdpT52BvdLZN83XmiD4pkanaMw9XjRNLZuGHz3yvSS7msvsrV8ZfxiCUGkNY45aRHXcfEEFJYDnL6GxjiymWjCwrCE51H9CuUwZJNQB1R6SJHpEdELVtQWuZPJoEXLXvsreFYUX8jbmnZG7UcjENmM46iJa2LsSrMwuT2iFZnAuMBU5JofWgThr6un8L
          Worker Public Proxy Addr: service0.region0.example.com:443

==> Boundary server started! Log data will stream in below:

{"id":"GYTYB05cao","source":"https://hashicorp.com/boundary/localhost.localdomain/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).startAuthRotationTicking","data":{"msg":"starting auth rotation ticking"}},"datacontentype":"application/cloudevents","time":"2023-09-18T04:34:33.90001624-04:00"}
{"id":"jaV6nou8sr","source":"https://hashicorp.com/boundary/localhost.localdomain/worker","specversion":"1.0","type":"error","data":{"error":"(nodeenrollment.protocol.attemptFetch) error base-64 decoding fetch response: illegal base64 data at input byte 8","error_fields":{},"id":"e_R0Nuf2y6Ji","version":"v0.1","op":"worker.(Worker).upstreamDialerFunc"},"datacontentype":"application/cloudevents","time":"2023-09-18T04:34:34.177848205-04:00"}
{"id":"VKnnGt9yg3","source":"https://hashicorp.com/boundary/localhost.localdomain/worker","specversion":"1.0","type":"error","data":{"error":"worker.(Worker).upstreamDialerFunc: unknown, unknown: error #0: (nodeenrollment.protocol.attemptFetch) error base-64 decoding fetch response: illegal base64 data at input byte 8","error_fields":{"Code":0,"Msg":"","Op":"worker.(Worker).upstreamDialerFunc","Wrapped":{}},"id":"e_HOxwj9z7iK","version":"v0.1","op":"worker.(Worker).upstreamDialerFunc"},"datacontentype":"application/cloudevents","time":"2023-09-18T04:34:34.177985946-04:00"}
{"id":"yKItvSdA19","source":"https://hashicorp.com/boundary/localhost.localdomain/worker","specversion":"1.0","type":"error","data":{"error":"(nodeenrollment.protocol.attemptFetch) error base-64 decoding fetch response: illegal base64 data at input byte 8","error_fields":{},"id":"e_OlPhT4f7y4","version":"v0.1","op":"worker.(Worker).upstreamDialerFunc"},"datacontentype":"application/cloudevents","time":"2023-09-18T04:34:35.181235117-04:00"}

while worker starting up, controller does not show any additional logs so I just stop the worker server since it keeps retrying

here is haproxy config for the controller

global
  log 127.0.0.1:514 local0

defaults
  log global
  retries 4
  timeout client 32s
  timeout server 32s
  timeout connect 8s

frontend default
  mode http
  bind *:80
  bind *:443 ssl crt /etc/haproxy/certs/ strict-sni

  timeout server 32s

  acl acl_acme path_beg /.well-known/acme-challenge/
  use_backend acme if acl_acme

  acl acl_boundary_controller hdr(host) -i boundary.example.com
  use_backend boundary_controller if acl_boundary_controller

backend acme
  mode http
  server zerossl 127.0.0.1:8088

backend boundary_controller
  mode http
  server svr0 127.0.0.1:9200

here is haproxy config for the worker

global
  log 127.0.0.1:514 local0

defaults
  log global
  retries 4
  timeout client 32s
  timeout server 32s
  timeout connect 8s

frontend default
  mode http
  bind *:80
  bind *:443 ssl crt /etc/haproxy/certs/ strict-sni

  timeout server 32s

  acl acl_acme path_beg /.well-known/acme-challenge/
  use_backend acme if acl_acme

  acl acl_boundary_agent hdr(host) -i service0.region0.example.com.com
  use_backend boundary_agent if acl_boundary_agent

backend boundary_agent
  mode http
  server svr0 127.0.0.1:9212

backend acme
  mode http
  server zerossl 127.0.0.1:8088
2

There are a couple of places Boundary should not be load-balanced – the worker needs direct access to all controllers and the clients need direct access to individual workers. It looks like your haproxy may be interfering with Boundary’s own internal TLS verification between workers and controllers.

For situations where you want a single point of ingress from the workers to the controller environment, if you’re using HCP Boundary or Boundary Enterprise, you can run a single multi-hop worker in the same subnet as the control plane, and workers in other subnets/environments can use that worker as their upstream.

3

hello, so I update controller and worker to use port 6443 for public cluster address and public worker address and use ssl passthrough instead, now everything is working as expected but then I have to open another port 6443 for tcp connection, but that is fine I think since communication between worker and controller are done over mtls

here are my latest configs

controller config

disable_mlock = true

listener "tcp" {
  address = "0.0.0.0:9200"
  purpose = "api"

  tls_disable   = true
}

listener "tcp" {
  address = "0.0.0.0:9201"
  purpose = "cluster"
}

controller {
  name = "boundary-controller-0"

  # boundary.example.com domain is for
  # admin ui and api that points to :9200
  public_cluster_addr = "cluster.boundary.example.com:6443"

  graceful_shutdown_wait_duration = "16s"

  database {
    url = "postgresql://boundary:boundary@localhost:5432/boundary"
  }
}

events {
  audit_enabled       = true
  sysevents_enabled   = true
  observations_enable = true
  sink "stderr" {
    name = "all-events"
    description = "All events sent to stderr"
    event_types = ["*"]
    format = "cloudevents-json"
  }
  sink {
    name = "file-sink"
    description = "All events sent to a file"
    event_types = ["*"]
    format = "cloudevents-json"
    file {
      path = "/var/log/boundary"
      file_name = "controller.log"
    }
    audit_config {
      audit_filter_overrides {
        sensitive = "redact"
        secret    = "redact"
      }
    }
  }
}

kms "aead" {
  purpose = "root"
  aead_type = "aes-gcm"
  key = "{{ root_key }}"
  key_id = "root"
}

kms "aead" {
  purpose    = "recovery"
  aead_type = "aes-gcm"
  key = "{{ recovery_key }}"
  key_id = "recovery"
}

kms "aead" {
  purpose    = "worker-auth"
  aead_type = "aes-gcm"
  key = "{{ worker_auth_key }}"
  key_id = "worker-auth"
}

worker config

listener "tcp" {
  purpose = "proxy"
  address = "0.0.0.0:9212"
}

worker {
  name = "boundary-worker-0"
  # this points to this:9212 from haproxy
  public_addr = "server0.region0.example.com:6443"
  initial_upstreams = [
    # this points to controller:9201 from haproxy
    "cluster.boundary.example.com:6443"
  ]
}

kms "aead" {
  purpose = "worker-auth"
  aead_type = "aes-gcm"
  key_id = "worker-auth"
  key = "{{ worker_auth_key }}"
}

haproxy at local onprem (where worker’s at as well)

frontend tcp_conns
  mode tcp

  bind *:6443

  option tcplog

  tcp-request inspect-delay 8s
  tcp-request content accept if { req_ssl_hello_type 1  }

  acl acl_boundary_server0 req.ssl_sni -i server0.region0.example.com
  use_backend boundary_server0 if acl_boundary_server0

backend boundary_server0
  mode tcp

  server svr0 127.0.0.1:9212

haproxy at controller

frontend tcp_conns
  mode tcp

  bind *:6443

  option tcplog

  tcp-request inspect-delay 8s
  tcp-request content accept if { req_ssl_hello_type 1  }

  acl acl_boundary_cluster req.ssl_sni -i cluster.boundary.example.com
  use_backend boundary_cluster if acl_boundary_cluster

backend boundary_cluster
  mode tcp

  server svr0 127.0.0.1:9201