Issue with OCI (Oracle) none binding of a publicly accessible IP to a NIC

jessequinn · October 26, 2020, 6:43pm

I am trying to mock up a demo of boundary on OCI; however, I believe there is an issue with the public ip and boundary.

I have the following one-in-all config:

disable_mlock = true

controller {
  name = "demo-controller-1"
  description = "A controller for a demo!"

  database {
    url = "postgresql://xxx"
  }
}

worker {
  name = "demo-worker-1"
  description = "A default worker created demonstration"
  controllers = [
    "0.0.0.0",
  ]
  address = "0.0.0.0"
  public_addr = "xxx"
}

listener "tcp" {
  address = "0.0.0.0"
  purpose = "api"
  tls_disable = true
//  tls_disable = false
//  tls_cert_file = "/etc/boundary/xxx.crt"
//  tls_key_file  = "/etc/boundary/xxx.key"
//  cors_enabled = true
//  cors_allowed_origins = ["xxx"]
}

listener "tcp" {
  address = "0.0.0.0"
  purpose = "cluster"
  tls_disable = true
//  tls_disable   = false
//  tls_cert_file = "/etc/boundary/xxx.crt"
//  tls_key_file  = "/etc/boundary/xxx.key"
}

listener "tcp" {
  address       = "0.0.0.0"
  purpose       = "proxy"
  tls_disable = true
//  tls_disable   = false
//  tls_cert_file = "/etc/boundary/xxx.crt"
//  tls_key_file  = "/etc/boundary/xxx.key"
}

# Root KMS configuration block: this is the root key for Boundary
# Use a production KMS such as AWS KMS in production installs
kms "aead" {
  purpose = "root"
  aead_type = "aes-gcm"
  key = "sP1fnF5Xz85RrXyELHFeZg9Ad2qt4Z4bgNHVGtD6ung="
  key_id = "global_root"
}

# Worker authorization KMS
# Use a production KMS such as AWS KMS for production installs
# This key is the same key used in the worker configuration
kms "aead" {
  purpose = "worker-auth"
  aead_type = "aes-gcm"
  key = "8fZBjCUfN0TzjEGLQldGY4+iE9AkOvCfjh7+p0GtRBQ="
  key_id = "global_worker-auth"
}

# Recovery KMS block: configures the recovery key for Boundary
# Use a production KMS such as AWS KMS for production installs
kms "aead" {
  purpose = "recovery"
  aead_type = "aes-gcm"
  key = "8fZBjCUfN0TzjEGLQldGY4+iE9AkOvCfjh7+p0GtRBQ="
  key_id = "global_recovery"
}

The worker is listening on 9202 and the public DNS, netstat -tulpn

tcp6       0      0 :::9200                 :::*                    LISTEN      -                   
tcp6       0      0 :::9201                 :::*                    LISTEN      -                   
tcp6       0      0 :::9202                 :::*                    LISTEN      -

output from boundary

Oct 26 18:24:53 hecate-elk2 boundary[13355]:                 Log Level: debug
Oct 26 18:24:53 hecate-elk2 boundary[13355]:                     Mlock: supported: true, enabled: false
Oct 26 18:24:53 hecate-elk2 boundary[13355]:               Public Addr: xxx:9202
Oct 26 18:24:53 hecate-elk2 boundary[13355]:                   Version: Boundary v0.1.1
Oct 26 18:24:53 hecate-elk2 boundary[13355]:               Version Sha: eccd68d73c3edf14863ecfd31f9023063b809d5a
Oct 26 18:24:53 hecate-elk2 boundary[13355]: ==> Boundary server started! Log data will stream in below:
Oct 26 18:24:53 hecate-elk2 boundary[13355]: 2020-10-26T18:24:53.835Z [INFO]  controller: cluster address: addr=[::]:9201
Oct 26 18:24:53 hecate-elk2 boundary[13355]: 2020-10-26T18:24:53.835Z [INFO]  worker: connected to controller: address=0.0.0.0:9201
Oct 26 18:24:53 hecate-elk2 boundary[13355]: 2020-10-26T18:24:53.839Z [INFO]  controller: worker successfully authed: name=demo-worker-1
Oct 26 18:24:53 hecate-elk2 boundary[13355]: 2020-10-26T18:24:53.851Z [INFO]  controller: worker successfully authed: name=demo-worker-1

However, i get connection refused when doing a telnet ip 9200/9202. The firewall is open for these ports. I believe this is because OCI like AWS does not bind the public ip to a NIC. So how exactly does this work? Any ideas would be most welcomed!

malnick · October 26, 2020, 8:25pm

Thanks for trying out Boundary @jessequinn!

You can set the advertised address for the worker via public_addr. That will allow you to use an EIP or other address not bound to an interface on the worker.

Let me know if this helps!

jessequinn · October 26, 2020, 9:23pm

that is what I have done. but i still have no access to the API on 9200. As according to your documentation both the 9200 and 9202 need to be accessible to the client.

or is it possible to remove the address from the api listener and the api will utilize the public_addr?

jeff · October 27, 2020, 4:22am

You didn’t include all of the output so we can’t see the listener information, and you aren’t specifying listener blocks in your config so it’s hard to know what it’s actually listening on. Can you include some more information?

jessequinn · October 27, 2020, 10:19am

Hi Jeff,

I included all listener blocks in the config above but i will include them again:

listener "tcp" {
  address = "0.0.0.0"
  purpose = "api"
  tls_disable = true
//  tls_disable = false
//  tls_cert_file = "/etc/boundary/xxxx.crt"
//  tls_key_file  = "/etc/boundary/xxxx.key"
//  cors_enabled = true
 // cors_allowed_origins = ["xxxx"]
}

listener "tcp" {
  address = "0.0.0.0"
  purpose = "cluster"
  tls_disable = true
//  tls_disable   = false
//  tls_cert_file = "/etc/boundary/xxxx.crt"
//  tls_key_file  = "/etc/boundary/xxxx.key"
}

listener "tcp" {
  address       = "0.0.0.0"
  purpose       = "proxy"
  tls_disable = true
//  tls_disable   = false
//  tls_cert_file = "/etc/boundary/xxxx.crt"
//  tls_key_file  = "/etc/boundary/xxxx.key"
}

Full output from journal -u boundary-server

Oct 26 19:06:47 hecate-elk2 boundary[12623]: ==> Boundary server configuration:
Oct 26 19:06:47 hecate-elk2 boundary[12623]:      [Recovery] AEAD Type: aes-gcm
Oct 26 19:06:47 hecate-elk2 boundary[12623]:          [Root] AEAD Type: aes-gcm
Oct 26 19:06:47 hecate-elk2 boundary[12623]:   [Worker-Auth] AEAD Type: aes-gcm
Oct 26 19:06:47 hecate-elk2 boundary[12623]:                       Cgo: disabled
Oct 26 19:06:47 hecate-elk2 boundary[12623]:                Listener 1: tcp (addr: "0.0.0.0:9200", max_request_duration: "1m30s", purpose: "api")
Oct 26 19:06:47 hecate-elk2 boundary[12623]:                Listener 2: tcp (addr: "0.0.0.0:9201", max_request_duration: "1m30s", purpose: "cluster")
Oct 26 19:06:47 hecate-elk2 boundary[12623]:                Listener 3: tcp (addr: "0.0.0.0:9202", max_request_duration: "1m30s", purpose: "proxy")
Oct 26 19:06:47 hecate-elk2 boundary[12623]:                 Log Level: debug
Oct 26 19:06:47 hecate-elk2 boundary[12623]:                     Mlock: supported: true, enabled: false
Oct 26 19:06:47 hecate-elk2 boundary[12623]:               Public Addr: xxx-public-url:9202
Oct 26 19:06:47 hecate-elk2 boundary[12623]:                   Version: Boundary v0.1.1
Oct 26 19:06:47 hecate-elk2 boundary[12623]:               Version Sha: eccd68d73c3edf14863ecfd31f9023063b809d5a
Oct 26 19:06:47 hecate-elk2 boundary[12623]: ==> Boundary server started! Log data will stream in below:
Oct 26 19:06:47 hecate-elk2 boundary[12623]: 2020-10-26T19:06:47.016Z [INFO]  controller: cluster address: addr=[::]:9201
Oct 26 19:06:47 hecate-elk2 boundary[12623]: 2020-10-26T19:06:47.016Z [INFO]  worker: connected to controller: address=0.0.0.0:9201
Oct 26 19:06:47 hecate-elk2 boundary[12623]: 2020-10-26T19:06:47.021Z [INFO]  controller: worker successfully authed: name=demo-worker-1
Oct 26 19:06:47 hecate-elk2 boundary[12623]: 2020-10-26T19:06:47.032Z [INFO]  controller: worker successfully authed: name=demo-worker-1

jeff · October 27, 2020, 1:50pm

Huh – yesterday only the second and third blocks from your original post were showing up for me. Now the first one (with the config) is too. Sorry about that…no idea if something was up with Discuss or PEBKAC

From what I can tell, everything looks fine. Are you able to telnet to those ports from the same machine? That would at least rule out some listener issue and confirm it’s something about the ingress to the machine in OCI.

jessequinn · October 27, 2020, 2:02pm

i can telnet, i can authorize through the api on the host machine. however, anything trying to connect to this machine is rejected. I know it isnt the ports. Why, i have elasticseach on 9200 and it worked. I moved that to a different port and try to telnet boundary on 9200 and i get connection refused.

jessequinn · October 27, 2020, 4:05pm

What i do not understand you have the worker with a public_addr on PORT 9202, how does this work with the API listener which listens on port 9200? Shouldn’t a public_addr exist for the api as well? Do you have a working example on AWS with EIP? I would assume this almost follows the same idea of OCI where no interface (NIC) is created for the public ip. Cause from what i can see the following is occurring: i have 0.0.0.0:9200 listening however, the public ip is not an interface, and is automatically rejected.

jessequinn · October 27, 2020, 8:57pm

I built a dockerfile etc and i got this working on the same machine. No difference in configs. just weird. I will post the docker-compose setup tomorrow for those that are interested.

jessequinn · October 28, 2020, 10:50am

For those interested in containerizing boundary

ARG ALPINE_VERSION=3.12

FROM alpine:${ALPINE_VERSION}
LABEL authors="Jesse Quinn <jesse.quinn@indyxa.com.br>"

ARG BOUNDARY_VERSION=0.1.1
ARG BOUNDARY_USERID

ADD https://releases.hashicorp.com/boundary/${BOUNDARY_VERSION}/boundary_${BOUNDARY_VERSION}_linux_amd64.zip /tmp/
#ADD https://releases.hashicorp.com/boundary/${BOUNDARY_VERSION}/boundary_${BOUNDARY_VERSION}_SHA256SUMS      /tmp/
#ADD https://releases.hashicorp.com/boundary/${BOUNDARY_VERSION}/boundary_${BOUNDARY_VERSION}_SHA256SUMS.sig  /tmp/

WORKDIR /tmp/

COPY run-boundary.sh /bin/run-boundary.sh

ENV BOUNDARY_VERSION=${BOUNDARY_VERSION} \
    BOUNDARY_USERNAME="boundary" \
    BOUNDARY_USERID=${BOUNDARY_USERID:-1051}

#RUN apk --update add --virtual verify gpgme \
# && gpg --keyserver pgp.mit.edu --recv-key 91A6E7F85D05C65630BEF18951852D87348FFC4C \
# && gpg --verify /tmp/boundary_${BOUNDARY_VERSION}_SHA256SUMS.sig \
# && cat boundary_${BOUNDARY_VERSION}_SHA256SUMS | grep linux_amd64 | sha256sum -c \
# && apk del verify \
RUN apk --update --no-cache add curl tini libcap bash python3 openssl net-tools ca-certificates \
 && adduser -D -u $BOUNDARY_USERID $BOUNDARY_USERNAME \
 && unzip boundary_${BOUNDARY_VERSION}_linux_amd64.zip \
 && mv boundary /bin/ \
 && chmod +x /bin/run-boundary.sh /bin/boundary \
 && mkdir /boundary /boundary/ssl /boundary/config \
 && rm -rf /tmp/* \
 && rm -rf /var/cache/apk/* \
 && setcap cap_ipc_lock=+ep $(readlink -f $(which boundary))

COPY certs/SectigoRSADomainValidationSecureServerCA.crt /usr/local/share/ca-certificates/SectigoRSADomainValidationSecureServerCA.crt
COPY certs/USERTrustRSA-AAACA-xSign.crt /usr/local/share/ca-certificates/USERTrustRSA-AAACA-xSign.crt
COPY certs/xxx.crt /boundary/ssl/xxx.crt
COPY keys/xxx.key /boundary/ssl/xxx.key
RUN chown -R $BOUNDARY_USERNAME /boundary /bin/boundary /bin/run-boundary.sh \
 && update-ca-certificates

WORKDIR /

EXPOSE 9200 9202 9201

USER $BOUNDARY_USERNAME

ENTRYPOINT ["/sbin/tini", "--", "/bin/run-boundary.sh"]

#!/usr/bin/env bash

if [[ -z ${POSTGRES_HOST} ]]; then
    export POSTGRES_HOST=postgres
fi
if [[ -z ${PORTGRES_PORT} ]]; then
    export PORTGRES_PORT=5432
fi
if [[ -z ${PORTGRES_USERNAME} ]]; then
    export PORTGRES_USERNAME=xxx
fi
if [[ -z ${PORTGRES_PASSWORD} ]]; then
    export PORTGRES_PASSWORD=xxx
fi

if [[ ! -f /boundary/config/config.hcl ]]; then
    cat <<EOF > /boundary/config/config.hcl
disable_mlock = true

controller {
  name = "demo-controller-1"
  description = "A controller for a demo!"

  database {
    url = "postgresql://${PORTGRES_USERNAME}:${PORTGRES_PASSWORD}@${POSTGRES_HOST}:${PORTGRES_PORT}/boundary?sslmode=disable"
  }
}

worker {
  name = "demo-worker-1"
  description = "A default worker created demonstration"
  controllers = [
    "0.0.0.0",
  ]
  address = "0.0.0.0"
  public_addr = "xxx"
}

listener "tcp" {
  address = "0.0.0.0"
  purpose = "api"
  tls_disable = false
  tls_cert_file = "/boundary/ssl/xxx.crt"
  tls_key_file  = "/boundary/ssl/xxx.key"
}

listener "tcp" {
  address = "0.0.0.0"
  purpose = "cluster"
  tls_disable   = false
  tls_cert_file = "/boundary/ssl/xxx.crt"
  tls_key_file  = "/boundary/ssl/xxx.key"
}

listener "tcp" {
  address       = "0.0.0.0"
  purpose       = "proxy"
  tls_disable   = false
  tls_cert_file = "/boundary/ssl/xxx.crt"
  tls_key_file  = "/boundary/ssl/xxx.key"
}

# Root KMS configuration block: this is the root key for Boundary
# Use a production KMS such as AWS KMS in production installs
kms "aead" {
  purpose = "root"
  aead_type = "aes-gcm"
  key = "xxx"
  key_id = "global_root"
}

# Worker authorization KMS
# Use a production KMS such as AWS KMS for production installs
# This key is the same key used in the worker configuration
kms "aead" {
  purpose = "worker-auth"
  aead_type = "aes-gcm"
  key = "xxx"
  key_id = "global_worker-auth"
}

# Recovery KMS block: configures the recovery key for Boundary
# Use a production KMS such as AWS KMS for production installs
kms "aead" {
  purpose = "recovery"
  aead_type = "aes-gcm"
  key = "xxx"
  key_id = "global_recovery"
}
EOF
fi

/bin/boundary database init -config /boundary/config/config.hcl
/bin/boundary server -config /boundary/config/config.hcl -log-level debug

version: '3.8'

x-logging:
  &default-logging
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"

networks:
  boundary-dev-net:
    driver: bridge

services:
  boundary-dev:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        - ALPINE_VERSION=3.12
        - BOUNDARY_VERSION=0.1.1
    container_name: hashicorp-boundary-dev
    restart: always
    environment:
      POSTGRES_HOST: postgres
      PORTGRES_PORT: 5432
      PORTGRES_USERNAME: xxx
      PORTGRES_PASSWORD: xxx
    cap_add:
      - IPC_LOCK
    networks:
      - boundary-dev-net
    ports:
      - 9200:9200
      - 9202:9202
    depends_on:
      - postgres
    logging: *default-logging
    labels:
      org.label-schema.group: "development"

  postgres:
    image: postgres:12.2
    container_name: hashicorp-postgres-dev
    restart: always
    environment:
      POSTGRES_USER: xxx
      POSTGRES_PASSWORD: xxx
      POSTGRES_DB: xxx
    stop_grace_period: 1m
    volumes:
      - /DATA/hashicorp/boundary/postgresql/data:/var/lib/postgresql/data:rw
    networks:
      - boundary-dev-net
    logging: *default-logging
    labels:
      - "org.label-schema.group=development"