Accessing AWS Internal Load Balancer with Boundary?

Hi,

I would like to confirm with you whether the following setup can be achieved with Boundary:

I’m running AWS EKS with Application Load Balancer (ALB). My app is running at URL, let’s say:

https://myapp.com

Inside this EKS, I’m running CI/CD pipeline with Jenkins and ArgoCD and those tools are running at URL:

https://myapp.com/jenkins
https://myapp.com/argocd

Of course Jenkins and ArgoCD are protected with password, but still I don’t like to have those tools to be accessed from public.

I suppose this setup cannot be achieved with single ALB.
I suppose ALB for myapp.com should be of course of type: internet facing.

Jenkins and ArgoCD should have the second ALB which will be of type: internal.

Now Boundary setup…

Let’s suppose I will have running this setup in PROD env.

So EKS runs in one VPC and Boundary will be running in another VPC.

  1. From what Boundary component should I allow traffic to this internal ALB?
  2. Should I only create a “static” Host Catalog and in there a host with hostname/IP of that internal ALB?
  3. Or can I somehow add this internal ALB through “dynamic” Host Catalog?
  4. Then I should connect like this: boundary connect http -targed-id=XYZ -host-id=id_of_internal_lb_host ?

Thank you very much for your opinions and suggestions.

  • Cross-VPC Boundary proxying should work as long as the user clients can reach the workers and the workers can reach both the target of the proxying, and the controllers, across whatever VPC boundaries exist.

  • Right now a static catalog is the only way you’ll get an ALB FQDN or IP in there.

  • If your ALB address is the only host in its host set, you don’t need to specify the host for it.

Where things get interesting is proxy access to an HTTPS target. By default the target’s TLS certificate may not include the localhost or 127.0.0.1 hosts as valid names/IPs on the certificate, so you’ll get a browser error about the cert being invalid. In your case, you control the ALB so you can probably easily add these to its TLS certificate, and then TLS targets using it should work over the Boundary proxy.

The other thing to note is that with targets intended to be accessed via a web browser over HTTP/HTTPS, you probably want to disable the connection limit per session in the target config by setting it to -1. Otherwise you risk using up your 1 allowed connection (by default) on your web browser trying to load things like favicon.ico first and not getting the actual web page.

Thank you for your answer. Basically, I’m just curious whether this is a “clean” solution for accessing private resources in the public cloud via browser. In other words, whether Boundary was originally intended to use in such a way.

On the homepage of Boundary there is a quote:

In the shift to the cloud, organizations need secure access to targets beyond their own perimeter.

I’m not sure whether my use case follows this quote. According to what you wrote it could be done but it seems a little bit to me as kind of a workaround and not a “clean/intended” Boundary approach.

Also, I wouldn’t like to use the old way approach which means allowing FW rules to the internal ALB IP from a public IP assigned by VPN or from SSH bastion.

Or is there any completely different solution for my use case which I’ve been overlooking?

Thanks

HTTP target access is an intended use case, it’s specifically host naming and addressing that throws a wrench into things here. You’d see the same issue with a port-forward to a host-routing load-balancer, for the same reason – you’re doing a layer-4 proxy to something that has layer-7 configuration that doesn’t like its layer 4 config changing underneath it.

For cross-VPC access, I admit I’m close to the limit of my AWS knowledge on this particular thing but I think VPC peering will allow you to have that cross-VPC traffic without it traversing the public Internet. Then you could have Boundary workers in your Boundary VPC, proxying to targets in a peered private VPC, and the user clients only need to have access to the workers in the Boundary VPC.

I’ve tried this out today. I’ve created simple Kubernetes nginx pod with simple ingress like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "test-ingress"
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: test-nginx-ing-service
            port: 
              number: 80

this creates Application Load Balancer (ALB) DNS, something like this:

k8s-bf65d75ad8-92236164.eu-central-1.elb.amazonaws.com

Normally, security group for this ALB has inbound rules allowed from 0.0.0.0/0 which means from all over the world.

I’ve changed this to the IP address of my Boundary worker node which runs in different VPC.

I’ve added ALB DNS to the host catalog of my Boundary and created nginx target with port 80.

If I run this cmd:

boundary connect http -target-id ttcp_WymAQxwn4W -host-id hst_s1kGGMmW4y                                                     
curl: (35) error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

I’m getting this error, I don’t know why.

However if I create a session from Boundary Desktop App like this:

and open this in the browser, then it’s working:

image

However, my original idea was, I’d like to have an address myapp.com/nginx which will be allowed via Boundary, not proxy url.

Does it help if I create some kind of Internal Load Balancer and some kind of private service from Boundary to this Internal LB or this boundary connect http always creates this proxy url, no matter what type of service is behind the hostname/IP provided in the host catalog. :frowning:

Thanks

The boundary connect http helper defaults to trying to proxy via an HTTPS connection to the target, to default to a more secure posture. If you want to proxy to an HTTP target, you can do that either by giving the option -scheme=http or by setting the environment variable BOUNDARY_CONNECT_HTTP_SCHEME to http.
You can do this, actually, but it involves additional setup. The simplest alternative is just to set up an HTTP proxy inside the private network, and use that as a target with Boundary. Connect to the target with the -port option set to a predetermined port (so you’ll have to do this using the CLI at present, not the desktop app). Then start a web browser with the Boundary connection host and port set as its proxy configuration, and you can access private HTTP endpoints by the hostname or IP the internal proxy can resolve for them.

In your case, I think you effectively already did the first part by setting up ingress-nginx. You should be able to use the nginx-ingress address as your target and the boundary connect host:port as your browser proxy and host-based routing (if you have any set up) to services behind the ingress ought to work.

1 Like

I’m not sure what you mean by “proxy inside the private network”. I just want to connect to the let’s say as mentioned before: myapp.com/nginx from my local PC from “anywhere in the world”. So I don’t work in any “private company network”. I work in different places.

Today I’ve tested something like this:

boundary connect -listen-port 12345 -target-id ttcp_ttwuGIVYPv                 

Proxy listening information:
  Address:             127.0.0.1
  Connection Limit:    -1
  Expiration:          Wed, 20 Apr 2022 00:21:54 CEST
  Port:                12345
  Protocol:            tcp
  Session ID:          s_Nfw8LjhjI5

Then I’ve configured an HTTP proxy like this:

Suddenly I am able to connect to my ALB DNS something like this:

k8s-bf65d75ad8-92236164.eu-central-1.elb.amazonaws.com

from my browser but I don’t understand why other web pages like google.com or whatever are working through this proxy.

and I also don’t understand why my Boundary LB is NOT working through this proxy:

If I turn off the proxy the Boundary LB is working again and k8s…amazonaws.com is not working (of course).

So this is not really good approach. And what if I want to securely connect to the more webpages like I mentioned in my first post:

https://myapp.com/jenkins
https://myapp.com/argocd

If I add both of them to the Host Set and I connect to the target-id only, Boundary randomly connects to the one host-id only.

So I’ll have to run 2 boundary connect commands with the port 12345 and with the port 54321 and I cannot create proxy for both of them simultaneously.

I’m starting to doubt whether I can achieve something like socks proxy through SSH host inside the VPC which has a public IP and has an access to everything inside of private subnets of VPC.

In this case “private network” would be your private resources inside AWS. Let me see if a couple of ASCII diagrams make things clearer. (They might not :slight_smile: )

This is how Boundary proxying usually works (I’m leaving out the control plane here since it’s not part of the actual proxying): the worker sits on the network perimeter. The client connects directly to the worker and the worker forwards that connection to the target endpoint, then forwards the response back, and the client doesn’t have any idea any proxying is happening at all.

                  |
[client] <--> [worker] <--> [target: webapp]
                  |

However, you run into some issues with that when you have to preserve hostname info, because now everything is connecting through 127.0.0.1 by default and that tends to throw off things like TLS negotiation and host-based routing. So one model you can use to overcome that is to explicitly add an HTTP proxy in the private area, make that the Boundary connect target, and configure proxying on the client, so the hostname becomes meaningful again:

                                             |
[client: proxy 127.0.0.1:some-port] <--> [worker] <--> [target: proxy:some-port] <--> [webapp]
                                             |

Now in that second diagram, I think a Kubernetes Ingress gateway should work fine as that proxy, at least for things the Ingress is configured to route to, which you already saw some of when you were able to connect to various things through it when you connected to that target with Boundary and used it as a proxy.

When you’ve got things set up that way, I think you should be able to connect to http://myapp.com/jenkins and http://myapp.com/argocd as long as myapp.com points at your Ingress gateway and it has path-based routes for those services. If that works, then https:// to both of those should work too, as long as the cert is signed by a trusted CA and the Ingress gateway is configured properly.

I’m not sure what’s going on with proxying to external hosts though. As far as I know, anything not explicitly configured as an Ingress should be routed to the default 404-responder backend, which I think is what you’re seeing with the Boundary LB address when you try to connect there (a Boundary admin GUI 404 looks different than that, you wouldn’t get an nginx error). I don’t know why that’s not happening with google.com and other external public hosts.


We’re getting into territory where I’m half-speculating on what should work based on pieces I’ve done, though, so I’m going to see if I can’t put something more explicit together tonight or tomorrow, because this is a use case that comes up in general a lot but this is an interesting specific variation of that pattern that I haven’t really worked out yet.

So just to make sure I have your parameters right, you want:

  • to be able to access myapp.com/[various application paths] behind a Kubernetes Ingress, over both http: and https:, with it working as it normally would
  • to be able to access other resources in that private network like the Boundary controller LB as well
  • ideally, for that to not provide access to things outside the private network.

Do you have any special configuration set on that Ingress resource or is it just path-based routing to your backends? Is myapp.com explicitly set as a host on those Ingress resources or are they just using the ‘*’ host so they route that path to those services no matter what hostname you use?

Yes, this is correct except when I install the Boundary with this setup, that Boundary LB with port 9200 is type: Network and it is publicly available → not in the private network of AWS:

../aws/lb.tf

resource "aws_lb" "controller" {
  name               = "${var.tag}-controller-${random_pet.test.id}"
  load_balancer_type = "network"
  internal           = false
  subnets            = aws_subnet.public.*.id

  tags = {
    Name = "${var.tag}-controller-${random_pet.test.id}"
  }
}

Nothing special, here is my usual ingress config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "myapp-ingress"
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports:  '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:account-id:certificate/cert-id
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=300
    external-dns.alpha.kubernetes.io/hostname: myapp.com
spec:
  rules:
  - http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: ssl-redirect
            port: 
              name: use-annotation
      - pathType: Prefix
        path: /
        backend:
          service:
            # this should be publicly available
            name: webapp-service 
            port: 
              number: 8080
      - pathType: Prefix
        path: /jenkins
        backend:
          service:
            # this should be available via Boundary only
            name: jenkins-service
            port: 
              number: 8080
      - pathType: Prefix
        path: /argocd
        backend:
          service:
            # this should be available via Boundary only
            name: argocd-service
            port: 
              number: 8080

In the namespace kube-system, I’m running those services:

kubectl get pods -n kube-system
                                                                                                                 
NAME                                                            READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-f9c569c8-t5pzj                     1/1     Running   0          26d
external-dns-6f8b5585df-hh5t2                                   1/1     Running   0          26d

aws-load-balancer-controller automatically creates ALB in AWS based on annotation:
kubernetes.io/ingress.class: alb

external-dns automatically creates a record in Route53 based on annotation: external-dns.alpha.kubernetes.io/hostname

As I mentioned earlier, this approach can be done through Socks proxy and for example with this foxy proxy addon, you can specify which hosts should go through this Socks proxy and which not. For example pattern like this:

https://myapp.com/*, would go through this proxy but: https://myapp.com, would not.

But this means you have to administrate SSH bastion and users who have access to this SSH bastion, all that tedious stuff that should be replaced by Boundary :slight_smile:

Today I’ve found and tested a completely different solution → AWS ALB with OIDC.

I’ve added these 3 annotations to my AWS ALB:

metadata:
  name: "test-ingress"
  annotations:
    alb.ingress.kubernetes.io/auth-type: oidc
    alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate
    alb.ingress.kubernetes.io/auth-idp-oidc: '{"issuer":"https://login.microsoftonline.com/some-id/v2.0","authorizationEndpoint":"https://login.microsoftonline.com/some-id/oauth2/v2.0/authorize","tokenEndpoint":"https://login.microsoftonline.com/some-id/v2.0/token","userInfoEndpoint":"https://graph.microsoft.com/oidc/userinfo","secretName":"aws-alb-secret"}'

and added this secret to K8s:

apiVersion: v1
kind: Secret
metadata:
  name: aws-alb-secret
data:
  clientID: some-base64
  clientSecret: some-base64

and now if I open https://myapp.com, then I got redirected to MS Azure login page. After successfull authentication I got redirected back to my https://myapp.com.

but anyway I’m still curious whether you can make this through Boundary, because Boundary would add there an extra layer of security :slight_smile: