My team is doing a POC of Hashicorp Boundary for access management and so far I love the product.
I have one question around setting up Boundary. For a bit of context before I ask the question, we are running on Microsoft Azure and will likely use Azure AD as the ldap provider.
We also extensively use Azure PaaS services such as function apps, managed databases, redis and queues and K8s clusters.
While I did find this demo from Ned Bellavance on how to set up Boundary on Azure(Ned, if you are reading this, thank you for the brilliant demo).
I want to know if it is possible to run the controllers and workers on a Kubernetes clusters instead of running on virtual machines.
but it looks like this setup is specifically for resources running on the kubernetes cluster?
Is it possible to setup controllers and workers on kubernetes to access all cloud resources irrespective of wether they are running on the kubernetes cluster?
Boundary can manage access across cloud and runtime platforms (in fact, that’s a big reason why we built it!). As long as your k8s cluster has networking configured to allow ingress and egress from the workers and controllers you should be good to go. The configuration here can get a little tricky, but at the end of the day, as long as a TCP connection can be made from the client to the worker and from the worker to the target, you shouldn’t have any issues.
Adding to what @malnick said, I have a demo environment that runs a Boundary worker inside Kubernetes (k3s in my case), exposed as a NodePort service with a fixed port number, and it works great. Some additional things to consider:
Controller/worker access out of Kubernetes to the Boundary database and KMS keys – not just network connectivity, but how will Boundary processes auth to those resources? Explore using some type of instance profile credentials if you can.
How do you want to pass the config to the Boundary processes? I did it with a mounted Kubernetes Secret just for demo convenience but that’s not the only way to do it.
Add worker tags to the config of workers inside Kubernetes if there are workers that aren’t – this is not required, but will allow you to create targets that reference Kubernetes services by native cluster DNS name, with a worker filter so they are always forwarded through an in-cluster worker. (Note that services in other namespaces than the worker will need to be referred to by their K8s service FQDN, e.g. myapp.namespace.service.cluster.local)
(For my next trick, I plan to try tying in-cluster and out-of-cluster app instances together with Consul and use the Consul service discovery DNS name as a target, so the worker filtering for in-cluster services will no longer be necessary. In theory this is fairly trivial, I just want to fully work out an example.)
Hi there,
So I just tried running an HCP Worker in Kubernetes and it doesn’t work. The moment, the pod gets re-created, it will no longer register in the HCP… the first register and everything works just fine and I already had some direct sessions to my kubernetes hosts with internal network. So that is great. But I can’t find any docs on where the session is stored in the worker and in general, it doesn’t seem to be cloud ready, which is kind of strange.
HCP workers need to store some local auth state. If you use a regular Deployment with no persistent storage, then that state gets wiped out if the pod restarts, and the new pod can’t reauthenticate with the initial token because the token expired on use. What you need for the HCP worker (and other PKI workers) is a persistent volume for the pod to use as the auth_storage_path so if the pod restarts, the worker state is still available.
I am trying to accomplish a similar setup. I have Both boundary controller and worker are in the EKS as pods within same namespace. My goal is to access a db in a separate VPC from the VPC that EKS sits on, though they are connected through a transit gateway. I want to be able to connect to this db through the Boundary desktop app which should show the target and eventually establish the connection from a local computer. I setup Boundary controller API as the ingress for accessing the UI but currently stuck on setting up workers to establish connection to the db itself. If you have any insights on how to make this work or any documentation to share, that would be very helpful!
I am mostly trying to figure out how to expose the workers when running as a pod in Kubernetes cluster, specifically when trying to connect to databases that are in different VPC but connected via transit gateway to the EKS. Controller is already setup and running in EKS in the same namespace as workers but having trouble getting the worker to connect to database.
Does the controller need to be exposed as an ingress or a load balancer for this to work?
The controller needs to be exposed somehow to the client, and so does the worker. The most primitive form of worker exposure would be a NodePort, but you can also use a LoadBalancer service as long as the TCP forwarding is 1:1 (i.e, you can’t use layer-7 routing to expose multiple workers behind one load balancer at different paths). However you do it, make sure the worker advertises an IP address or hostname that clients can connect to).
For the database connection, as long as the worker has TCP connectivity to the database and the database works behind layer-4 TCP proxying, you should be good to go.
Thanks for your input. My configuration uses an open source helm chart and sets up the worker.hcl from the Configmap.yaml file during the init process along with creating services to expose etc. I realized it does setup the public_addr to localhost:9202 by default. When I try to connect to a database I get: error fetching connection to send session teardown request to worker: “Unable to connect to worker at localhost:9202” error.
Now if I expose workers service as NodePort or LoadBalancer at the creation of pods, I need to be able to get IP address that NodePort assigns to dynamically to be added to the configmap for the worker configuration am i right?
Since layer-7 is no feasible I assume the service would be a Network Load Balancer for workers to connect If decide to take that route.
Yeah, you need a predictable name for the worker to configure the worker to advertise – e.g. you can assign a pre-existing DNS record to the load balancer via annotations (if that’s supported for the underlying infrastructure, a la AWS), or you could just create, DNS-name and manage the LB separately and use ordinary NodePort services in Kubernetes.