Running Waypoint on KinD cluster with MetalLB

abangser · January 11, 2023, 9:57am

Hi all, I have been diving into the forum here and the issues and don’t think this is covered yet, I apologise if I have missed something!

I am trying to run Waypoint on a KinD cluster. As stated in the CLI output and the docs I know this requires a LoadBalancer type which I have used the suggested MetalLB install. I have actually even tried used this HashiCorp provided setup script.

I am not sure how to complete a login Here are the logs below from my helm install but inability to login. This seems to be a MetalLB issue maybe? But I have used MetalLB for other URLs before so I am a bit unsure.

Anyone else have success with this? See below for my output from install to fail to login.

$ helm install waypoint hashicorp/waypoint \ 
    --namespace waypoint \
    --set server.runArgs={-url-enabled=false}
NAME: waypoint
LAST DEPLOYED: Wed Jan 11 09:41:30 2023
NAMESPACE: waypoint
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Waypoint!

Waypoint will take a few minutes to bootstrap. Once it is bootstrapped, you
can log in to the local CLI by running the following command. If this command
fails, please wait a few more minutes and try again.

  $ waypoint login -from-kubernetes

Once logged in, you can use the "waypoint ui" command to open the web interface.
Alternately, you can visit the address of the load balancer service configured.

If anything fails to configure properly you can uninstall this release:

  $ helm uninstall waypoint

$ k get svc -A                                
NAMESPACE        NAME              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                    AGE
default          kubernetes        ClusterIP      10.96.0.1       <none>        443/TCP                                                    5m15s
kube-system      kube-dns          ClusterIP      10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP                                     5m13s
metallb-system   webhook-service   ClusterIP      10.96.41.56     <none>        443/TCP                                                    5m9s
waypoint         waypoint-server   ClusterIP      None            <none>        9702/TCP,9701/TCP                                          2m30s
waypoint         waypoint-ui       LoadBalancer   10.96.164.171   172.18.0.1    80:31038/TCP,443:32236/TCP,9701:31469/TCP,9702:30875/TCP   2m30s

$ k describe endpoints -n waypoint waypoint-ui
Name:         waypoint-ui
Namespace:    waypoint
Labels:       app=waypoint
              app.kubernetes.io/managed-by=Helm
              chart=waypoint-0.1.17
              component=ui
              heritage=Helm
              release=waypoint
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2023-01-11T09:41:52Z
Subsets:
  Addresses:          10.244.0.10
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    grpc     9701  TCP
    https    9702  TCP
    http     9703  TCP
    https-2  9702  TCP

Events:  <none>

$ waypoint login -from-kubernetes -from-kubernetes-namespace=waypoint                                                                                                           
Waypoint server URL: 172.18.0.1:9701
! Error reconnecting with token: error connecting to server: context deadline exceeded

briancain · January 11, 2023, 4:11pm

Hey there @abangser !

Sorry to hear about the login troubles with Waypoint. That is an interesting error. If you’re not using helm for a specific reason (like custom helm values or something), I suggest trying to re-install Waypoint again on Kind+Metallb with the waypoint server install CLI. I am the one who provided that script, and it’s what I use daily for Waypoint dev, so it should be setting up the load balancer properly for Waypoint server. I generally always use the waypoint server install script, which actually uses that helm chart behind the scenes to install Waypoint.

If the CLI command doesn’t work, let us know and we will provide further assistance. Thank you!

abangser · January 11, 2023, 8:05pm

Thanks so much for the very quick reply Brian! That script was magic and do really appreciate it being included in the repo.

I have since debugged this as definitely a Docker for Mac issue

For anyone else who comes in behind, while MetalLB works like a charm and allocates an IP, that IP is not networked to the host computer. This presents as a healthy LoadBalancer Service with healthy Endpoints, but inability to curl / load the IP on any port.

My work around was to actually patch the Helm chart as a NodePort service instead of LoadBalancer and allocate static NodePorts for key ports:

helm template waypoint hashicorp/waypoint \
    yq eval "select(.metadata.name == \"waypoint-ui\").spec.type |= \"NodePort\" |
             select(.metadata.name == \"waypoint-ui\").spec.ports = [
                { \"name\": \"http\", \"port\": 80, \"targetPort\": \"http\" },
                {\"name\": \"https\", \"port\": 443, \"nodePort\": 30003, \"targetPort\": \"https\"},
                {\"name\": \"grpc\", \"port\": 9701, \"nodePort\": 30001, \"targetPort\": \"grpc\"},
                {\"name\": \"https-2\", \"port\": 9702, \"nodePort\": 30002, \"targetPort\": \"https\"}
             ]"

Absolutely not ideal, but assuming that Mac (and more specifically M1’s that can’t run TapTun with Hypervisors) won’t be production environments so probably acceptable for now. My particular use case is KinD Clusters where I am able to expose these NodePorts via node extraPortMappings.

  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  nodes:
  - role: control-plane
    extraPortMappings:
    - containerPort: 30001
      hostPort: 30001 # Waypoint grpc
    - containerPort: 30002
      hostPort: 30002 # Waypoint https-2
    - containerPort: 30003
      hostPort: 30003 # Waypoint https

Thanks again and will report back if I find any better solutions,
Abby

abangser · January 12, 2023, 10:05am

Actually Brian I wonder if you know if that NodePort trick could cause an issue with the URL Service? I get these errors on repeat:

waypoint-server-0 waypoint 2023-01-12T01:10:46.191Z [DEBUG] waypoint.server.singleprocess.url_service: API token not set in config, initializing guest account
waypoint-server-0 waypoint 2023-01-12T01:10:46.191Z [DEBUG] waypoint.server.singleprocess.url_service: connecting to URL service to retrieve guest token: addr=api.waypoint.run:443 tls=true
waypoint-server-0 waypoint 2023-01-12T01:10:46.191Z [DEBUG] waypoint.server.singleprocess.url_service: waiting on server connection state to become ready
waypoint-server-0 waypoint 2023-01-12T01:10:46.814Z [DEBUG] waypoint.server.singleprocess.url_service: connection is ready

From my understanding, URL Service is turned on automatically here, though I have successfully templated this with it off using this:

helm template template hashicorp/waypoint --set server.runArgs={-url-enabled=false}

With this error, the request seems to be getting here in the code, which (if I am reading this correctly) means that initURLClientBlocking is successfully running and since I am not seeing failed to connect to the URL service It feels like the unauth’d call is working

With all of that I don’t think this has to do with my networking, but wanted to see if you had any insight since I am mangling things a bit

I can add any more context, but everything else in the logs looks healthy and the behaviour seems fine. In that, I can waypoint login and after that I can waypoint init and waypoint up just fine, just without getting a URL:

The deploy was successful! A Waypoint deployment URL is shown below. This
can be used internally to check your deployment and is not meant for external
traffic. You can manage this hostname using "waypoint hostname."

   Release URL: http://10.96.166.248:80