I am deploying Consul API Gateway on Kubernetes Cluster on AWS (EKS). I am following this documentation: Control Access into the Service Mesh with Consul API Gateway | Consul - HashiCorp Learn and it did work well.
I have a question regarding one point; which component is responsible of Creating the Load-Balancer in AWS and how can I configure it.
The Load-Balancer is created after applying this step:
kubectl apply -f consul-api-gateway.yaml -n consul
consul-api-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: api-gateway
namespace: consul
spec:
gatewayClassName: consul-api-gateway
listeners:
- protocol: HTTPS
port: 8443
name: https
allowedRoutes:
namespaces:
from: Selector
selector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- brain
- consul
- vault
tls:
certificateRefs:
- name: consul-server-cert
Is there any tutorial or documentation on how to make Consul API Gateway create a production ready Load-Balancer according to best practices?
Hi @kingindanord,
The Load Balancer is created/configured according to the serviceType parameter used in the Gateway’s corresponding GatewayClassConfig.
From a helm-based install, this is exposed via the apiGateway.managedGatewayClass.serviceType option that sets up the default GatewayClass object (named consul-api-gateway) and its corresponding configuration. That GatewayClass gets used in the tutorial by passing in the proper value under gatewayClassName in the Gateway definition.
Our Load Balancer behavior is to expose any ports that the Gateway has a listener bound to and delegate most of the rest to just stock cloud-provider behavior. There is one major exception to this, however, which will hopefully get you what you need.
Many of the cloud providers allow configuration of a Load Balancer via annotations. We expose an option on the GatewayClassConfig (and in the helm chart), to specify annotations that you want to copy from the Gateway definition itself onto the provisioned Load Balancer. The option is called copyAnnotations.service and can be found at apiGateway.managedGatewayClass.copyAnnotations.service in the helm chart.
By leveraging that you should be able to allow any annotations that you might want copied from a Gateway to the Load Balancer and then just annotate the Gateway when you create it. For example, adding external-dns.alpha.kubernetes.io/hostname as an allowed annotation would enable you to annotate the Gateway object with something like external-dns.alpha.kubernetes.io/hostname: myhost.com, and, if you have ExternalDNS properly configured, the corresponding LoadBalancer will get the same annotation and you’ll get DNS records created automatically for it.
If you’re wanting to modify any sort of provisioned Load Balancer behavior on say something like AWS, you’d add the corresponding annotations to the copyAnnotations.service allowlist and then add the annotated value to your Gateway definition. Likewise, here’s the list of corresponding annotations that can be added to GKE Load Balancers. And for AKS.
Let me know if this helps or if you have any other questions about behavior.
3 Likes
That was very helpful, thanks.
Hi @andrewstucki,
After reading your explaination and the attached links, I did try today to recreate the API Gateway but unfortunately I am still not able to modify the behavior of the provisioned Load Balancer
first I added the copyAnnotations to the consul helm chart values
helm/consul.yml
...
apiGateway:
enabled: true
image: hashicorp/consul-api-gateway:0.4.0
managedGatewayClass:
enabled: true
copyAnnotations:
service:
annotations: |
- external-dns.alpha.kubernetes.io/hostname
- service.beta.kubernetes.io/aws-load-balancer-name
- service.beta.kubernetes.io/aws-load-balancer-type
- service.beta.kubernetes.io/aws-load-balancer-nlb-target-type
- service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
- service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
upgrade the helm chart
helm upgrade consul hashicorp/consul --namespace consul -f helm/consul.yml
then, I’ve created a costume API Gateway class with a list of allowed Annotations
kubectl apply -f CostumeGatewayClass.yaml
CostumeGatewayClass.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GatewayClass
metadata:
name: test-gateway-class
spec:
controllerName: "hashicorp.com/consul-api-gateway-controller"
parametersRef:
group: api-gateway.consul.hashicorp.com
kind: GatewayClassConfig
name: test-gateway-class-config
description: test gateway
---
apiVersion: api-gateway.consul.hashicorp.com/v1alpha1
kind: GatewayClassConfig
metadata:
finalizers:
- gateway-class-exists-finalizer.api-gateway.consul.hashicorp.com
generation: 2
labels:
app: consul
component: api-gateway
name: test-gateway-class-config
spec:
consul:
authentication:
managed: true
method: consul-k8s-auth-method
ports:
grpc: 8502
http: 8501
scheme: https
copyAnnotations:
service:
- external-dns.alpha.kubernetes.io/hostname
- service.beta.kubernetes.io/aws-load-balancer-name
- service.beta.kubernetes.io/aws-load-balancer-type
- service.beta.kubernetes.io/aws-load-balancer-nlb-target-type
- service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
- service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
deployment:
defaultInstances: 1
maxInstances: 8
minInstances: 1
image:
consulAPIGateway: hashicorp/consul-api-gateway:0.4.0
envoy: envoyproxy/envoy:v1.22.4
logLevel: trace
serviceType: LoadBalancer
and finally I create the API Gateway with the needed Annotations and reference to the custom API Gateway Class
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: api-gateway
namespace: consul
annotations:
service.beta.kubernetes.io/aws-load-balancer-name: "test-nlb"
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "50"
spec:
gatewayClassName: test-gateway-class
listeners:
- protocol:
port: 8443
name: https
allowedRoutes:
namespaces:
from: Selector
selector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- brain
- consul
- vault
tls:
certificateRefs:
- name: consul-server-cert
The Load Balancer is being created but it does not seem to take the values from the Annotations. as you can see in the picture the name and healthcheck-timeout did not change.
I am not sure if I am making a silly mistake somewhere by puting the wrong quotation or indentations. Any help is appreciated.
Hi @kingindanord,
From a cursory look over your configuration, that seems like the proper configuration to me. Can you dump the Gateway’s corresponding LoadBalancer-type Service that’s getting created in Kubernetes? It should be in the same namespace/with the same name as the Gateway you created. If everything is configured correctly you should see the identical annotations on the Service object.
That said, when you add the annotations to the Gateway, make sure you delete and let the Service object get re-created. I recall seeing somewhere in Amazon’s LB controller docs that they only pick up certain annotations at Service creation time, rather than when a Service is modified.
Thanks for the hint regarding checking the LoadBalancer service. I found out in the events that there is error in the values I am giving within the annotations (like the health-checks timeout is higher than the interval). So the configuration is actually working
1 Like
