Consul API gateway with service mesh on EKS. Gateway ready - false

Hi,

I’m trying to setup Consul API Gateway with TLS for my service mesh.

Here are the steps I followed:

  1. Applied CRDs
kubectl apply --kustomize="github.com/hashicorp/consul-api-gateway/config/crd?ref=v0.2.0"
``
2. created values.yaml for helm with the following content:
```yaml
global:
  name: consul
connectInject:
  enabled: true
  default: true
  transparentProxy:
    defaultEnabled: true
  namespaceSelector: |
    matchLabels:
      connect-inject : enabled
controller:
  enabled: true
dns:
  enabled: true
apiGateway:
  enabled: true
  image: "hashicorp/consul-api-gateway:0.2.0"
  managedGatewayClass:
    serviceType: LoadBalancer
    useHostPorts: true
  1. Installed HelmChart with my values:
helm upgrade --install consul hashicorp/consul -f values.yaml
  1. Created TLS secret
kubectl create secret tls api-gw-cert --cert=api-gw.pem --key=api-gw-key.pem
  1. create consul-api-gateway.yaml with the following content:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
  name: selling-fears-gateway
spec:
  gatewayClassName: consul-api-gateway
  listeners:
  - protocol: HTTPS
    port: 8443
    name: https
    allowedRoutes:
      namespaces:
        from: Same
    tls:
      certificateRefs:
        - name: api-gw-cert
  1. Executed kubectl apply -f consul-api-gateway.yaml

API Gateway Controller and gateway pods are running

❯ kubectl get pods| grep gateway
consul-api-gateway-controller-97c4477d9-gjfct   1/1     Running   3          54m
selling-fears-gateway-68f5fb44-bjpp6            1/1     Running   0          49m

ELB was created for the gateway

❯ kubectl get svc | grep -i gateway
consul-api-gateway-controller   ClusterIP      172.20.91.149    <none>                                                                    9090/TCP                                                                  7h26m
selling-fears-gateway           LoadBalancer   172.20.66.53     abbd7ec5cde7348f4bde6443a180fc0e-605426254.eu-west-1.elb.amazonaws.com    8443:30991/TCP                                                            50m

Gateway resource isn’t ready and doesn’t get address:

❯ kubectl get gateway
NAME                    CLASS                ADDRESS   READY   AGE
selling-fears-gateway   consul-api-gateway             False   52m

Errors in consul-api-gateway-controller

2022-04-28T17:00:14.279Z [ERROR] k8s/logger.go:23: consul-api-gateway-server.controller-runtime.controller.gateway: Reconciler error: name=selling-fears-gateway namespace=default reconciler group=gateway.networking.k8s.io reconciler kind=Gateway
  error=
  | 1 error occurred:
  | 	* Gateway.gateway.networking.k8s.io "selling-fears-gateway" is invalid: status.addresses.value: Invalid value: "": status.addresses.value in body should be at least 1 chars long
  |

No errors in gateway pod

 kubectl logs selling-fears-gateway-68f5fb44-bjpp6 | grep -v info | grep -v Deprecated
{"timestamp":"2022-04-28 16:21:46.505","thread":"13","level":"warning","name":"main","source":"source/server/server.cc:761","message":"there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections"}

Kubernetes version - v1.21.5-eks-9017834
Consul version V1.12.0

Hi! I’m investigating this for you, but don’t have an immediate answer at the moment. Is this your first time setting up the gateway, or were you able to use 0.1.0 without this issue?

2 Likes

This is the first time I’m setting up the gateway. But I tried same configuration with API Gateway image 0.1.0 and the api was in a crash loop due to segmentation fault.

1 Like

Ok! And are you using self signed certificates or something else?

Yes - using self signed. Here they are ( not a real environment :slight_smile: )
Key

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAxI2S+qApXgEXN67fpBOAVs2UwyXnsBMetX7b807OVa5xdQUw
5g9adfz550y6hBlsnO7+kP1xQbHV2/shN5CpmupeUUaqZz21a1z5whMqZUrBssm+
vDxsI/n6Fhl7kXwFi2aGe+OUaxuf853H9OQwtnTDA5HXnGwfbNqS6TMTuJGrnutD
WuYQ9Zz+/dndjjRGXZVRPpvbjy9CjJxL8Ghm3Hza8M8Eqrzl4D1Ek7muykOFKY94
JLrq1cPzuVn2/y3t6ql4sjMQK/fXxdtwh7GySaLJlu+otjkTa4xbHRaQVLLz4iXj
6gBl1MChY9imlSNpjMEbZCIIdDq0Y3lb0Sb02wIDAQABAoIBAFH+KSEp7PfNBq1w
4tRkWjZbvkIGLvdxkm7uA70k08hEZAoH51UhdIAhzvIhPPFcVcXFoSZEw5k/IVKK
GVo/m7EHMd8/1lgJEwQ9nebK7fWhUNpPdKS7o/UCE8RiTvzqurljRJir4D8qH/iV
ilNrWbLTVILJtSMIq7dSGtTzPLfSOZmPp8fZ6FYSsu9S7/xOjQ78jEjjNdiRfe2x
HlQveaV1aacZtehtcvdDeRAQZls+oCIEXP+S6+mPbpTE8yvt3LmA2x3bUBJwEU3v
EexL4UB9urWYTrPxVEaieesb5BlprIQU1HQ24MFQyNYEJXAQ2PwfCeSmqK34XU35
w+aGiQECgYEA4T41ylKHffCbyID7jnG0DnU1j8+mKhoqIVhMkC0uBwVGUCTpPTSs
if2cuDTbx6B9kR4/BpOzGb+KDZA4d0+GhQBk9dMN4BydtcO/T2bsIMbryD2p7GNl
QA8VR7qByAcP7vA2YT0HBGGmxGnicEBLjMlJVfPFgzeKc4qGALz41lsCgYEA32Rz
ub3wSM0mD0yxOAlPJee9sRupwR2a0QnFUN7mdOHLMXgksfg/PYe32h+b972Z/Hc4
JOU443Hnrr4/BhQBtZLz/DF1mw3rCE3l24XKnqnWZCdYB6nKMChsPvDGIz4kQr1k
NTeMLdMV32wZQdGm8d6+pYUue/+URXhVRn6Fo4ECgYEAtVARlrrWbI+Jp14EoUZw
DY9WPVyGwq9rKIpen1RvD6G0VwFPa0CCf1XSmQmbvVc4nN9/FnlAm8Jui7qDaa9v
dpK5spRhP/1pCo726iDMhRn7ZKYWqb3dHDLIC4RbwjvFHK7q511rz6AX0VX2vCtV
ZZAGY5Umchj8b0Ob2O5FVK8CgYEAqdUUpgFgy/grFzaXBKMPKSIldKAzTj3TlVh4
SiMr0XBXqiRMwYCZk426mHmveYkLqIR6ipI5zbCrEo5QG67aHdC67OAtKNRL+uQ9
+8abZER6WWoP4sOSk0ooATHLcL+tkY+qv0qbp7ryxgjIquFYqklNZ0j5LgwADVCO
hClsMAECgYBhJG+rha7fL8x5Ni18yZP7EAOY3AIw8sqgfPmF3ftmpKKWyImFl511
37MRP5Bfg7HtN+2+a++wDmPcjZYiMjUGRUl2bpYBwrdSkK2y4zQ1R4MCnCLLHF2Y
hFbB5AfvXsVNZOUUhF6//hCFqX9x/YiEAgV1eFGTmzI/HY6378BAsA==
-----END RSA PRIVATE KEY-----

Cert

-----BEGIN CERTIFICATE-----
MIIEFzCCAv+gAwIBAgIUYStks0Q8MDhq86/dGWDG0M3TxfcwDQYJKoZIhvcNAQEL
BQAwgYAxCzAJBgNVBAYTAkdCMRAwDgYDVQQIEwdFbmdsYW5kMQ8wDQYDVQQHEwZM
b25kb24xFjAUBgNVBAoTDVNlbGxpbmcgRmVhcnMxFjAUBgNVBAsTDVNlY3VyaXR5
IFRlYW0xHjAcBgNVBAMTFVNlbGxpbmcgRmVhcnMgUm9vdCBDQTAeFw0yMjA0Mjgx
MDEyMDBaFw0zMjA0MjUxMDEyMDBaMHMxCzAJBgNVBAYTAkdCMRAwDgYDVQQIEwdF
bmdsYW5kMQ8wDQYDVQQHEwZMb25kb24xFjAUBgNVBAoTDVNlbGxpbmcgRmVhcnMx
ETAPBgNVBAsTCFNlY3VyaXR5MRYwFAYDVQQDEw1zZWxsaW5nLWZlYXJzMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxI2S+qApXgEXN67fpBOAVs2UwyXn
sBMetX7b807OVa5xdQUw5g9adfz550y6hBlsnO7+kP1xQbHV2/shN5CpmupeUUaq
Zz21a1z5whMqZUrBssm+vDxsI/n6Fhl7kXwFi2aGe+OUaxuf853H9OQwtnTDA5HX
nGwfbNqS6TMTuJGrnutDWuYQ9Zz+/dndjjRGXZVRPpvbjy9CjJxL8Ghm3Hza8M8E
qrzl4D1Ek7muykOFKY94JLrq1cPzuVn2/y3t6ql4sjMQK/fXxdtwh7GySaLJlu+o
tjkTa4xbHRaQVLLz4iXj6gBl1MChY9imlSNpjMEbZCIIdDq0Y3lb0Sb02wIDAQAB
o4GUMIGRMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNV
HRMBAf8EAjAAMB0GA1UdDgQWBBTUZLjJB4wzV6IdNjaBvuz/NUoBWjAfBgNVHSME
GDAWgBQYoMe1EspahdotLX5RIVaLgf4S3TAcBgNVHREEFTATghFzZWxsaW5nLWZl
YXJzLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEARXiZtG/XsdbAl9rOM3U+elzzZ3cT
62SKppX1/J3tygrI8+wl2lYI3sx8QkqVjVYAEQq3br4qY7+I9FJZIJvBdwPxnaaA
ARAS5xJTlnFrr/oxToVO5arAz1627UAEg7Jc4TrmcojCpPplDtCx/pDVFr3+7az3
/HP8ZEtryCHkbhv81029+82fSvf5zsul5AGJTmsei22eBc0VINhPRGieE3yHLj5K
HRfis0DRRL8lX1Wqs1q3jPBfZmmzCYwhVMSxRUHYMXQ4CPAHfCvwHLaNFVuWwdX6
1SExyk+3cNRVgAJTeHTWqNlDLtHXO1Zr1oUDvunEgg7coW87Ro/cl/QT8A==
-----END CERTIFICATE-----
2 Likes

Hi @lev2 I verified your issue, and found what looks like a bug in our handling of gateways that are exposed via LoadBalancer services – it looks like this bug happens to fit the default way that EKS provisions LoadBalancer services (with a hostname rather than an ip address for the service’s ExternalIP).

The fix is here, and we should have a patch release coming shortly. So you can keep eyes over there and I’ll also give you a heads up when the patch release happens.

If this happens to be a cluster you can run with some experimental changes, I pushed a test image with that fix to public.ecr.aws/d1c7c4d0/testing123:1, so if you so desire you can verify that this fixes your issue. Let me know if for some reason it doesn’t. Thanks!

Hi @lev2 ,

We’ve just released v0.2.1 and it has the fix for the bug you ran into. Try installing that and let us know if you encounter any problems or have questions.

Thanks for trying Consul API Gateway. We would love to hear any feedback you might have about it, good, bad or otherwise.

Regards,
Jeff

@lev2 also a few suggestions for trying this out. I noticed the use of useHostPorts: true – generally you should not use that option in any sort of production environment. Using it causes the container ports to be mapped to the same ports on the running worker node (basically using a hostPort-based mapping), rather than doing dynamic port mapping on the node. In most environments this will likely eventually cause pod scheduling issues due to insufficient port availability – so you should probably remove it.

Additionally, I noticed the use of port 8443 for the gateway listener. Since this gateway is going to be fronted by a LoadBalancer instance, you probably just want a standard https port of 443.

I’m assuming both of these values were taken from the learn guide, but those values aren’t fully reflective of a real-life deployment due to deploying on a kind cluster.

Let us know if you have any other issues, especially with the patch release!

@andrewstucki and @Jeff-Apple, first of all thanks a lot for the quick response and fixed version.
I can see that gateway becomes ready now and that address is available.

Regarding learn tutorial I think it should be updated both because now in 0.2.1 you can do cross namespace references and the tutorial deploys consul workload and application workloads in 2 different namespaces ( default and consul ).
Also I think it’s crucial to add to learn tutorial “service_defaults.yaml” with protocol http. It took me a while to understand why my mindless copy paste didn’t work :slight_smile: until I started comparing every file in git to my workloads. For me it wasn’t obvious that service that wasn’t explicitly defined with protocol http won’t work with HTTPRoute resource.