Ingress Gateway readiness check fails, i/o timeout

Consul
,
1

I have a consul setup, which i’ve configured via the helm chart here: consul-k8s/Chart.yaml at main · hashicorp/consul-k8s · GitHub. I’ve included my values below. I’m currently experiencing an issue where the ingress gateway is failing its readiness and liveness probes. The error that k8s shows me is Liveness probe failed: dial tcp 172.27.71.234:21000: i/o timeout however if I cURL that endpoint I receive a 404 from envoy.

$ curl --verbose http://172.27.71.234:21000
*   Trying 172.27.71.234:21000...
* Connected to 172.27.71.234 (172.27.71.234) port 21000 (#0)
> GET / HTTP/1.1
> Host: 172.27.71.234:21000
> User-Agent: curl/7.77.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< date: Tue, 13 Dec 2022 19:21:18 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host 172.27.71.234 left intact

I’ve tried enabling debug logs but they don’t shed any light on why the probes would be failing. Any thoughts?

global:
  enabled: true
  name: t-consul-use1
  datacenter: dc1
  gossipEncryption:
    autoGenerate: true
  tls:
    enabled: true
    enableAutoEncrypt: true
    verify: true
  acls:
    manageSystemACLs: true
  metrics:
    enabled: true
    enableAgentMetrics: true
server:
  enabled: true
  replicas: 3
  connect: true
  resources:
    requests:
      memory: "512Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "100m"
ui:
  enabled: true
  service:
    type: LoadBalancer
connectInject:
  enabled: true
  default: false
  failurePolicy: "Ignore"
client:
  enable: true
ingressGateways:
  enabled: true
  service: LoadBalancer