Cannot configure Consul API gateway to listen on kubernetes Node port

Hello,

I’m trying to configure an API gateway on our Kubernetes cluster and expose the port on each node. The gateway is deployed with HTTPRoutes and everything seems running fine in Kubernetes, except that I cannot reach it.
I’m not sure about it, but I cannot see the configured port passed as a parameter and it seems suspicious to me. Inside the container, there is nothing listening on the configured port.

Consul is installed via Helm and here is the resulting Pod yaml:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: <containerID>
    cni.projectcalico.org/podIP: 10.42.2.230/32
    cni.projectcalico.org/podIPs: 10.42.2.230/32
    consul.hashicorp.com/connect-inject: "false"
    kubernetes.io/psp: global-unrestricted-psp
  creationTimestamp: "2022-10-21T08:54:00Z"
  generateName: <generated name>
  labels:
    api-gateway.consul.hashicorp.com/created: "1666342440"
    api-gateway.consul.hashicorp.com/managed: "true"
    api-gateway.consul.hashicorp.com/name: my-gateway
    api-gateway.consul.hashicorp.com/namespace: my-namespace
    pod-template-hash: 6c899464b8
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:cni.projectcalico.org/containerID: {}
          f:cni.projectcalico.org/podIP: {}
          f:cni.projectcalico.org/podIPs: {}
    manager: Go-http-client
    operation: Update
    subresource: status
    time: "2022-10-21T08:54:00Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    manager: kubelet
    operation: Update
    subresource: status
    time: "2022-10-24T09:44:27Z"
  name: <Pod name>
  namespace: my-namespace
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: <ReplicaSet name>
    uid: 94137bc4-dc7a-456a-a76f-10b64fcb8f33
  resourceVersion: "51565801"
  uid: ddd9f032-eb01-4e50-937f-21501aece168
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchLabels:
              api-gateway.consul.hashicorp.com/created: "1666342440"
              api-gateway.consul.hashicorp.com/managed: "true"
              api-gateway.consul.hashicorp.com/name: my-gateway
              api-gateway.consul.hashicorp.com/namespace: my-namespace
          topologyKey: kubernetes.io/hostname
        weight: 1
  containers:
  - command:
    - /bin/sh
    - -ec
    - |2-

      export CONSUL_CACERT=/consul/tls/ca.pem
      cat <<EOF >/consul/tls/ca.pem
      -----BEGIN CERTIFICATE-----
      <certificate>
      -----END CERTIFICATE-----

      EOF

      exec /bootstrap/consul-api-gateway exec -log-json \
        -log-level info \
        -gateway-host "$(IP)" \
        -gateway-name my-gateway \
        -consul-http-address $(HOST_IP) \
        -consul-http-port 8501 \
        -consul-xds-port  8502 \
        -envoy-bootstrap-path /bootstrap/envoy.json \
        -envoy-sds-address consul-api-gateway-controller.consul.svc \
        -envoy-sds-port 9090
    env:
    - name: IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    - name: HOST_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.hostIP
    image: envoyproxy/envoy:v1.23.1
    imagePullPolicy: IfNotPresent
    name: consul-api-gateway
    ports:
    - containerPort: 20000
      name: ready
      protocol: TCP
    - containerPort: 1443
      hostPort: 1443
      name: my-port
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /ready
        port: 20000
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /bootstrap
      name: bootstrap
    - mountPath: /certs
      name: certs
    - mountPath: /consul/tls
      name: ca
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-t6d8x
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - cp
    - /bin/consul-api-gateway
    - /bootstrap/consul-api-gateway
    image: hashicorp/consul-api-gateway:0.4.0
    imagePullPolicy: IfNotPresent
    name: consul-api-gateway-init
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /bootstrap
      name: bootstrap
    - mountPath: /certs
      name: certs
    - mountPath: /consul/tls
      name: ca
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-t6d8x
      readOnly: true
  nodeName: <kubernetes node>
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: bootstrap
  - emptyDir: {}
    name: certs
  - emptyDir: {}
    name: ca
  - name: kube-api-access-t6d8x
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-10-21T12:45:20Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-10-24T09:44:28Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-10-24T09:44:28Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-10-21T08:54:00Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://891a23829e977d64af2033ebe99f666e5711cef637f3c7558de13b7a831d2936
    image: docker.io/envoyproxy/envoy:v1.23.1
    imageID: docker.io/envoyproxy/envoy@sha256:9a1a68fc2adbc43896977bcff1333b3942a82af6157c90857be0fb33a5bf6e69
    lastState:
      terminated:
        containerID: containerd://375a9040925f46e8c4bc770d42d616052f04e3b82486e25b5d10e670942bb582
        exitCode: 1
        finishedAt: "2022-10-24T09:39:18Z"
        reason: Error
        startedAt: "2022-10-24T09:38:48Z"
    name: consul-api-gateway
    ready: true
    restartCount: 57
    started: true
    state:
      running:
        startedAt: "2022-10-24T09:44:27Z"
  hostIP: 192.168.9.31
  initContainerStatuses:
  - containerID: containerd://a6b17cc178528f20b7e2dd69efd2fb6fac56ee9fefa392cc5ebde7e8d46e87ed
    image: docker.io/hashicorp/consul-api-gateway:0.4.0
    imageID: docker.io/hashicorp/consul-api-gateway@sha256:d37e0db75715c69b5684a3bf2ff7375ab34709ccd7733ebc8cffe2cfeeb4a1ca
    lastState: {}
    name: consul-api-gateway-init
    ready: true
    restartCount: 5
    state:
      terminated:
        containerID: containerd://a6b17cc178528f20b7e2dd69efd2fb6fac56ee9fefa392cc5ebde7e8d46e87ed
        exitCode: 0
        finishedAt: "2022-10-24T08:48:48Z"
        reason: Completed
        startedAt: "2022-10-24T08:48:48Z"
  phase: Running
  podIP: 10.42.2.230
  podIPs:
  - ip: 10.42.2.230
  qosClass: BestEffort
  startTime: "2022-10-21T08:54:00Z"

We can see that the port I want to use is declared (1443), but the command launching the gateway does not reference this port, so I don’t see how the app could be listening on that port.

Here is my gateway conf:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    api-gateway.consul.hashicorp.com/config: '{"serviceType":"NodePort","useHostPorts":true,"consul":{"authentication":{},"scheme":"https","ports":{"http":8501,"grpc":8502}},"image":{"consulAPIGateway":"hashicorp/consul-api-gateway:0.4.0","envoy":"envoyproxy/envoy:v1.23.1"},"copyAnnotations":{},"logLevel":"info","deployment":{"defaultInstances":1,"maxInstances":8,"minInstances":1}}'
    objectset.rio.cattle.io/applied: <data>
    objectset.rio.cattle.io/id: dd05420f-ddb6-4765-b2ff-bf8c0a3ae453
  creationTimestamp: "2022-10-21T08:54:00Z"
  generation: 1
  labels:
    objectset.rio.cattle.io/hash: c9df638a206ff7e4392cecd735052d40e453af79
    manager: consul-api-gateway
    operation: Update
    subresource: status
    time: "2022-10-21T08:54:02Z"
  name: my-gateway
  namespace: my-namespace
  resourceVersion: "51800801"
  uid: dcc61944-07e6-4d92-b4cd-85487dedb07a
spec:
  gatewayClassName: consul-api-gateway
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    hostname: <my external hostname>
    name: my-app
    port: 1443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: <secret name>
        namespace: my-namespace
      mode: Terminate
status:
  addresses:
  - type: IPAddress
    value: <node 1 IP>
  - type: IPAddress
    value: <node 2 IP>
  conditions:
  - lastTransitionTime: "2022-10-24T15:42:06Z"
    message: Ready
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  - lastTransitionTime: "2022-10-24T15:42:06Z"
    message: Scheduled
    observedGeneration: 1
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-10-24T15:42:06Z"
    message: InSync
    observedGeneration: 1
    reason: InSync
    status: "True"
    type: InSync
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: NoConflicts
      observedGeneration: 1
      reason: NoConflicts
      status: "False"
      type: Conflicted
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: Attached
      observedGeneration: 1
      reason: Attached
      status: "False"
      type: Detached
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: Ready
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: Ready
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: ResolvedRefs
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: my-app
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

Am I on the right way ?

Hi @val :wave:

I don’t see it included here, but I’m assuming that you have the service type that Consul API Gateway is using configured in your values.yaml file when installing the Consul Helm chart, something like this:

...
apiGateway:
  enabled: true
  image: "hashicorp/consul-api-gateway:0.4"
  managedGatewayClass:
    serviceType: NodePort

With the above config, the Consul API Gateway controller will create a Kubernetes Service with type: NodePort (docs) for each Gateway that you create using the built in consul-api-gateway config class.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: api-gateway
spec:
  gatewayClassName: consul-api-gateway
  listeners:
  - protocol: HTTPS
    port: 8443
    name: https
    allowedRoutes:
      namespaces:
        from: Same
    tls:
      certificateRefs:
        - name: consul-server-cert

Given the above example, I can go and see the corresponding Kubernetes Service and the node port that it’s assigned. This Service is what maps from the port on the Node to the port that you see exposed on the Pod in your original post. The Service is named the same thing as the Gateway that it corresponds with.

$ kubectl apply --filename gateway.yaml
...

# After waiting for the controller to reconcile the above
$ kubectl get service api-gateway --output yaml
apiVersion: v1
kind: Service
metadata:
  name: api-gateway
  ...
spec:
  ...
  ports:
  - name: https
    nodePort: 31001
    port: 8443
    protocol: TCP
    targetPort: 8443
  selector:
    api-gateway.consul.hashicorp.com/created: "1666639998"
    api-gateway.consul.hashicorp.com/managed: "true"
    api-gateway.consul.hashicorp.com/name: api-gateway
    api-gateway.consul.hashicorp.com/namespace: consul
  ...

The value at spec.ports[].nodePort is the port that will be open on the node and routing to the API gateway. I can test this by attaching an HTTPRoute that echos some information and curling the node’s IP address (seen on the Gateway in your original post) with that nodePort from inside my container:

$ curl --insecure --silent https://172.18.0.2:31001/echo
{
  ...
  "path": "/echo",
  "namespace": "default",
  "service": "echo-2",
  "pod": "echo-2-596645f7f5-ddclt"
}

In the docs that I linked above for NodePort Services, you can see that Kubernetes automatically assigns a port in a specific range, so it will necessarily differ from the port specified on your Gateway. Kubernetes does offer the ability to specify a port within that specific range; however, we don’t support that today.

Hopefully this helps you out. I imagine you can already find the Service named my-gateway with your existing setup and curl your node(s) on the nodePort that you find there to access the services that you’ve attached with HTTPRoute(s). Let me know where this gets you and if I can provide more helpful information.

Cheers,
Nathan

Thanks Nathan for your answer !

Yes I have all that and I actually have a service listening on node port 30059. But it does not work on this port either.
I already tried to do a curl inside the gateway pod on port 1443 and it prints a connection failed as well… That’s why I was wondering how the consul-api-gateway script executed in the container could launch everything and listen on that port, when there’s actually no argument having this port number. We just tell k8s to listen on that port, but if there’s no process listening on that port, I don’t see how it could work! But I cannot find out what is missing in my configuration to produce that.

Here is my Helm chart yaml:

apiGateway:
  controller:
    annotations: null
    nodeSelector: null
    priorityClassName: ''
    replicas: 2
    service:
      annotations: null
  enabled: true
  image: hashicorp/consul-api-gateway:0.4.0
  initCopyConsulContainer:
    resources:
      limits:
        cpu: 50m
        memory: 150Mi
      requests:
        cpu: 50m
        memory: 25Mi
  logLevel: trace
  managedGatewayClass:
    copyAnnotations:
      service: null
    deployment: null
    enabled: true
    nodeSelector: null
    serviceType: NodePort
    useHostPorts: true
  resources:
    limits:
      cpu: 100m
      memory: 100Mi
    requests:
      cpu: 100m
      memory: 100Mi
  serviceAccount:
    annotations: null
client:
  affinity: null
  annotations: null
  containerSecurityContext:
    aclInit: null
    client: null
    tlsInit: null
  dataDirectoryHostPath: null
  dnsPolicy: null
  enabled: '-'
  exposeGossipPorts: false
  extraConfig: |
    {}
  extraContainers: null
  extraEnvironmentVars: {}
  extraLabels: null
  extraVolumes: null
  grpc: true
  hostNetwork: false
  image: null
  join: null
  nodeMeta:
    host-ip: ${HOST_IP}
    pod-name: ${HOSTNAME}
  nodeSelector: null
  priorityClassName: ''
  resources:
    limits:
      cpu: 100m
      memory: 100Mi
    requests:
      cpu: 100m
      memory: 100Mi
  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    runAsUser: 100
  serviceAccount:
    annotations: null
  snapshotAgent:
    caCert: null
    configSecret:
      secretKey: null
      secretName: null
    enabled: false
    interval: 1h
    replicas: 2
    resources:
      limits:
        cpu: 50m
        memory: 50Mi
      requests:
        cpu: 50m
        memory: 50Mi
    serviceAccount:
      annotations: null
  tolerations: ''
  updateStrategy: null
connectInject:
  aclBindingRuleSelector: serviceaccount.name!=default
  aclInjectToken:
    secretKey: null
    secretName: null
  affinity: null
  annotations: null
  cni:
    cniBinDir: /opt/cni/bin
    cniNetDir: /etc/cni/net.d
    enabled: false
    logLevel: null
    multus: false
    resourceQuota:
      pods: 5000
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 75m
        memory: 75Mi
    securityContext:
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
    updateStrategy: null
  consulNamespaces:
    consulDestinationNamespace: default
    mirroringK8S: false
    mirroringK8SPrefix: ''
  default: false
  disruptionBudget:
    enabled: true
    maxUnavailable: null
  enabled: true
  envoyExtraArgs: null
  failurePolicy: Fail
  image: null
  imageConsul: null
  initContainer:
    resources:
      limits:
        cpu: 50m
        memory: 150Mi
      requests:
        cpu: 50m
        memory: 25Mi
  k8sAllowNamespaces:
    - '*'
  k8sDenyNamespaces: null
  logLevel: ''
  metrics:
    defaultEnableMerging: false
    defaultEnabled: '-'
    defaultMergedMetricsPort: 20100
    defaultPrometheusScrapePath: /metrics
    defaultPrometheusScrapePort: 20200
  namespaceSelector: |
    matchExpressions:
      - key: "kubernetes.io/metadata.name"
        operator: "NotIn"
        values: ["kube-system","local-path-storage","longhorn-system","cattle-fleet-clusters-system","cattle-fleet-local-system","cattle-fleet-system","cattle-global-data","cattle-global-nt","cattle-impersonation-system","cattle-system","fleet-default","fleet-local"]
  nodeSelector: null
  overrideAuthMethodName: ''
  priorityClassName: ''
  replicas: 2
  resources:
    limits:
      cpu: 50m
      memory: 50Mi
    requests:
      cpu: 50m
      memory: 50Mi
  serviceAccount:
    annotations: null
  sidecarProxy:
    concurrency: 2
    resources:
      limits:
        cpu: null
        memory: null
      requests:
        cpu: null
        memory: null
  tolerations: null
  transparentProxy:
    defaultEnabled: true
    defaultOverwriteProbes: true
controller:
  aclToken:
    secretKey: null
    secretName: null
  affinity: null
  enabled: true
  logLevel: ''
  nodeSelector: null
  priorityClassName: ''
  replicas: 1
  resources:
    limits:
      cpu: 100m
      memory: 50Mi
    requests:
      cpu: 100m
      memory: 50Mi
  serviceAccount:
    annotations: null
  tolerations: null
dns:
  additionalSpec: null
  annotations: null
  clusterIP: null
  enableRedirection: false
  enabled: '-'
  type: ClusterIP
externalServers:
  enabled: false
  grpcPort: 8503
  hosts: null
  httpsPort: 8501
  k8sAuthMethodHost: null
  tlsServerName: null
  useSystemRoots: false
global:
  acls:
    bootstrapToken:
      secretKey: null
      secretName: null
    createReplicationToken: false
    manageSystemACLs: false
    partitionToken:
      secretKey: null
      secretName: null
    replicationToken:
      secretKey: null
      secretName: null
  adminPartitions:
    enabled: false
    name: default
    service:
      annotations: null
      nodePort:
        https: null
        rpc: null
        serf: null
      type: LoadBalancer
  consulAPITimeout: 5s
  consulSidecarContainer:
    resources:
      limits:
        cpu: 20m
        memory: 50Mi
      requests:
        cpu: 20m
        memory: 25Mi
  datacenter: dc1
  domain: consul
  enableConsulNamespaces: false
  enablePodSecurityPolicies: false
  enabled: true
  enterpriseLicense:
    enableLicenseAutoload: true
    secretKey: null
    secretName: null
  federation:
    createFederationSecret: false
    enabled: false
    k8sAuthMethodHost: null
    primaryDatacenter: null
    primaryGateways: null
  gossipEncryption:
    autoGenerate: false
    secretKey: ''
    secretName: ''
  image: hashicorp/consul:1.13.2
  imageEnvoy: envoyproxy/envoy:v1.23.1
  imageK8S: hashicorp/consul-k8s-control-plane:0.49.0
  imagePullSecrets: null
  logJSON: false
  logLevel: info
  metrics:
    agentMetricsRetentionTime: 1m
    enableAgentMetrics: false
    enableGatewayMetrics: true
    enabled: false
  name: consul
  openshift:
    enabled: false
  peering:
    enabled: false
    tokenGeneration:
      serverAddresses:
        source: ''
        static: null
  recursors: null
  secretsBackend:
    vault:
      adminPartitionsRole: ''
      agentAnnotations: null
      ca:
        secretKey: null
        secretName: connect-inject/cert/ca
      connectCA:
        additionalConfig: |
          {}
        address: http://vault.common.svc.cluster.local:8200
        authMethodPath: kubernetes
        intermediatePKIPath: connect-intermediate-dc1/
        rootPKIPath: connect-root/
      connectInject:
        caCert:
          secretName: connect-inject/cert/ca
        tlsCert:
          secretName: connect-inject/issue/connect-inject
      connectInjectRole: connect-inject
      consulCARole: consul-ca
      consulClientRole: consul-client
      consulServerRole: consul-server
      consulSnapshotAgentRole: ''
      controller:
        caCert:
          secretName: controller/cert/ca
        tlsCert:
          secretName: controller/issue/controller
      controllerRole: controller
      enabled: true
      manageSystemACLsRole: ''
  tls:
    caCert:
      secretKey: null
      secretName: pki/cert/ca
    caKey:
      secretKey: null
      secretName: null
    enableAutoEncrypt: true
    enabled: true
    httpsOnly: true
    serverAdditionalDNSSANs: null
    serverAdditionalIPSANs: null
    verify: true
  systemDefaultRegistry: ''
ingressGateways:
  defaults:
    affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: {{ template "consul.name" . }}
                release: "{{ .Release.Name }}"
                component: ingress-gateway
            topologyKey: kubernetes.io/hostname
    annotations: null
    consulNamespace: default
    initCopyConsulContainer:
      resources:
        limits:
          cpu: 50m
          memory: 150Mi
        requests:
          cpu: 50m
          memory: 25Mi
    nodeSelector: null
    priorityClassName: ''
    replicas: 2
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 100m
        memory: 100Mi
    service:
      additionalSpec: null
      annotations: null
      ports:
        - nodePort: null
          port: 8080
        - nodePort: null
          port: 8443
      type: ClusterIP
    serviceAccount:
      annotations: null
    terminationGracePeriodSeconds: 10
    tolerations: null
    topologySpreadConstraints: ''
  enabled: false
  gateways:
    - name: ingress-gateway
meshGateway:
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "consul.name" . }}
              release: "{{ .Release.Name }}"
              component: mesh-gateway
          topologyKey: kubernetes.io/hostname
  annotations: null
  consulServiceName: mesh-gateway
  containerPort: 8443
  dnsPolicy: null
  enabled: false
  hostNetwork: false
  hostPort: null
  initCopyConsulContainer:
    resources:
      limits:
        cpu: 50m
        memory: 150Mi
      requests:
        cpu: 50m
        memory: 25Mi
  initServiceInitContainer:
    resources:
      limits:
        cpu: 50m
        memory: 50Mi
      requests:
        cpu: 50m
        memory: 50Mi
  nodeSelector: null
  priorityClassName: ''
  replicas: 2
  resources:
    limits:
      cpu: 100m
      memory: 100Mi
    requests:
      cpu: 100m
      memory: 100Mi
  service:
    additionalSpec: null
    annotations: null
    enabled: true
    nodePort: null
    port: 443
    type: LoadBalancer
  serviceAccount:
    annotations: null
  tolerations: null
  topologySpreadConstraints: ''
  wanAddress:
    port: 443
    source: Service
    static: ''
prometheus:
  enabled: false
server:
  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "consul.name" . }}
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: kubernetes.io/hostname
  annotations: null
  bootstrapExpect: null
  connect: true
  containerSecurityContext:
    server: null
  disruptionBudget:
    enabled: true
    maxUnavailable: null
  enabled: '-'
  exposeGossipAndRPCPorts: false
  exposeService:
    annotations: null
    enabled: '-'
    nodePort:
      grpc: null
      http: null
      https: null
      rpc: null
      serf: null
    type: LoadBalancer
  extraConfig: |
    {}
  extraContainers: null
  extraEnvironmentVars: {}
  extraLabels: null
  extraVolumes:
    - load: 'false'
      name: vault-ca
      type: secret
  image: null
  nodeSelector: null
  ports:
    serflan:
      port: 8301
  priorityClassName: ''
  replicas: 2
  resources:
    limits:
      cpu: 100m
      memory: 100Mi
    requests:
      cpu: 100m
      memory: 100Mi
  securityContext:
    fsGroup: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    runAsUser: 100
  serverCert:
    secretName: pki/issue/consul-server
  service:
    annotations: null
  serviceAccount:
    annotations: null
  storage: 10Gi
  storageClass: null
  tolerations: ''
  topologySpreadConstraints: ''
  updatePartition: 0
syncCatalog:
  aclSyncToken:
    secretKey: null
    secretName: null
  addK8SNamespaceSuffix: false
  affinity: null
  annotations: null
  consulNamespaces:
    consulDestinationNamespace: default
    mirroringK8S: false
    mirroringK8SPrefix: ''
  consulNodeName: k8s-sync
  consulPrefix: null
  consulWriteInterval: null
  default: true
  enabled: true
  extraLabels: null
  image: null
  k8sAllowNamespaces:
    - '*'
  k8sDenyNamespaces:
    - kube-system
    - kube-public
    - longhorn-system
    - cattle-*
  k8sPrefix: null
  k8sSourceNamespace: null
  k8sTag: null
  logLevel: ''
  nodePortSyncType: ExternalFirst
  nodeSelector: null
  priorityClassName: ''
  resources:
    limits:
      cpu: 50m
      memory: 50Mi
    requests:
      cpu: 50m
      memory: 50Mi
  serviceAccount:
    annotations: null
  syncClusterIPServices: true
  toConsul: true
  toK8S: true
  tolerations: null
terminatingGateways:
  defaults:
    affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: {{ template "consul.name" . }}
                release: "{{ .Release.Name }}"
                component: terminating-gateway
            topologyKey: kubernetes.io/hostname
    annotations: null
    consulNamespace: default
    extraVolumes: null
    initCopyConsulContainer:
      resources:
        limits:
          cpu: 50m
          memory: 150Mi
        requests:
          cpu: 50m
          memory: 25Mi
    nodeSelector: null
    priorityClassName: ''
    replicas: 2
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 100m
        memory: 100Mi
    serviceAccount:
      annotations: null
    tolerations: null
    topologySpreadConstraints: ''
  enabled: false
  gateways:
    - name: terminating-gateway
tests:
  enabled: true
ui:
  dashboardURLTemplates:
    service: ''
  enabled: '-'
  ingress:
    annotations: null
    enabled: false
    hosts: null
    ingressClassName: ''
    pathType: Prefix
    tls: null
  metrics:
    baseURL: http://prometheus-server
    enabled: '-'
    provider: prometheus
  service:
    additionalSpec: null
    annotations: null
    enabled: true
    nodePort:
      http: null
      https: null
    port:
      http: 80
      https: 443
    type: null
webhookCertManager:
  tolerations: null

And does this actually means that we can manually change the nodePort, but it will be overridden at each chart update?

Hi @val . The Service with type NodePort is what handles forwarding requests to a node on port 30059 over to the gateway pod on port 1443. It sounds like the Service is being created in your setup.

The next thing I would look at is whether the HTTPRoute is successfully attaching to the Gateway and is successfully resolving the services that it’s pointing to.

You can check the status of the routes themselves using

$ kubectl get httproute <name> --output yaml

and then the status of the gateway using

$ kubectl get gateway <name> --output yaml

On the gateway’s status, you should see some information about the listeners and a number of attached routes. Based on those statuses, does it appear everything is healthy?

Nathan

Everything seams alright on that side, here are my routes:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  creationTimestamp: "2022-10-21T08:48:51Z"
  generation: 1
  name: my-route-1
  namespace: my-nalmespace
  resourceVersion: "51563760"
  uid: 1388a4df-020b-49b3-ad3d-fd7f766448c8
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: my-gateway
    sectionName: my-app
  rules:
  - backendRefs:
    - group: api-gateway.consul.hashicorp.com
      kind: MeshService
      name: <Consul service name>
      port: 8090
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: <url prefix>
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: <url prefix>
status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: Route accepted.
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: ResolvedRefs
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: hashicorp.com/consul-api-gateway-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: my-gateway
      sectionName: my-app

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"name":"my-route-1","namespace":"my-namespace"},"spec":{"parentRefs":[{"name":"my-gateway","sectionName":"my-app"}],"rules":[{"backendRefs":[{"group":"api-gateway.consul.hashicorp.com","kind":"MeshService","name":"<Consul service name>","port":8080}],"filters":[{"type":"URLRewrite","urlRewrite":{"path":{"replacePrefixMatch":"<url prefix>","type":"ReplacePrefixMatch"}}}],"matches":[{"path":{"type":"PathPrefix","value":"<url prefix>"}}]}]}}
  creationTimestamp: "2022-10-20T11:50:39Z"
  generation: 1
  name: my-route-2
  namespace: my-namespace
  resourceVersion: "48980966"
  uid: 9b6b6a8b-7aa1-400b-b362-2ed18b1be660
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: my-gateway
    sectionName: my-app
  rules:
  - backendRefs:
    - group: api-gateway.consul.hashicorp.com
      kind: MeshService
      name: <Consul service name>
      port: 8080
      weight: 1
    filters:
    - type: URLRewrite
      urlRewrite:
        path:
          replacePrefixMatch: <url prefix>
          type: ReplacePrefixMatch
    matches:
    - path:
        type: PathPrefix
        value: <url prefix>
status:
  parents:
  - conditions:
    - lastTransitionTime: "2022-10-21T13:44:48Z"
      message: Route accepted.
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2022-10-21T13:44:48Z"
      message: ResolvedRefs
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: hashicorp.com/consul-api-gateway-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: my-gateway
      sectionName: my-app

And here is my gateway:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  annotations:
    api-gateway.consul.hashicorp.com/config: '{"serviceType":"NodePort","useHostPorts":true,"consul":{"authentication":{},"scheme":"https","ports":{"http":8501,"grpc":8502}},"image":{"consulAPIGateway":"hashicorp/consul-api-gateway:0.4.0","envoy":"envoyproxy/envoy:v1.23.1"},"copyAnnotations":{},"logLevel":"info","deployment":{"defaultInstances":1,"maxInstances":8,"minInstances":1}}'
    objectset.rio.cattle.io/applied: H4sIAAAAAAAA/3yRzY7bMAyE34Vn2/X6Jz+69tCeimIT9FLsgZaoRI0sChKToAj87oWSxW7RAnukhvo4Q94Ao/tBKTsOoOCAQlf83QSSK6eTC4fmtMmN40+Xp4kEn6CCkwsGFHx5tEIFMwkaFAR1AwyBBcVxyKXk6RdpySRNctxoFPFUaK4QjGnHoWttbcy0qof1aqynztp6shvdYo80jD0sFXicyH+IO2I+ggK9NXbVb7BrV9auaei3nSZt1v3Yjp0Z2gJEu94WaMCZQIEk1FQbutSHtzxFyhF10aMUsXzIkXTx8Nr32WPO3x4QzSGffY3R/UXxLgsFShnUzxug93wl88xnoXuUtyH3yiaeQcGu8JalgiNneXVo6NLcXTYxkSHvm5jcBf5PABVETgLqaRj6CmJiYc0eFHzd77/voAJ5bFFTEmedRqFnsg9/h8TnCAre77sjnUje53zo4991vVQwsykve0qzCygl18uy/AkAAP//yQXMrnACAAA
    objectset.rio.cattle.io/id: dd05420f-ddb6-4765-b2ff-bf8c0a3ae453
  creationTimestamp: "2022-10-21T08:54:00Z"
  generation: 1
  labels:
    objectset.rio.cattle.io/hash: c9df638a206ff7e4392cecd735052d40e453af79
  name: my-gateway
  namespace: my-namespace
  resourceVersion: "53605272"
  uid: dcc61944-07e6-4d92-b4cd-85487dedb07a
spec:
  gatewayClassName: consul-api-gateway
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    hostname: <my host>
    name: my-app
    port: 1443
    protocol: HTTPS
    tls:
      certificateRefs:
      - group: ""
        kind: Secret
        name: <SSL certificate secret name>
        namespace: my-namespace
      mode: Terminate
status:
  addresses:
  - type: IPAddress
    value: 192.168.9.32
  - type: IPAddress
    value: 192.168.9.31
  conditions:
  - lastTransitionTime: "2022-10-26T13:48:06Z"
    message: Ready
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  - lastTransitionTime: "2022-10-26T13:48:06Z"
    message: Scheduled
    observedGeneration: 1
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2022-10-26T13:48:06Z"
    message: InSync
    observedGeneration: 1
    reason: InSync
    status: "True"
    type: InSync
  listeners:
  - attachedRoutes: 2
    conditions:
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: NoConflicts
      observedGeneration: 1
      reason: NoConflicts
      status: "False"
      type: Conflicted
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: Attached
      observedGeneration: 1
      reason: Attached
      status: "False"
      type: Detached
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: Ready
      observedGeneration: 1
      reason: Ready
      status: "True"
      type: Ready
    - lastTransitionTime: "2022-10-24T09:41:23Z"
      message: ResolvedRefs
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    name: my-app
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute

@nathancoleman Do you have any other idea on what could go wrong in my case?

Hi @val ,
Apologies for the delay. I was just looking at the routes you shared and noticed you’re using a MeshService backend. Any chance you’ve checked the logs for the consul-api-gateway-controller pod? There was a mixing RBAC rule in the Consul Helm chart that I just added recently here but that hasn’t been released yet. I’m thinking you might see some error logs in the consul-api-gateway-controller saying it can’t read the MeshService resource, or if there might be any other helpful logs there.

Hi @nathancoleman,

I’ve effectively encountered the issue you’ve fixed with MeshServices. I already manually added the RBAC rule after looking at logs in the consul-api-gateway and it was running ok after that.
I’ve found something that might be useful in logs (even though I spent a little bit of time into that before, but I missed it).

Here is the error :

"type": "InSync",
| "status": "False",
| "observedGeneration": 1,
| "lastTransitionTime": "2022-11-02T15:01:06Z",
| "reason": "SyncError",
| "message": "error adding service router config entries: 1 error occurred:\n\t* Unexpected response code: 500 (rpc error making call: rpc error making call: discovery chain \"my-app-gateway-603e9e5\" uses inconsistent protocols; service \"my-app\" has \"tcp\" which is not \"http\")\n\n"
| } 

In my service I have TCP, but I cannot set it to HTTP anyway. My application gateway is configured to run on protocol HTTPS… I don’t see what configuration does not match :sweat_smile: And my-app-gateway-603e9e5 does not match any pod or anything I’ve looked at, I’ve no idea about what it could be. Pods are my-app-gateway-6c899464b8-2xgnw and my-app-gateway-6c899464b8-lxn7n

1 Like

That log is super helpful @val , thanks!

What you’re running into is that the ServiceConfigEntry created by the API Gateway controller here has its protocol set to http. You’ll need to change the protocol for your service, my-app, to http to match the upstream service by creating a ServiceDefaults for it (docs).

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: my-app
  namespace: <namespace>
spec:
  protocol: http

For our own Learn guides, we create this alongside the K8s Service, such as here. Unfortunately, I don’t see it clearly called out in any documentation that this is a requirement when using HTTPRoutes. I’ll work on getting this added to the docs.

Thank you @nathancoleman for your answer !

Sorry, I’ve another issue that is probably not linked (probably certificate issuing with Vault), I will open another thread on that subject.

I’ve implemented what you told me, I’ll let you know if it solves the issue when everything is sorted. I’ll let you know !

1 Like

Thank you @nathancoleman, it works !

I had another issue after that, I received no certificate when accessing the API. After extending allowed node port and restarting the Kubernetes cluster, everything was working :slight_smile:

I’m looking forward your patch for meshservices so I can edit the helm chart without manually setting the right on the ClusterRole :wink:

Hi @val , dropping by to let you know that the MeshService patch is now available in versions 0.49.1 and 1.0.1 of the Consul Helm chart