Hello,
I’m trying to configure an API gateway on our Kubernetes cluster and expose the port on each node. The gateway is deployed with HTTPRoutes and everything seems running fine in Kubernetes, except that I cannot reach it.
I’m not sure about it, but I cannot see the configured port passed as a parameter and it seems suspicious to me. Inside the container, there is nothing listening on the configured port.
Consul is installed via Helm and here is the resulting Pod yaml:
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/containerID: <containerID>
cni.projectcalico.org/podIP: 10.42.2.230/32
cni.projectcalico.org/podIPs: 10.42.2.230/32
consul.hashicorp.com/connect-inject: "false"
kubernetes.io/psp: global-unrestricted-psp
creationTimestamp: "2022-10-21T08:54:00Z"
generateName: <generated name>
labels:
api-gateway.consul.hashicorp.com/created: "1666342440"
api-gateway.consul.hashicorp.com/managed: "true"
api-gateway.consul.hashicorp.com/name: my-gateway
api-gateway.consul.hashicorp.com/namespace: my-namespace
pod-template-hash: 6c899464b8
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:cni.projectcalico.org/containerID: {}
f:cni.projectcalico.org/podIP: {}
f:cni.projectcalico.org/podIPs: {}
manager: Go-http-client
operation: Update
subresource: status
time: "2022-10-21T08:54:00Z"
- apiVersion: v1
fieldsType: FieldsV1
manager: kubelet
operation: Update
subresource: status
time: "2022-10-24T09:44:27Z"
name: <Pod name>
namespace: my-namespace
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: <ReplicaSet name>
uid: 94137bc4-dc7a-456a-a76f-10b64fcb8f33
resourceVersion: "51565801"
uid: ddd9f032-eb01-4e50-937f-21501aece168
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
api-gateway.consul.hashicorp.com/created: "1666342440"
api-gateway.consul.hashicorp.com/managed: "true"
api-gateway.consul.hashicorp.com/name: my-gateway
api-gateway.consul.hashicorp.com/namespace: my-namespace
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- command:
- /bin/sh
- -ec
- |2-
export CONSUL_CACERT=/consul/tls/ca.pem
cat <<EOF >/consul/tls/ca.pem
-----BEGIN CERTIFICATE-----
<certificate>
-----END CERTIFICATE-----
EOF
exec /bootstrap/consul-api-gateway exec -log-json \
-log-level info \
-gateway-host "$(IP)" \
-gateway-name my-gateway \
-consul-http-address $(HOST_IP) \
-consul-http-port 8501 \
-consul-xds-port 8502 \
-envoy-bootstrap-path /bootstrap/envoy.json \
-envoy-sds-address consul-api-gateway-controller.consul.svc \
-envoy-sds-port 9090
env:
- name: IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
image: envoyproxy/envoy:v1.23.1
imagePullPolicy: IfNotPresent
name: consul-api-gateway
ports:
- containerPort: 20000
name: ready
protocol: TCP
- containerPort: 1443
hostPort: 1443
name: my-port
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /ready
port: 20000
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /bootstrap
name: bootstrap
- mountPath: /certs
name: certs
- mountPath: /consul/tls
name: ca
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-t6d8x
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
initContainers:
- command:
- cp
- /bin/consul-api-gateway
- /bootstrap/consul-api-gateway
image: hashicorp/consul-api-gateway:0.4.0
imagePullPolicy: IfNotPresent
name: consul-api-gateway-init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /bootstrap
name: bootstrap
- mountPath: /certs
name: certs
- mountPath: /consul/tls
name: ca
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-t6d8x
readOnly: true
nodeName: <kubernetes node>
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: bootstrap
- emptyDir: {}
name: certs
- emptyDir: {}
name: ca
- name: kube-api-access-t6d8x
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-10-21T12:45:20Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-10-24T09:44:28Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-10-24T09:44:28Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-10-21T08:54:00Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://891a23829e977d64af2033ebe99f666e5711cef637f3c7558de13b7a831d2936
image: docker.io/envoyproxy/envoy:v1.23.1
imageID: docker.io/envoyproxy/envoy@sha256:9a1a68fc2adbc43896977bcff1333b3942a82af6157c90857be0fb33a5bf6e69
lastState:
terminated:
containerID: containerd://375a9040925f46e8c4bc770d42d616052f04e3b82486e25b5d10e670942bb582
exitCode: 1
finishedAt: "2022-10-24T09:39:18Z"
reason: Error
startedAt: "2022-10-24T09:38:48Z"
name: consul-api-gateway
ready: true
restartCount: 57
started: true
state:
running:
startedAt: "2022-10-24T09:44:27Z"
hostIP: 192.168.9.31
initContainerStatuses:
- containerID: containerd://a6b17cc178528f20b7e2dd69efd2fb6fac56ee9fefa392cc5ebde7e8d46e87ed
image: docker.io/hashicorp/consul-api-gateway:0.4.0
imageID: docker.io/hashicorp/consul-api-gateway@sha256:d37e0db75715c69b5684a3bf2ff7375ab34709ccd7733ebc8cffe2cfeeb4a1ca
lastState: {}
name: consul-api-gateway-init
ready: true
restartCount: 5
state:
terminated:
containerID: containerd://a6b17cc178528f20b7e2dd69efd2fb6fac56ee9fefa392cc5ebde7e8d46e87ed
exitCode: 0
finishedAt: "2022-10-24T08:48:48Z"
reason: Completed
startedAt: "2022-10-24T08:48:48Z"
phase: Running
podIP: 10.42.2.230
podIPs:
- ip: 10.42.2.230
qosClass: BestEffort
startTime: "2022-10-21T08:54:00Z"
We can see that the port I want to use is declared (1443), but the command launching the gateway does not reference this port, so I don’t see how the app could be listening on that port.
Here is my gateway conf:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
annotations:
api-gateway.consul.hashicorp.com/config: '{"serviceType":"NodePort","useHostPorts":true,"consul":{"authentication":{},"scheme":"https","ports":{"http":8501,"grpc":8502}},"image":{"consulAPIGateway":"hashicorp/consul-api-gateway:0.4.0","envoy":"envoyproxy/envoy:v1.23.1"},"copyAnnotations":{},"logLevel":"info","deployment":{"defaultInstances":1,"maxInstances":8,"minInstances":1}}'
objectset.rio.cattle.io/applied: <data>
objectset.rio.cattle.io/id: dd05420f-ddb6-4765-b2ff-bf8c0a3ae453
creationTimestamp: "2022-10-21T08:54:00Z"
generation: 1
labels:
objectset.rio.cattle.io/hash: c9df638a206ff7e4392cecd735052d40e453af79
manager: consul-api-gateway
operation: Update
subresource: status
time: "2022-10-21T08:54:02Z"
name: my-gateway
namespace: my-namespace
resourceVersion: "51800801"
uid: dcc61944-07e6-4d92-b4cd-85487dedb07a
spec:
gatewayClassName: consul-api-gateway
listeners:
- allowedRoutes:
namespaces:
from: Same
hostname: <my external hostname>
name: my-app
port: 1443
protocol: HTTPS
tls:
certificateRefs:
- group: ""
kind: Secret
name: <secret name>
namespace: my-namespace
mode: Terminate
status:
addresses:
- type: IPAddress
value: <node 1 IP>
- type: IPAddress
value: <node 2 IP>
conditions:
- lastTransitionTime: "2022-10-24T15:42:06Z"
message: Ready
observedGeneration: 1
reason: Ready
status: "True"
type: Ready
- lastTransitionTime: "2022-10-24T15:42:06Z"
message: Scheduled
observedGeneration: 1
reason: Scheduled
status: "True"
type: Scheduled
- lastTransitionTime: "2022-10-24T15:42:06Z"
message: InSync
observedGeneration: 1
reason: InSync
status: "True"
type: InSync
listeners:
- attachedRoutes: 2
conditions:
- lastTransitionTime: "2022-10-24T09:41:23Z"
message: NoConflicts
observedGeneration: 1
reason: NoConflicts
status: "False"
type: Conflicted
- lastTransitionTime: "2022-10-24T09:41:23Z"
message: Attached
observedGeneration: 1
reason: Attached
status: "False"
type: Detached
- lastTransitionTime: "2022-10-24T09:41:23Z"
message: Ready
observedGeneration: 1
reason: Ready
status: "True"
type: Ready
- lastTransitionTime: "2022-10-24T09:41:23Z"
message: ResolvedRefs
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: my-app
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
Am I on the right way ?
And my-app-gateway-603e9e5 does not match any pod or anything I’ve looked at, I’ve no idea about what it could be. Pods are my-app-gateway-6c899464b8-2xgnw and my-app-gateway-6c899464b8-lxn7n
