Call upstream Consul services via localhost envoy without a specific port

Consul
1

Hi all,

I have been setting up Consul with Kong as ingress controller as per this guide: Consul with Kong as Ingress controller.

On that scenario, once the call comes from the Kong Ingress, say to service-a, it would call service-b that calls service-c. Note that service-a could also call a service-d and I would like those calls to happen via the envoy. My services have variables that determine the Gateway protocol, host and port that will determine the base uri for ALL upstream calls.

I have seen this being done in 2 ways:

  1. Using the (protocol)://(service-name) for the call: that works when service-a only calls 1 service, but if there are more services it would not;
  2. Adding upstream services as consul.hashicorp.com/connect-service-upstreams annotations, however I end in the same place given I have to use different ports.

I have also tried creating ServiceRoutes with the proper Path, that clearly show me a new entry on the envoy that needs to call the other API, however I can’t seen to find the port I can call on localhost that will do the matching, I have tried the outbound_listener port 15001 but that did not work http://localhost:15001/service/b/v1.

Please find below the Consul / Kong helm configurations, Consul CRDs and the examples of service-{a,b,c}.

I would really appreciate some guidance on the proper / best way to achieve this.

Thanks a lot in advance!

consul-config.yaml - for Helm install

global:
  name: consul
  logLevel: debug
  # Bootstrap ACLs within Consul. This is highly recommended.
#  acls:
#    manageSystemACLs: true
  image: "my-private-registry-proxy/consul:1.10.1"
  imageK8S: "my-private-registry-proxy/hashicorp/consul-k8s-control-plane:0.33.0"
  imageEnvoy: "my-private-registry-proxy/envoyproxy/envoy-alpine:v1.18.3"
  metrics:
    enabled: true
    enableAgentMetrics: true
prometheus:
  enabled: true
server:
  replicas: 1
client:
  enabled: true
connectInject:
  enabled: true
  transparentProxy:
    defaultEnabled: true
  logLevel: debug
controller:
  enabled: true
ui:
  enabled: true
terminatingGateways:
  enabled: true

consul-default-intentions-deny-all.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: default
spec:
  destination:
    name: "*"
  sources:
    - name: "*"
      action: deny

consul-global-proxy-defaults.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
 name: global
spec:
 config:
   protocol: http

kong-config.yaml (for Helm install):

deployment:
  log_level: "debug"
ingressController:
  serviceAccount:
    name: kong-helm-kong-proxy
  image:
    repository: my-private-registry-proxy/kong/kubernetes-ingress-controller
    tag: "1.3.1"
### Enable Consul Integration
podAnnotations:
 consul.hashicorp.com/connect-inject: "true"
 consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: 8000,8443
 consul.hashicorp.com/transparent-proxy: "true"
 consul.hashicorp.com/transparent-proxy-overwrite-probes: "true"
podLabels:
  aadpodidbinding: akv-mi
image:
  repository: my-private-registry-proxy/kong
  tag: "2.5.0"

consul-kong-service-defaults.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
 name: kong-helm-kong-proxy
spec:
 protocol: http

consul-kong-service-intentions.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: default
spec:
  destination:
    name: "*"
  sources:
    - name: "*"
      action: deny
    - name: "kong-helm-kong-proxy"
      action: allow

service-a.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-a
spec:
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/transparent-proxy-overwrite-probes: "true"
#        consul.hashicorp.com/connect-service-upstreams: "service-b:8008" # given I have to specify a port here it is not ideal
        prometheus.io/scrape: "true"
        prometheus.io/port: "9102"
        app: service-a
    spec:
      containers:
      - env:
        - name: API_SVC_PORT
          value: "80"
        - name: GATEWAY_SCHEME
          value: http
        - name: GATEWAY_HOST
          value: service-b # this is where I want to have localhost without port so it would match the path and go via envoy
        image: my-private-registry/cap/docker-service-a:1
        name: service-a
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
  name: service-a
  labels:
    runs: service-a
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: service-a
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: service-a
  annotations:
    kubernetes.io/ingress.class: kong
spec:
  rules:
    - http:
        paths:
        - backend:
            service:
              name: service-a
              port:
                number: 80
          path: /service/a/v1
          pathType: Prefix
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: service-a
spec:
  destination:
    name: service-b
  sources:
    - name: service-a
      permissions:
        - action: allow
          http:
            pathExact: /service/b/v1
            methods: ['GET']
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  name: service-a
spec:
  routes:
  - match:
      http:
        pathPrefix: /service/a/v1
    destination:
      service: service-a
...

service-b.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-b
spec:
  selector:
    matchLabels:
      app: service-b
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/transparent-proxy-overwrite-probes: "true"
#        consul.hashicorp.com/connect-service-upstreams: "service-b:8008" # given I have to specify a port here it is not ideal
        prometheus.io/scrape: "true"
        prometheus.io/port: "9102"
        app: service-b
    spec:
      containers:
      - env:
        - name: API_SVC_PORT
          value: "80"
        - name: GATEWAY_SCHEME
          value: http
        - name: GATEWAY_HOST
          value: service-c # this is where I want to have localhost without port so it would match the path and go via envoy
        image: my-private-registry/cap/docker-service-b:1
        name: service-b
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
  name: service-b
  labels:
    runs: service-b
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: service-b
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: service-b
  annotations:
    kubernetes.io/ingress.class: kong
spec:
  rules:
    - http:
        paths:
        - backend:
            service:
              name: service-b
              port:
                number: 80
          path: /service/b/v1
          pathType: Prefix
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: service-b
spec:
  destination:
    name: service-c
  sources:
    - name: service-b
      permissions:
        - action: allow
          http:
            pathExact: /service/c/v1
            methods: ['GET']
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  name: service-b
spec:
  routes:
  - match:
      http:
        pathPrefix: /service/b/v1
    destination:
      service: service-b
...

service-c.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: service-c
spec:
  selector:
    matchLabels:
      app: service-c
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/transparent-proxy-overwrite-probes: "true"
        prometheus.io/scrape: "true"
        prometheus.io/port: "9102"
        app: service-c
    spec:
      containers:
      - env:
        - name: API_SVC_PORT
          value: "80"
        - name: GATEWAY_SCHEME
          value: http
        - name: GATEWAY_HOST
          value: does-not-matter # in this case it does not call anything else
        image: my-private-registry/cap/docker-service-c:1
        name: service-c
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    ingress.kubernetes.io/service-upstream: "true"
  name: service-c
  labels:
    runs: service-c
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
      name: http
  selector:
    app: service-b
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: service-c
  annotations:
    kubernetes.io/ingress.class: kong
spec:
  rules:
    - http:
        paths:
        - backend:
            service:
              name: service-c
              port:
                number: 80
          path: /service/c/v1
          pathType: Prefix
# This service does not call other services so no intentions declared here
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  name: service-c
spec:
  routes:
  - match:
      http:
        pathPrefix: /service/c/v1
    destination:
      service: service-c
...
2

Update: the above the be implemented as below:

  1. Disable transparent proxy on Consul and Kong;
  2. Add a single upsteam annotation to the service that needs to access upstreams, i.e.: consul.hashicorp.com/connect-service-upstreams: “service-a-outbound:10000”;
  3. Set your hostname as localhost and port as 10000;
  4. Make sure you have a route for all the services it needs to access, i.e.:
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  name: service-a-outbound
spec:
  routes:
  - match:
      http:
        pathPrefix: /service/b/path/v1
    destination:
      service: service-b
  - match:
      http:
        pathPrefix: /service/c/path/v1
    destination:
      service: service-c

Refer to this code for a complete example: https://github.com/phan-t/terraform-consul-master/tree/main/examples/applications/fake-service

Limitations:

  1. The service Topology does not seem to be updated - this is something that Hashicorp will improve for Virtual Services;
  2. You cannot apply different limits for different outbounds - ServiceDefaults > upstreamConfig > overrides. Only defaults makes sense in this case.