Envoy sidecar bootstrap config - add one line

Consul
1

Hi folks,

I want to update the bootstrap config that is used by the transparent envoy sidecars. I just need to add one line to the existing default config:

i.e. add:
forward_client_cert_details: ALWAYS_FORWARD_ONLY

so that the bootstrap config appears as:

<snip>

"filter_chains": [
            {
              "filters": [
                {
                  "name": "envoy.filters.network.http_connection_manager",
                  "typedConfig": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                    "stat_prefix": "envoy_prometheus_metrics",
                    "codec_type": "HTTP1",
                    "forward_client_cert_details": "ALWAYS_FORWARD_ONLY",
<snip>

I’ve read through the docs and I can’t seem to find a simple way to achieve this through any of the Consul Connect CRDs.

Is my only option to define a full-blown bootstrap template per?

Or are there simpler options?

PS: the use-case for this is to try pass the identity of a down-stream service into an upstream service.

TIA

2

Hi @byrneo I believe the only option here is to provide a template through the escape hatch option.

3

Thanks for the response, @karl-cardenas-coding!

I’m new to this and I’ve made no changes yet. I notice when I dump the generated envoy config, that under dynamic_listeners it has dynamically generated:

  1. a public listener (port 200000);
  2. an outbound listener (port 15000)

If i wanted to make forward_client_cert_details: ALWAYS_FORWARD_ONLY become a default in the filter chain for all services I deploy to my cluster, I’m guessing I would need it to be included in the filter-chain for both the dynamically generated public listener and outbound listener in each envoy sidecar?

If so, I’m a bit unclear as to what escape-hatch option would best achieve this? If i were to use envoy_public_listener_json - then i think that would only enable it on incoming requests, but not for the outbound traffic?

Here’s what I see has been generated for the outbound listener. I think i would need a new http filter to be added here also in order to set forward_client_cert_details: ALWAYS_FORWARD_ONLY

 {
          "name": "outbound_listener:127.0.0.1:15001",
          "active_state": {
            "version_info": "9ca59cd76f3ad0fd80de2ef9694128a1bbb9d852d62f915d5f153e70d73eb584",
            "listener": {
              "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
              "name": "outbound_listener:127.0.0.1:15001",
              "address": {
                "socket_address": {
                  "address": "127.0.0.1",
                  "port_value": 15001
                }
              },
              "filter_chains": [
                {
                  "filters": [
                    {
                      "name": "envoy.filters.network.tcp_proxy",
                      "typed_config": {
                        "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
                        "stat_prefix": "upstream.original-destination",
                        "cluster": "original-destination"
                      }
                    }
                  ]
                }
              ],
              "listener_filters": [
                {
                  "name": "envoy.filters.listener.original_dst"
                }
              ],
              "traffic_direction": "OUTBOUND"
            },
            "last_updated": "2021-11-22T19:47:18.464Z"
          }

Alternatively, would there be any option to somehow set this in bootstrap so that it would take effect for all dynamic listeners?

4

In case it helps anyone, I configured a custom envoy_public_listener_json and used it with a proxy default.

Steps:

cat <<EOF | jq '. | @json'
{
    "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "name": "public_listener:0.0.0.0:20000",
    "address": {
      "socket_address": {
        "address": "0.0.0.0",
        "port_value": 20000
      }
    },
    "filterChains": [
      {
        "filters": [
          {
            "name": "envoy.filters.network.http_connection_manager",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
              "stat_prefix": "public_listener",
              "forward_client_cert_details": "APPEND_FORWARD",
              "set_current_client_cert_details": {
               "subject": true,
               "dns": true,
               "uri": true
              },
              "route_config": {
                "name": "public_listener",
                "virtual_hosts": [
                  {
                    "name": "public_listener",
                    "domains": [
                      "*"
                    ],
                    "routes": [
                      {
                        "match": {
                          "prefix": "/"
                        },
                        "route": {
                          "cluster": "local_app"
                        }
                      }
                    ]
                  }
                ]
              },
              "http_filters": [
                {
                  "name": "envoy.filters.http.router"
                }
              ]
            }
          }
        ]
      }
    ]
  }
EOF

Then, took the output from above and used it as follows:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  config:
    envoy_public_listener_json: "{\"@type\":\"type.googleapis.com/envoy.config.listener.v3.Listener\",\"name\":\"public_listener:0.0.0.0:20000\",\"address\":{\"socket_address\":{\"address\":\"0.0.0.0\",\"port_value\":20000}},\"filterChains\":[{\"filters\":[{\"name\":\"envoy.filters.network.http_connection_manager\",\"typed_config\":{\"@type\":\"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager\",\"stat_prefix\":\"public_listener\",\"forward_client_cert_details\":\"APPEND_FORWARD\",\"set_current_client_cert_details\":{\"subject\":true,\"dns\":true,\"uri\":true},\"route_config\":{\"name\":\"public_listener\",\"virtual_hosts\":[{\"name\":\"public_listener\",\"domains\":[\"*\"],\"routes\":[{\"match\":{\"prefix\":\"/\"},\"route\":{\"cluster\":\"local_app\"}}]}]},\"http_filters\":[{\"name\":\"envoy.filters.http.router\"}]}}]}]}"

Still testing this out but can verify that the client-cert/SPIFFE-ID is now being passed to the upstream service.

5

Thanks for sharing @byrneo . Let me know what the end result is, I’d love to update the docs with a similar example.

6

hey @karl-cardenas-coding - it seems to be working out fine. When i make a call from a client service to an echo service i can now see the X-Forwarded-Client-Cert header containing the SPIFFE URI of the downstream client (which is what I was hoping to achieve).

The only thing that seems amiss is that when i apply the ProxyDefaults resource (see manifest in previous message) to the k8s cluster, it results in the following error in the consul server:

2021-12-01T15:00:52.680Z	ERROR	controller.proxydefaults	Reconciler error	{"reconciler group": "consul.hashicorp.com", "reconciler kind": "ProxyDefaults", "name": "global", "namespace": "default", "error": "Operation cannot be fulfilled on proxydefaults.consul.hashicorp.com \"global\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227

But…it still seems to successfully create it:

kubectl get proxydefaults.consul.hashicorp.com global
NAME     SYNCED   LAST SYNCED   AGE
global   True     23h           23h
7

Hmmm…intersting. @byrneo any chance I could ask you to open a GitHub issue for this? That way we can track it officially

8

Hi @byrneo that is a known issue due to a race condition within the controller but it does resolve itself immediately.

2 Likes