Consul v1.10.0-beta1 escape_hatch Envoy

Consul
1

I’m trying to add an ext_auth filter into Envoy using Consul’s escape hatch, but as soon as I start the side car I get “StreamAggregatedResources gRPC config stream closed: 2, Any JSON doesn’t have ‘@type’” This appears to be a configuration version mismatch between Consul and Envoy. Using the same configuration in a standalone Envoy works correctly and falls over when the same filter configuration is added to envoy_public_listener_json in the proxy_defaults.

I’m running the latest Envoy version 1.18.2, and the 1.10.0 beta1 version of Consul as the latest version of Consul does not appear to work with 1.18.2

service.hcl
service {
name = “my-server”
id = “my-server-1”
port = 8080
tags = [“v1”]
meta = {
version = “1”
}
connect {
sidecar_service {
port = 20000
}
}

check {
id = “my-server-check”
http = “http://localhost:8080/system/v1/ping”
method = “GET”
interval = “10s”
timeout = “1s”
}
}

proxy-defaults.hcl
Kind = “proxy-defaults”
Name = “my-server”
Config {
protocol = “http”

envoy_public_listener_json = <<EOF
    {
        "name": "public_listener",
        "address": {
            "socket_address": {
                "address": "0.0.0.0",
                "port_value": 21000
            }
        },
        "name": "envoy.filters.network.http_connection_manager",
                                        "typed_config": {
                                            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                                            "stat_prefix": "ingress_http",
                                            "route_config": {
                                                "name": "local_route",
                                                "virtual_hosts": [
                                                    {
                                                        "name": "local_service",
                                                        "domains": [
                                                            "*"
                                                        ],
                                                        "routes": [
                                                            {
                                                                "match": {
                                                                    "prefix": "/"
                                                                },
                                                                "route": {
                                                                    "cluster": "service_envoyproxy_io",
                                                                    "host_rewrite_literal": "localhost"
                                                                }
                                                            }
                                                        ]
                                                    }
                                                ]
                                            },
                                            "http_filters": [
                                                {
                                                    "name": "envoy.ext_authz",
                                                    "typed_config": {
                                                        "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz",
                                                        "http_service": {
                                                            "server_uri": {
                                                                "uri": "localhost:8090",
                                                                "cluster": "ext-authz",
                                                                "timeout": "0.250s"
                                                            }
                                                        }
                                                    }
                                                },
                                                {
                                                    "name": "envoy.filters.http.router"
                                                }
                                            ],
                                            "access_log": [
                                                {
                                                    "name": "envoy.access_loggers.stdout",
                                                    "typed_config": {
                                                        "@type": "type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog"
                                                    }
                                                }
                                            ]
                                        }
                                    }
    }

EOF

envoy_extra_static_clusters_json = <<EOF2
{
“name”: “ext-authz”,
“type”: “STRICT_DNS”,
“connect_timeout”: “0.250s”,
“load_assignment”: {
“cluster_name”: “ext-authz”,
“endpoints”: [
{
“lb_endpoints”: [
{
“endpoint”: {
“address”: {
“socket_address”: {
“address”: “127.0.0.1”,
“port_value”: 8090
}
}
}
}
]
}
]
}
}
EOF2
}

consul config write proxy-defaults.hcl
consul services register service.hcl
consul connect envoy -sidecar-for my-server-1 -admin-bind 127.0.0.1:19000 &

2

Hi there @zugarekd

Welcome to the HashiCorp Discuss Forums :slight_smile: Thank you for trying out the Consul 1.10 Beta! Currently Envoy 1.18 support is an open PR. Please watch that PR as it gets merged in. It shows as slated for 1.10, so it could land in Beta-2.

If you have some extra time, I’m curious - can you replicate this with 1.17?

Thanks for posting!

3

Hi there,

I looked into this and I have identified 4 problems:

  • (fixable) Your envoy_public_listener_json block has extra trailing curly brackets which means it’s not valid json (for escape hatches these things are decoded at use-time not write-time).
  • (fixable) Your envoy_public_listener_json block needs to be a Listener resource, but it’s not quite there (see below for corrected version).
  • (fixable in 1.10) Turns out that since Consul itself doesn’t use the v3 ExtAuthz structs at all, during compilation those envoyproxy/go-control-plane protobuf packages are not linked into the final Consul binary. This means that your escape hatch definitions can’t be decoded before being sent down to Envoy. I have an open PR to address this for 1.10 GA xds: ensure that all envoyproxy/go-control-plane protobuf symbols are linked into the final binary by rboyer · Pull Request #10131 · hashicorp/consul · GitHub
  • (???) The version of envoyproxy/go-control-plane we use is v0.9.5 which does not have definitions for envoy.extensions.access_loggers.stream.v3.StdoutAccessLog at all. For that we would have to upgrade the go-control-plane library, but that process is currently blocked on some not at all fun gogo/protobuf vs golang/protobuf dependency issues. This is likely not going to happen for 1.10 GA.
Kind = "proxy-defaults"
Name = "my-server"
Config {
  protocol = "http"

  envoy_public_listener_json = <<EOF
{
  "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
  "name": "public_listener:0.0.0.0:21000",
  "address": {
    "socket_address": {
      "address": "0.0.0.0",
      "port_value": 21000
    }
  },
  "filterChains": [
    {
      "filters": [
        {
          "name": "envoy.filters.network.http_connection_manager",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
            "stat_prefix": "ingress_http",
            "route_config": {
              "name": "local_route",
              "virtual_hosts": [
                {
                  "name": "local_service",
                  "domains": [
                    "*"
                  ],
                  "routes": [
                    {
                      "match": {
                        "prefix": "/"
                      },
                      "route": {
                        "cluster": "service_envoyproxy_io",
                        "host_rewrite_literal": "localhost"
                      }
                    }
                  ]
                }
              ]
            },
            "http_filters": [
              {
                "name": "envoy.filters.http.ext_authz",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz",
                  "http_service": {
                    "server_uri": {
                      "uri": "localhost:8090",
                      "cluster": "ext-authz",
                      "timeout": "0.250s"
                    }
                  }
                }
              },
              {
                "name": "envoy.filters.http.router"
              }
            ],
            "access_log": [
              {
                "name": "envoy.access_loggers.stdout",
                "typed_config": {
                  "@type": "type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog"
                }
              }
            ]
          }
        }
      ]
    }
  ]
}
EOF

  envoy_extra_static_clusters_json = <<EOF2
{
  "name": "ext-authz",
  "type": "STRICT_DNS",
  "connect_timeout": "0.250s",
  "load_assignment": {
    "cluster_name": "ext-authz",
    "endpoints": [
      {
        "lb_endpoints": [
          {
            "endpoint": {
              "address": {
                "socket_address": {
                  "address": "127.0.0.1",
                  "port_value": 8090
                }
              }
            }
          }
        ]
      }
    ]
  }
}
EOF2
}