Problems with api-gateway in consul and nomad

Nomad
1

I’m trying to configure an API gateway on Consul and Nomad. I have both solutions integrated and Nomad Workloads enabled without any issues. I’ve also enabled grpc_tls and HTTPS. I’m following the API gateway tutorial on this page: Deploy a Consul API Gateway on Nomad | Nomad | HashiCorp Developer . However, when I deploy the final step in the “api” alloc in Nomad, I get this error: [2026-07-01 15:30:58.947][1][info][main] [source/server/server.cc:491] response trailer map: 144 bytes: grpc-message,grpc-status
[2026-07-01 15:30:58.956][1][info][main] [source/server/server.cc:845] runtime: layers:

name: base
static_layer:
re2.max_program_size.error_level: 1048576
[2026-07-01 15:30:58.957][1][info][admin] [source/server/admin/admin.cc:66] admin address: 0.0.0.0:19000
[2026-07-01 15:30:58.958][1][info][config] [source/server/configuration_impl.cc:160] loading tracing configuration
[2026-07-01 15:30:58.958][1][info][config] [source/server/configuration_impl.cc:119] loading 0 static secret(s)
[2026-07-01 15:30:58.958][1][info][config] [source/server/configuration_impl.cc:125] loading 1 cluster(s)
[2026-07-01 15:30:59.000][1][info][config] [source/server/configuration_impl.cc:129] loading 0 listener(s)
[2026-07-01 15:30:59.000][1][info][config] [source/server/configuration_impl.cc:146] loading stats configuration
[2026-07-01 15:30:59.001][1][info][runtime] [source/common/runtime/runtime_impl.cc:584] RTDS has finished initialization
[2026-07-01 15:30:59.001][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:236] cm init: initializing cds
[2026-07-01 15:30:59.001][1][warning][main] [source/server/server.cc:910] There is no configured limit to the number of allowed active downstream connections. Set a limit in envoy.resource_monitors.downstream_connections resource monitor.
[2026-07-01 15:30:59.001][1][info][main] [source/server/server.cc:951] starting main dispatch loop
[2026-07-01 15:31:44.281][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:193] DeltaAggregatedResources gRPC config stream to local_agent closed since 45s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination.

Any help please

2

First step to debug would be to use grpc rather than grpc_tls to make sure everything else is working, the errors don’t show it but I expect it is the issue as envoy sidecar isn’t getting the cert.

You can configure this in the Consul block in the Nomad client config.

consul {
  address = "127.0.0.1:8500"
  token = "<redacted>"
  grpc_address = "127.0.0.1:8502"
...
}
3

Hello, I’ve updated the Consul configuration on the clients and made some progress, but now I’m having other problems running the example. The following error appears in the sidecard: “[2026-07-06 16:49:47.455][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:227] DeltaAggregatedResources gRPC config stream to local_agent closed since 1224s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: remote connection failure{upstream_reset_before_response_started{remote_connection_failure|TLS_error:|268435703:SSL_routines:OPENSSL_internal:WRONG_VERSION_NUMBER:TLS_error_end}}” I’ve attached the files with the logs and the Ansible template I use in the Consul clients.

Help please

4
7

This is envoy failing to communicate with Consul for XDS updates. Can you share your Nomad and Consul client config too? Primarily wanting to see if your grpc listener is on the expected port and is listening on the right interface.

8

There is the consul client config file: UNTITLED - Pastebin Service

And this is the nomad client config file: UNTITLED - Pastebin Service

9

I have gone a little further and I have corrected many problems, and I have deployed the api-gateway job and the example application, in “connect-proxy-hello-app”, I get this in the logs: “[2026-07-07 15:59:12.274][1][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:227] DeltaAggregatedResources gRPC config stream to local_agent closed since 65416s ago: 14, upstream connect error or disconnect/reset before reset headers reason: remote connection. failure{upstream_reset_before_response_started{remote_connection_failure|TLS_error:|268435703:SSL_routines:OPENSSL_internal:WRONG_VERSION_NUMBER:TLS_error_end}}” And I don’t know where to look to determine the error or where it’s occurring.
Ports 8501, 8502, and 8503 are open

10

The thing that stood out was the API Gateway container probably can’t communicate to the Consul agent which is only exposing the listener to localhost.

11

I went a little further, there are no more errors but I can’t access hello-app through the api-gateway and I got this error:

➜ ~ consul config read -kind http-route -name my-http-route
{
“Kind”: “http-route”,
“Name”: “my-http-route”,
“Parents”: [
{
“Kind”: “api-gateway”,
“Name”: “my-api-gateway”,
“SectionName”: “my-http-listener”
}
],
“Rules”: [
{
“Filters”: {
“Headers”: null,
“URLRewrite”: null,
“RetryFilter”: null,
“TimeoutFilter”: null,
“JWT”: null
},
“ResponseFilters”: {
“Headers”: null
},
“Matches”: [
{
“Headers”: null,
“Method”: “”,
“Path”: {
“Match”: “prefix”,
“Value”: “/hello”
},
“Query”: null
}
],
“Services”: [
{
“Name”: “hello-app”,
“Weight”: 0,
“Filters”: {
“Headers”: null,
“URLRewrite”: null,
“RetryFilter”: null,
“TimeoutFilter”: null,
“JWT”: null
},
“ResponseFilters”: {
“Headers”: null
}
}
]
}
],
“Hostnames”: null,
“CreateIndex”: 147146,
“ModifyIndex”: 151036,
“Status”: {
“Conditions”: [
{
“Type”: “Accepted”,
“Status”: “False”,
“Reason”: “InvalidDiscoveryChain”,
“Message”: “route protocol does not match targeted service protocol”,
“Resource”: {
“Kind”: “”,
“Name”: “”,
“SectionName”: “”
},
“LastTransitionTime”: “2026-07-06T19:24:16.822845074Z”
},
{
“Type”: “Bound”,
“Status”: “False”,
“Reason”: “FailedToBind”,
“Message”: “failed to bind route my-http-route to gateway my-api-gateway with listener ‘my-http-listener’”,
“Resource”: {
“Kind”: “api-gateway”,
“Name”: “my-api-gateway”,
“SectionName”: “my-http-listener”
},
“LastTransitionTime”: “2026-07-06T19:24:16.828519502Z”
}
]
}
}

12

I suspect your route protocol is HTTP because it’s doing L7 path matching but the service is TCP. This can be fixed just for hello-app with a service-defaults configuration that will look similar to this:

$ cat ./hello-app-defaults.hcl

Kind      = "service-defaults"
Name      = "hello-app"
Namespace = "default"
Protocol  = "http"

$ consul config write ./hello-app-defaults.hcl

or alternatively set for all backend services using a proxy-defaults configuration

The other possibility, and it may be both, is the API Gateway listener protocol isn’t set, which defaults to TCP, this is what I have configured, excuse the terraform but the Listener config should look similar when pulled out.

resource "consul_config_entry" "api_gateway" {
  kind = "api-gateway"
  name = "api-gateway"

  config_json = jsonencode({
    Listeners = [
      {
        Name = "http-listener"
        Port = 8093
        Protocol = "http"
        TLS = {
          Certificates = null
        }

      }    
    ]    
  })
}