Consul sidecar force tcp

Hello Community!
I am trying to connect to MariaDB nomad job from Drupal job using the sidecar proxy.

Here is my setup:

Name               Address   Port  Status  Leader  Raft Version  Build  Datacenter   Region
new-app1-t.global  10.1.0.2  4648  alive   true    3             1.7.5  hetzner-dev  global
new-app2-t.global  10.1.0.3  4648  alive   false   3             1.7.5  hetzner-dev  global
new-app3-t.global  10.1.0.4  4648  alive   false   3             1.7.5  hetzner-dev  global
---------
ID        Node Pool  DC           Name           Class   Drain  Eligibility  Status
8***3  default    hetzner-dev  new-client3-t  <none>  false  eligible     ready
4***b  default    hetzner-dev  new-client2-t  <none>  false  eligible     ready
5***a10  default    hetzner-dev  new-client1-t  <none>  false  eligible     ready
---------
Node           Address        Status  Type    Build   Protocol  DC           Partition  Segment
new-server1-t  10.1.0.2:8301  alive   server  1.17.1  2         hetzner-dev  default    <all>
new-server2-t  10.1.0.3:8301  alive   server  1.17.1  2         hetzner-dev  default    <all>
new-server3-t  10.1.0.4:8301  alive   server  1.17.1  2         hetzner-dev  default    <all>
new-client1-t  10.1.0.5:8301  alive   client  1.17.1  2         hetzner-dev  default    <default>
new-client2-t  10.1.0.6:8301  alive   client  1.17.1  2         hetzner-dev  default    <default>
new-client3-t  10.1.0.7:8301  alive   client  1.17.1  2         hetzner-dev  default    <default>

When I use mariadb client from drupal container, the connection just hangs:

/opt/drupal# mariadb -u root -p -P 33306 -h 127.0.0.1
Enter password:

(yes, local_bind_port = 33306)

Here is what I get when I use curl to test the connection:

root@a855670c044b:/opt/drupal# curl 127.0.0.1:33306

upstream connect error or disconnect/reset before headers. reset reason: protocol error

Now, when I use telnet, I get the most useful(IMO) reply:

root@a855670c044b:/opt/drupal# telnet 127.0.0.1 33306
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

1234test  (this is my test input, just to get any answer from the service)  

HTTP/1.1 400 Bad Request
content-length: 11
content-type: text/plain
date: Wed, 20 Mar 2024 21:17:31 GMT
server: envoy
connection: close

Bad RequestConnection closed by foreign host.

So, since I get reply from Envoy and not MariaDB server, it seems like Consul and Envoy try to handle it at L7 instead of L4.
So the question is: how do I force L4 (TCP) instead of L7 (HTTP) for my sidecar?

Here is what I have tried:

    service {
      name     = "web-svc"
      port     = "web"
      provider = "consul"
      connect {
      sidecar_service {
        proxy {
          config {
            protocol = "tcp"
            }
          upstreams {
            destination_name = "mariadb"
            local_bind_port = 33306
            }
          }
        }
      }

Pleas advise!

@andrii.kravchenko.of Have you registered mariadb in the service mesh and deployed a sidecar for it? Do you mind sharing that configuration?

Hello, Blake!

Yes, mariadb is registered as a service through the nomad job.

**andriikravchenko@new-app1-t**:**~/wip-app1/ansible-nomad/Jobs**$ consul catalog services

consul

haproxy

mariadb

mariadb-sidecar-proxy

nomad

nomad-client

web-svc

web-svc-sidecar-proxy

Here is a cutout from mariadb job:

    network {
      mode = "bridge"
      port "mysql" {
        to = 3306
      }
    }

    service {
      name = "mariadb"
      port = "3306"
      provider = "consul"
      connect {
        sidecar_service {}
      }
    }

And here is a cutout from drupal nomad job:

  group "drupal-web" {
    count = 3
    network {
      port "web" {
        to = 80
      }
      mode = "bridge"
    }
    ephemeral_disk {
      migrate = true
      size    = 10000
      sticky  = true
    }
    service {
      name     = "web-svc"
      port     = "web"
      provider = "consul"
      connect {
      sidecar_service {
        proxy {
          config {
            protocol = "tcp"
            }
          upstreams {
            destination_name = "mariadb"
            local_bind_port = 33306
            }
          }
        }
      }

    }

Maybe you have a best practice nomad+consul connect example for databases that communicate through TCP?

And here is some extra sidecar information:

andriikravchenko@new-app1-t:~/wip-app1/ansible-nomad/Jobs$ curl http://localhost:8500/v1/catalog/service/web-svc-sidecar-proxy | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4648    0  4648    0     0  1364k      0 --:--:-- --:--:-- --:--:-- 2269k
[
  {
    "ID": "fbebe7b7-e44b-ee9f-dd68-2a6bc48ae4dd",
    "Node": "new-client1-t",
    "Address": "10.1.0.5",
    "Datacenter": "hetzner-dev",
    "TaggedAddresses": {
      "lan": "10.1.0.5",
      "lan_ipv4": "10.1.0.5",
      "wan": "10.1.0.5",
      "wan_ipv4": "10.1.0.5"
    },
    "NodeMeta": {
      "consul-network-segment": "",
      "consul-version": "1.17.1"
    },
    "ServiceKind": "connect-proxy",
    "ServiceID": "_nomad-task-b35b5c5b-dc74-793a-7fb2-eb5ac5a398e3-group-drupal-web-web-svc-web-sidecar-proxy",
    "ServiceName": "web-svc-sidecar-proxy",
    "ServiceTags": [],
    "ServiceAddress": "10.1.0.5",
    "ServiceTaggedAddresses": {
      "consul-virtual": {
        "Address": "240.0.7.73",
        "Port": 20721
      },
      "lan_ipv4": {
        "Address": "10.1.0.5",
        "Port": 20721
      },
      "wan_ipv4": {
        "Address": "10.1.0.5",
        "Port": 20721
      }
    },
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-source": "nomad"
    },
    "ServicePort": 20721,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "DestinationServiceName": "web-svc",
      "DestinationServiceID": "_nomad-task-b35b5c5b-dc74-793a-7fb2-eb5ac5a398e3-group-drupal-web-web-svc-web",
      "LocalServiceAddress": "127.0.0.1",
      "LocalServicePort": 31663,
      "Mode": "",
      "Config": {
        "bind_address": "0.0.0.0",
        "bind_port": 20721,
        "envoy_stats_tags": [
          "nomad.alloc_id=b35b5c5b-dc74-793a-7fb2-eb5ac5a398e3",
          "nomad.group=drupal-web",
          "nomad.job=drupal",
          "nomad.namespace=default"
        ],
        "protocol": "tcp"
      },
      "Upstreams": [
        {
          "DestinationType": "service",
          "DestinationName": "mariadb",
          "Datacenter": "",
          "LocalBindPort": 33306,
          "Config": {
            "protocol": "http2"
          },
          "MeshGateway": {}
        }
      ],
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "ServiceLocality": null,
    "CreateIndex": 524275,
    "ModifyIndex": 563497
  },
  {
    "ID": "6a49a0be-6238-4d4f-d61f-ec9c7950b6e2",
    "Node": "new-client2-t",
    "Address": "10.1.0.6",
    "Datacenter": "hetzner-dev",
    "TaggedAddresses": {
      "lan": "10.1.0.6",
      "lan_ipv4": "10.1.0.6",
      "wan": "10.1.0.6",
      "wan_ipv4": "10.1.0.6"
    },
    "NodeMeta": {
      "consul-network-segment": "",
      "consul-version": "1.17.1"
    },
    "ServiceKind": "connect-proxy",
    "ServiceID": "_nomad-task-5ef6d9ba-92b4-6a8d-cfd1-414aa814c2ca-group-drupal-web-web-svc-web-sidecar-proxy",
    "ServiceName": "web-svc-sidecar-proxy",
    "ServiceTags": [],
    "ServiceAddress": "10.1.0.6",
    "ServiceTaggedAddresses": {
      "consul-virtual": {
        "Address": "240.0.7.73",
        "Port": 31513
      },
      "lan_ipv4": {
        "Address": "10.1.0.6",
        "Port": 31513
      },
      "wan_ipv4": {
        "Address": "10.1.0.6",
        "Port": 31513
      }
    },
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-source": "nomad"
    },
    "ServicePort": 31513,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "DestinationServiceName": "web-svc",
      "DestinationServiceID": "_nomad-task-5ef6d9ba-92b4-6a8d-cfd1-414aa814c2ca-group-drupal-web-web-svc-web",
      "LocalServiceAddress": "127.0.0.1",
      "LocalServicePort": 20626,
      "Mode": "",
      "Config": {
        "bind_address": "0.0.0.0",
        "bind_port": 31513,
        "envoy_stats_tags": [
          "nomad.alloc_id=5ef6d9ba-92b4-6a8d-cfd1-414aa814c2ca",
          "nomad.group=drupal-web",
          "nomad.job=drupal",
          "nomad.namespace=default"
        ],
        "protocol": "tcp"
      },
      "Upstreams": [
        {
          "DestinationType": "service",
          "DestinationName": "mariadb",
          "Datacenter": "",
          "LocalBindPort": 33306,
          "Config": {
            "protocol": "http2"
          },
          "MeshGateway": {}
        }
      ],
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "ServiceLocality": null,
    "CreateIndex": 560163,
    "ModifyIndex": 563498
  },
  {
    "ID": "6a49a0be-6238-4d4f-d61f-ec9c7950b6e2",
    "Node": "new-client2-t",
    "Address": "10.1.0.6",
    "Datacenter": "hetzner-dev",
    "TaggedAddresses": {
      "lan": "10.1.0.6",
      "lan_ipv4": "10.1.0.6",
      "wan": "10.1.0.6",
      "wan_ipv4": "10.1.0.6"
    },
    "NodeMeta": {
      "consul-network-segment": "",
      "consul-version": "1.17.1"
    },
    "ServiceKind": "connect-proxy",
    "ServiceID": "_nomad-task-a3cc9850-3c63-f872-96c4-0d5f496c9441-group-drupal-web-web-svc-web-sidecar-proxy",
    "ServiceName": "web-svc-sidecar-proxy",
    "ServiceTags": [],
    "ServiceAddress": "10.1.0.6",
    "ServiceTaggedAddresses": {
      "consul-virtual": {
        "Address": "240.0.7.73",
        "Port": 24767
      },
      "lan_ipv4": {
        "Address": "10.1.0.6",
        "Port": 24767
      },
      "wan_ipv4": {
        "Address": "10.1.0.6",
        "Port": 24767
      }
    },
    "ServiceWeights": {
      "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {
      "external-source": "nomad"
    },
    "ServicePort": 24767,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
      "DestinationServiceName": "web-svc",
      "DestinationServiceID": "_nomad-task-a3cc9850-3c63-f872-96c4-0d5f496c9441-group-drupal-web-web-svc-web",
      "LocalServiceAddress": "127.0.0.1",
      "LocalServicePort": 24144,
      "Mode": "",
      "Config": {
        "bind_address": "0.0.0.0",
        "bind_port": 24767,
        "envoy_stats_tags": [
          "nomad.alloc_id=a3cc9850-3c63-f872-96c4-0d5f496c9441",
          "nomad.group=drupal-web",
          "nomad.job=drupal",
          "nomad.namespace=default"
        ],
        "protocol": "tcp"
      },
      "Upstreams": [
        {
          "DestinationType": "service",
          "DestinationName": "mariadb",
          "Datacenter": "",
          "LocalBindPort": 33306,
          "Config": {
            "protocol": "http2"
          },
          "MeshGateway": {}
        }
      ],
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "ServiceLocality": null,
    "CreateIndex": 524278,
    "ModifyIndex": 563499
  }
]
andriikravchenko@new-app1-t