I'm encountering an issue with setting up the service mesh

I’m encountering an issue with setting up the service mesh. Currently, I have two nodes that can see each other:

Node   Address              Status  Type    Build   Protocol  DC   Partition  Segment
node1  176.25.218.126:8301  alive   server  1.19.0  2         dc1  default    <all>
node2  25.265.25.45:8301    alive   server  1.19.0  2         dc1  default    <all>

On one node, I’ve configured a simple setup:

{
  "service": {
    "name": "web",
    "tags": [
      "front"
    ],
    "port": 80,
    "check": {
      "args": ["curl", "localhost"],
      "interval": "10s"
    },
    "connect": {
      "sidecar_service": {
        "proxy": {
          "upstreams": [{
            "destination_name": "db",
            "local_bind_port": 33306
          }]
        }
      }
    }
  }
}

However, I’m getting the following error:

Jul 03 17:50:58 loki consul[3127038]: 2024-07-03T17:50:58.354+0300 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=peering-trust-bundles error="error filling agent cache: failed to list all discovery chains referring to \"web\": failed to fetch discovery chain for \"web\": no cluster ca config setup"
Jul 03 17:50:58 loki consul[3127038]: agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=peering-trust-bundles error="error filling agent cache: failed to list all discovery chains referring to \"web\": failed to fetch discovery chain for \"web\": no cluster ca config setup"
Jul 03 17:50:58 loki consul[3127038]: 2024-07-03T17:50:58.357+0300 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=discovery-chain:db error="error filling agent cache: no cluster ca config setup"
Jul 03 17:50:58 loki consul[3127038]: agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=discovery-chain:db error="error filling agent cache: no cluster ca config setup"

It seems that it cannot find the certificate in the CA. I followed the documentation to create the certificate at Certificate Authority - Built-in Service Mesh CA | Consul | HashiCorp Developer, and it appears to have been set up correctly:

curl localhost:8500/v1/connect/ca/configuration
{"Provider":"consul","Config":{"IntermediateCertTTL":"8760h","LeafCertTTL":"72h","PrivateKey":"-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCX5GxDjpUAx72r\nJ65tWcFuP4xX2HOVy9YgvnGWm+MnWtKMIIkUUrD68HvZ3kZAr2XEu0Lfzoup5lAz\nEucufcDM2I9piE/SRDnsPHZV9dcBemzBGm0JxESwbG7gERWJX0g/EK5b0Jj2GPmy\nqkjq6Csn6aE3e8m/YfoDaQ9A9/+4Qwyqv27QywK0iDhgJFDG+XGnCLTzmzusbVnx\nThSj1u9/IKNpC1rn34nCe6xVxoaFfMhErktcvqBYpOkzL/dLK8ivLQWY2oHgljwP\ncERKyR0BtzYdkGY8h63WxqY0Y0fMqVw3a+Sre0prHrWmBgTfqtSFtWZ9xSTnuWTE\nFeNtIzxDAgMBAAECggEABSziR3b/OrSTkNZcShuJzLDbnk6guZTlad4zTpdnKPse\nO2oMYaXjo9k3zxeUuzV+uTJMOAX4mkRfrS3StLHSNOiYEldzqM1sT9FLkug6eDFs\nrTOyTWfjk0NOj9HlMvYNJyDxX+1+mLwHHgBRVbetaMG8HYDZQe0bsKwOuf75NpT8\n0XT9Y8uOCso+ZQQwdb4Sollrrle2DJYJVjD6NaN9u49k9Ntq1/wY2rXNlOkUppDu\n5XdqYxIDRuHjqf3E2rB2moIsK4YtblwVf2FFu4FMMNGK1uO2uNdUWHc5SaDADxLo\nXXkp01Pq7+/9HyyDYwdmjfDBFD4rN70ynwi6QtoTEQKBgQC9JtEoCYgJusuV+/zT\nmyHPKCCtI/z+7ti8fe7AP69nBWh357MlJQdQNhYMTENOO4Y/QLdPqxq2fH2GdZ+x\n6PROye9qhKjar+vqL4KQLaqjl7KTCitQ3nixaTgpY864IuXCofwEQNo+SN2JfP66\nunJQzBQPGen8H2Bm5xWU6gJxiwKBgQDNkqCQv5Y5BaCGgL+RIIvP2KqT7t73WMx0\nnH5aKt3jnI5+tTxFa9Yt/dGzceCXI9WUUcWz/FeirZbfJJQvxy6AFug7nX9jB62p\nLiRgOElFd5OjJBHIJf3h9/syxZH11MJnNht240OVCQqYXZPHY5yIKDnuJbDNiVxG\naAY7/PTHKQKBgGYnSpo225N1oHG2D4swowfAAjW/0+jSkZbq5EBLpK0czJjFN4j+\nAH8fVrT5kvfzScNrbhTGsbyQazJs3/wzdY3/nR+H2bGRhPWLBxZas2d6i+TBfzdD\njycxTjV8Q65Kmo+M8BBT/gnZTFvud5vdCdP9A3BMbIPPZj7s2kj8QKmVAoGAZIha\nga9QFd7MnDqGdMDH3wOEmAfvfmMsybJmkE36aX98qDaVnRWZjrBcRzdfsEpT/s7m\nspireuENsV7AIYV1Fisds1nYnGN66AuqumuMWa6awuO21nzN5h5R+IhqUyCTWco6\nMDAAMvhS+CR7Sr1KJcQWYIKKLloIiF4C/n88bdECgYBSAvgX4PwQwZkTTe8rULTS\nK9sE4xjy7ydoLHAKgJVTmv3dzKqMIZy3cvPPACPmXd3r2jtDP0tq7JghfvViASj0\nqX+evZUk8Mp6twhLK9DwwiDGGLPyrXEWXMuH5bmBIBZfzbAwlS9AL43OK2t9P04f\np7w6uOvAhuT3zteROFUv3g==\n-----END PRIVATE KEY-----","RootCert":"-----BEGIN CERTIFICATE-----\nMIIDazCCAlOgAwIBAgIUOB/1COFxAnnxD7mtuozimC44FdkwDQYJKoZIhvcNAQEL\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yNDA3MDMxMzM2NTdaFw0zNDA3\nMDExMzM2NTdaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQCX5GxDjpUAx72rJ65tWcFuP4xX2HOVy9YgvnGWm+Mn\nWtKMIIkUUrD68HvZ3kZAr2XEu0Lfzoup5lAzEucufcDM2I9piE/SRDnsPHZV9dcB\nemzBGm0JxESwbG7gERWJX0g/EK5b0Jj2GPmyqkjq6Csn6aE3e8m/YfoDaQ9A9/+4\nQwyqv27QywK0iDhgJFDG+XGnCLTzmzusbVnxThSj1u9/IKNpC1rn34nCe6xVxoaF\nfMhErktcvqBYpOkzL/dLK8ivLQWY2oHgljwPcERKyR0BtzYdkGY8h63WxqY0Y0fM\nqVw3a+Sre0prHrWmBgTfqtSFtWZ9xSTnuWTEFeNtIzxDAgMBAAGjUzBRMB0GA1Ud\nDgQWBBT/MkU2JN+U1GGHJASQYe2GvhlKfjAfBgNVHSMEGDAWgBT/MkU2JN+U1GGH\nJASQYe2GvhlKfjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBM\nlHKZ5EdN1d6b3s8y55ouub5NxkWSdFqyMRDLjizTB/kTVPC+X3hT/XACA1CZY/vV\nj4o2A6YjqdREXzdqJf9PMFb5jYKm1Jl0fM3/k0MlBAbbcTsRDsvRN+QvuDVFYZ/9\nEg93yTQWC+Ne2Ot4GtDmNTnuvl8SBkCb75ktHkyiE07116P44b2oPZ+o823HfFOk\n/wlie6jjHmtpR+Pb8vtAznE/QJDLGH+GzwGE11uSAc1nBnaVtIT2OLa1AYp7TlS5\nEfM0wbpgQHPZZcr1X+WvbZwIpZ7lixZ2HSq2eY+bT9XnDmUDeXQje9IZgcH6CW4L\n8jiUwiSf11Asj02CsvmY\n-----END CERTIFICATE-----"},"State":null,"ForceWithoutCrossSigning":false,"CreateIndex":5,"ModifyIndex":83

Yet, I keep encountering the same error:

024-07-03T17:50:58.357+0300 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=discovery-chain:db error="error filling agent cache: no cluster ca config setup"

Could you please help me understand why it’s unable to fetch the certificate?

Hi @bogdan-atl

Welcome to the HashiCorp Forums!

Could you share the Consul configuration file and the Consul Leader nodes startup logs?

Following the doc, did you end up creating a custom private key and root certificate? or just enabled connect using connect { enabled = true } config?

Also, what is the output of consul connect ca get-config command?

Hi, my configuration looks like this

Node 1 Configuration:

{
  "server": true,
  "datacenter": "dc1",
  "node_name": "node1",
  "data_dir": "/var/lib/consul",
  "bind_addr": "176.25.218.126",
  "client_addr": "0.0.0.0",
  "retry_join": ["25.265.25.45", "176.25.218.126"],
  "encrypt": "BE2FFVPwyxGhF/tnlfO4dckMoDoU1UOW/AdJ/7QkTNI=",
  "log_level": "warn",
  "enable_syslog": true,
  "enable_local_script_checks": true,
  "enable_script_checks": true,
  "leave_on_terminate": true,
  "bootstrap": true,
  "acl": {
    "enabled": false,
    "default_policy": "deny",
    "enable_token_persistence": true
  },
  "ui_config": {
    "enabled": true
  }
}

tls.hcl for Node 1:

> ca_file = "/etc/consul.d/consul-agent-ca.pem"
> cert_file = "/etc/consul.d/dc1-server-consul-0.pem"
> key_file = "/etc/consul.d/dc1-server-consul-0-key.pem"
> verify_incoming = true
> verify_outgoing = true
> verify_server_hostname = true
> auto_encrypt {
>   allow_tls = true
> }

Node 2 Configuration:

{
  "server": true,
  "datacenter": "dc1",
  "node_name": "node2",
  "data_dir": "/var/lib/consul",
  "bind_addr": "25.265.25.45",
  "client_addr": "0.0.0.0",
  "retry_join": ["25.265.25.45", "176.25.218.126"],
  "encrypt": "BE2FFVPwyxGhF/tnlfO4dckMoDoU1UOW/AdJ/7QkTNI=",
  "log_level": "warn",
  "enable_syslog": true,
  "enable_local_script_checks": true,
  "enable_script_checks": true,
  "leave_on_terminate": true,
  "acl": {
    "enabled": false,
    "default_policy": "deny",
    "enable_token_persistence": true,
    "tokens": {
      "default": "19ec34fc-1776-3c53-5ebd-d44619278942",
      "agent": "58e74001-c57e-e260-6823-ffc3a92a9f76"
    }
  },
  "ui_config": {
    "enabled": true
  }
}

tls.hcl for Node 2:


ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc1-server-consul-1.pem"
key_file = "/etc/consul.d/dc1-server-consul-1-key.pem"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
auto_encrypt {
  allow_tls = true
}

consul connect ca get-config -token “2d1c4a25-7b08-dcbd-6071-9a0c827e28ae”

{
        "Provider": "consul",
        "Config": {
                "IntermediateCertTTL": "8760h",
                "LeafCertTTL": "72h",
                "PrivateKey": "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCX5GxDjpUAx72r\nJ65tWcFuP4xX2HOVy9YgvnGWm+MnWtKMIIkUUrD68HvZ3kZAr2XEu0Lfzoup5lAz\nEucufcDM2I9piE/SRDnsPHZV9dcBemzBGm0JxESwbG7gERWJX0g/EK5b0Jj2GPmy\nqkjq6Csn6aE3e8m/YfoDaQ9A9/+4Qwyqv27QywK0iDhgJFDG+XGnCLTzmzusbVnx\nThSj1u9/IKNpC1rn34nCe6xVxoaFfMhErktcvqBYpOkzL/dLK8ivLQWY2oHgljwP\ncERKyR0BtzYdkGY8h63WxqY0Y0fMqVw3a+Sre0prHrWmBgTfqtSFtWZ9xSTnuWTE\nFeNtIzxDAgMBAAECggEABSziR3b/OrSTkNZcShuJzLDbnk6guZTlad4zTpdnKPse\nO2oMYaXjo9k3zxeUuzV+uTJMOAX4mkRfrS3StLHSNOiYEldzqM1sT9FLkug6eDFs\nrTOyTWfjk0NOj9HlMvYNJyDxX+1+mLwHHgBRVbetaMG8HYDZQe0bsKwOuf75NpT8\n0XT9Y8uOCso+ZQQwdb4Sollrrle2DJYJVjD6NaN9u49k9Ntq1/wY2rXNlOkUppDu\n5XdqYxIDRuHjqf3E2rB2moIsK4YtblwVf2FFu4FMMNGK1uO2uNdUWHc5SaDADxLo\nXXkp01Pq7+/9HyyDYwdmjfDBFD4rN70ynwi6QtoTEQKBgQC9JtEoCYgJusuV+/zT\nmyHPKCCtI/z+7ti8fe7AP69nBWh357MlJQdQNhYMTENOO4Y/QLdPqxq2fH2GdZ+x\n6PROye9qhKjar+vqL4KQLaqjl7KTCitQ3nixaTgpY864IuXCofwEQNo+SN2JfP66\nunJQzBQPGen8H2Bm5xWU6gJxiwKBgQDNkqCQv5Y5BaCGgL+RIIvP2KqT7t73WMx0\nnH5aKt3jnI5+tTxFa9Yt/dGzceCXI9WUUcWz/FeirZbfJJQvxy6AFug7nX9jB62p\nLiRgOElFd5OjJBHIJf3h9/syxZH11MJnNht240OVCQqYXZPHY5yIKDnuJbDNiVxG\naAY7/PTHKQKBgGYnSpo225N1oHG2D4swowfAAjW/0+jSkZbq5EBLpK0czJjFN4j+\nAH8fVrT5kvfzScNrbhTGsbyQazJs3/wzdY3/nR+H2bGRhPWLBxZas2d6i+TBfzdD\njycxTjV8Q65Kmo+M8BBT/gnZTFvud5vdCdP9A3BMbIPPZj7s2kj8QKmVAoGAZIha\nga9QFd7MnDqGdMDH3wOEmAfvfmMsybJmkE36aX98qDaVnRWZjrBcRzdfsEpT/s7m\nspireuENsV7AIYV1Fisds1nYnGN66AuqumuMWa6awuO21nzN5h5R+IhqUyCTWco6\nMDAAMvhS+CR7Sr1KJcQWYIKKLloIiF4C/n88bdECgYBSAvgX4PwQwZkTTe8rULTS\nK9sE4xjy7ydoLHAKgJVTmv3dzKqMIZy3cvPPACPmXd3r2jtDP0tq7JghfvViASj0\nqX+evZUk8Mp6twhLK9DwwiDGGLPyrXEWXMuH5bmBIBZfzbAwlS9AL43OK2t9P04f\np7w6uOvAhuT3zteROFUv3g==\n-----END PRIVATE KEY-----",
                "RootCert": "-----BEGIN CERTIFICATE-----\nMIIDazCCAlOgAwIBAgIUOB/1COFxAnnxD7mtuozimC44FdkwDQYJKoZIhvcNAQEL\nBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM\nGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yNDA3MDMxMzM2NTdaFw0zNDA3\nMDExMzM2NTdaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEw\nHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQCX5GxDjpUAx72rJ65tWcFuP4xX2HOVy9YgvnGWm+Mn\nWtKMIIkUUrD68HvZ3kZAr2XEu0Lfzoup5lAzEucufcDM2I9piE/SRDnsPHZV9dcB\nemzBGm0JxESwbG7gERWJX0g/EK5b0Jj2GPmyqkjq6Csn6aE3e8m/YfoDaQ9A9/+4\nQwyqv27QywK0iDhgJFDG+XGnCLTzmzusbVnxThSj1u9/IKNpC1rn34nCe6xVxoaF\nfMhErktcvqBYpOkzL/dLK8ivLQWY2oHgljwPcERKyR0BtzYdkGY8h63WxqY0Y0fM\nqVw3a+Sre0prHrWmBgTfqtSFtWZ9xSTnuWTEFeNtIzxDAgMBAAGjUzBRMB0GA1Ud\nDgQWBBT/MkU2JN+U1GGHJASQYe2GvhlKfjAfBgNVHSMEGDAWgBT/MkU2JN+U1GGH\nJASQYe2GvhlKfjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBM\nlHKZ5EdN1d6b3s8y55ouub5NxkWSdFqyMRDLjizTB/kTVPC+X3hT/XACA1CZY/vV\nj4o2A6YjqdREXzdqJf9PMFb5jYKm1Jl0fM3/k0MlBAbbcTsRDsvRN+QvuDVFYZ/9\nEg93yTQWC+Ne2Ot4GtDmNTnuvl8SBkCb75ktHkyiE07116P44b2oPZ+o823HfFOk\n/wlie6jjHmtpR+Pb8vtAznE/QJDLGH+GzwGE11uSAc1nBnaVtIT2OLa1AYp7TlS5\nEfM0wbpgQHPZZcr1X+WvbZwIpZ7lixZ2HSq2eY+bT9XnDmUDeXQje9IZgcH6CW4L\n8jiUwiSf11Asj02CsvmY\n-----END CERTIFICATE-----"
        },
        "State": null,
        "ForceWithoutCrossSigning": false,
        "CreateIndex": 5,
        "ModifyIndex": 835
}
Jul 04 13:10:31 loki consul[3455740]: 2024-07-04T13:10:31.437+0300 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=peering-trust-bundles error="error filling agent cache: failed to list all discovery chains referring to \"web\": failed to fetch discovery chain for \"web\": no cluster ca config setup"
Jul 04 13:10:31 loki consul[3455740]: agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=peering-trust-bundles error="error filling agent cache: failed to list all discovery chains referring to \"web\": failed to fetch discovery chain for \"web\": no cluster ca config setup"
Jul 04 13:10:31 loki consul[3455740]: 2024-07-04T13:10:31.438+0300 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=discovery-chain:db error="error filling agent cache: no cluster ca config setup"
Jul 04 13:10:31 loki consul[3455740]: agent.proxycfg: Failed to handle update from watch: kind=connect-proxy proxy=web-sidecar-proxy service_id=web-sidecar-proxy id=discovery-chain:db error="error filling agent cache: no cluster ca config setup"

I mentioned initially running the configuration without generating the CA, resulting in a similar error regarding the CA certificate not being found. I then used consul connect ca get-config with a specific token to resolve this, which appears to have resolved the issue related to the CA certificate.

Unfortunately, I could not reproduce your issue using the same configurations you used. I see the same error, but it only lasts during the early startup phase of the agent when I restart it.

Are you seeing this error repeating, and are you unable to launch the sidecar proxies?

I would recommend bumping up the log_level to debug and restart the agent to capture the initial startup logs (of the leader agent as the leader is the one who sets up the CA) to understand why the leader cannot initialize the CA.

I restarted the agent with debug logging on the master and noticed the following entries in the logs:

Jul 04 13:13:08 livelinux consul[3096617]: agent: (LAN) joined: number_of_nodes=1
Jul 04 13:13:08 livelinux consul[3096617]: agent: systemd notify failed: error="No socket"
Jul 04 13:13:08 livelinux consul[3096617]: agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1
Jul 04 13:13:09 livelinux consul[3096617]: 2024-07-04T13:13:09.556Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:09 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:10 livelinux consul[3096617]: 2024-07-04T13:13:10.557Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:10 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:11 livelinux consul[3096617]: 2024-07-04T13:13:11.558Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:11 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:12 livelinux consul[3096617]: 2024-07-04T13:13:12.558Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:12 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:13 livelinux consul[3096617]: 2024-07-04T13:13:13.559Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:13 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:14 livelinux consul[3096617]: 2024-07-04T13:13:14.560Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:14 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:15 livelinux consul[3096617]: 2024-07-04T13:13:15.560Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:15 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:15 livelinux consul[3096617]: 2024-07-04T13:13:15.929Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jul 04 13:13:15 livelinux consul[3096617]: agent.anti_entropy: failed to sync remote state: error="No cluster leader"
Jul 04 13:13:16 livelinux consul[3096617]: 2024-07-04T13:13:16.561Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:16 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:17 livelinux consul[3096617]: 2024-07-04T13:13:17.562Z [DEBUG] agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:17 livelinux consul[3096617]: agent.server.cert-manager: CA has not finished initializing
Jul 04 13:13:18 livelinux consul[3096617]: 2024-07-04T13:13:18.073Z [WARN]  agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=

It looks like the cluster hasn’t elected a leader yet. Wait for the leader’s election and capture the logs that show after that. A healthy leader is required for the cluster to function correctly.

It is also recommended that you run three servers minimum Consensus Protocol | Raft | Consul | HashiCorp Developer

Good afternoon, this was a temporary error… then everything worked for me, thank you. Now I have more issues with visualizing traffic in the UI. I followed all the steps to set up Prometheus and connected it to Consul with the following configuration:

"ui_config": {
  "enabled": true,
  "metrics_provider": "prometheus",
  "metrics_proxy": {
    "base_url": "http://localhost:8428"
  }
}

In the UI, after this, I get an infinite loading of metrics or an error saying “No Metrics Available.” I started examining the requests in the logs:

Jul 09 09:30:35 loki consul[1062772]: agent.http: Request finished: method=GET url="/v1/internal/ui/metrics-proxy/api/v1/query?query=sum(rate(envoy_tcp_downstream_cx_total%7Bconsul_source_service%3D%22balancer%22%2Cconsul_source_datacenter%3D%22dc1%22%2Cconsul_source_namespace%3D%22default%22%2Cenvoy_tcp_prefix%3D~%22upstream.*%22%7D%5B15m%5D))%20by%20(consul_upstream_service%2Cconsul_upstream_datacenter%2Cconsul_upstream_namespace)&time=1720506635.862" from=193.23.55.191:50008 latency=3.931082ms
Jul 09 09:30:35 loki consul[1062772]: agent.http: Request finished: method=GET url="/v1/internal/ui/metrics-proxy/api/v1/query?query=8%20*%20sum(rate(envoy_tcp_downstream_cx_rx_bytes_total%7Bconsul_source_service%3D%22balancer%22%2Cconsul_source_datacenter%3D%22dc1%22%2Cconsul_source_namespace%3D%22default%22%2Cenvoy_tcp_prefix%3D~%22upstream.*%22%7D%5B15m%5D))%20by%20(consul_upstream_service%2Cconsul_upstream_datacenter%2Cconsul_upstream_namespace)&time=1720506635.862" from=193.23.55.191:50013 latency=3.14911ms
Jul 09 09:30:35 loki consul[1062772]: 2024-07-09T09:30:35.055+0300 [DEBUG] agent.http: Request finished: method=GET url="/v1/internal/ui/metrics-proxy/api/v1/query?query=8%20*%20sum(rate(envoy_tcp_downstream_cx_rx_bytes_total%7Bconsul_source_service%3D%22balancer%22%2Cconsul_source_datacenter%3D%22dc1%22%2Cconsul_source_namespace%3D%22default%22%2Cenvoy_tcp_prefix%3D~%22upstream.*%22%7D%5B15m%5D))%20by%20(consul_upstream_service%2Cconsul_upstream_datacenter%2Cconsul_upstream_namespace)&time=1720506635.862" from=193.23.55.191:50013 latency=3.14911ms
Jul 09 09:30:35 loki consul[1062772]: 2024-07-09T09:30:35.412+0300 [DEBUG] agent.ui_metrics_proxy: proxying request: to="http://localhost:8428/api/v1/query?query=8%20*%20sum(rate(envoy_tcp_downstream_cx_tx_bytes_total%7Bconsul_source_service%3D%22balancer%22%2Cconsul_source_datacenter%3D%22dc1%22%2Cconsul_source_namespace%3D%22default%22%2Cenvoy_tcp_prefix%3D~%22upstream.*%22%7D%5B15m%5D))%20by%20(consul_upstream_service%2Cconsul_upstream_datacenter%2Cconsul_upstream_namespace)&time=1720506635.862"
Jul 09 09:30:35 loki consul[1062772]: agent.ui_metrics_proxy: proxying request: to="http://localhost:8428/api/v1/query?query=8%20*%20sum(rate(envoy_tcp_downstream_cx_tx_bytes_total%7Bconsul_source_service%3D%22balancer%22%2Cconsul_source_datacenter%3D%22dc1%22%2Cconsul_source_namespace%3D%22default%22%2Cenvoy_tcp_prefix%3D~%22upstream.*%22%7D%5B15m%5D))%20by%20(consul_upstream_service%2Cconsul_upstream_datacenter%2Cconsul_upstream_namespace)&time=1720506635.862"
Jul 09 09:30:35 loki consul[1062772]: 2024-07-09T09:30:35.417+0300 [DEBUG] agent.ui_metrics_proxy: proxying request: to="http://localhost:8428/api/v1/query?query=sum(rate(envoy_tcp_downstream_cx_no_route%7Bconsul_source_service%3D%22balancer%22%2Cconsul_source_datacenter%3D%22dc1%22%2Cconsul_source_namespace%3D%22default%22%2Cenvoy_tcp_prefix%3D~%22upstream.*%22%7D%5B15m%5D))%20by%20(consul_upstream_service%2Cconsul_upstream_datacenter%2Cconsul_upstream_namespace)&time=1720506635.862"
Jul 09 09:30:35 loki consul[1062772]: 2024-07-09T09:30:35.417+0300 [DEBUG] agent.http: Request finished: method=GET url="/v1/internal/ui/metrics-proxy/api/v1/query?query=8%20*%20sum(rate(envoy_tcp_downstream_cx_tx_bytes_total%7Bconsul_source_service%3D%22balancer%22%2Cconsul_source_datacenter%3D%22dc1%22%2Cconsul_source_namespace%3D%22default%22%2Cenvoy_tcp_prefix%3D~%22upstream.*%22%7D%5B15m%5D))%20by%20(consul_upstream_service%2Cconsul_upstream_datacenter%2Cconsul_upstream_namespace)&time=1720506635.862" from=193.23.55.191:50011 latency=5.713019ms

In Prometheus, I manually checked the PromQL queries related to Envoy and found nothing. It doesn’t send metrics from the sidecar services. I studied the documentation and found nothing about metrics from sidecar services.

@bogdan-atl , considering the original issue of this thread is solved, I humbly request you to open a new thread for the telemetry-related query.