Hello all,
We are using the nomad and consul to self host all out services. When we are communicating between 2 services via consul, consul is getting restarting that is having a large payload and taking more than 40 secs to finish. Can some one help me to config the consul to allow these kind of requests on consul or better way to handle this.
Logs:
Jan 23 10:27:16 [HOSTNAME] consul[246585]: 2025-01-23T10:27:16.952Z [INFO] agent.server.raft.logstore.verifier: verification checksum OK: elapsed="750.688µs" leaderChecksum=6451bcc1a82b6118 rangeEnd=204644 rangeStart=204615 readChecksum=6451bcc1a82b6118
Jan 23 10:27:36 [HOSTNAME] systemd[1]: consul.service: start operation timed out. Terminating.
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent: Caught: signal=terminated
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent: Graceful shutdown disabled. Exiting
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent: Requesting shutdown
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.server: shutting down server
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="federation state pruning"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="peering deferred deletion"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="CA root pruning"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="CA root expiration metric"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="CA signing expiration metric"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="raft log verifier"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="federation state anti-entropy"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="streaming peering resources"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.api_gateway_controller: error watching controller subscription: controller=apiGatewayController topic=HTTPRoute err="subscription closed by server, server is shutting down"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.v2_exports_shim: encountered an error while streaming exported services: error="subscription closed by server, server is shutting down"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopped routine: routine="CA root expiration metric"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.controller-runtime: error received from watch: controller=consul.io/reaper-controller managed_type=internal.v1.Tombstone error="rpc error: code = Canceled desc = context canceled"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.api_gateway_controller: error watching controller subscription: controller=apiGatewayController topic=TCPRoute err="subscription closed by server, server is shutting down"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="metrics for streaming peering resources"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="intermediate cert renew watch"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopping routine: routine="config entry controllers"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [WARN] agent.server.serf.lan: serf: Shutdown without a Leave
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.server.peering_metrics: stopping peering metrics
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="raft log verifier"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="peering deferred deletion"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="federation state anti-entropy"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="federation state pruning"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="streaming peering resources"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="metrics for streaming peering resources"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="intermediate cert renew watch"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopped routine: routine="config entry controllers"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.api_gateway_controller: error watching controller subscription: controller=apiGatewayController topic=BoundAPIGateway err="subscription closed by server, server is shutting down"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopped routine: routine="metrics for streaming peering resources"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopped routine: routine="federation state anti-entropy"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopped routine: routine="federation state pruning"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.api_gateway_controller: error watching controller subscription: controller=apiGatewayController topic=InlineCertificate err="subscription closed by server, server is shutting down"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopped routine: routine="peering deferred deletion"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.api_gateway_controller: error watching controller subscription: controller=apiGatewayController topic=APIGateway err="subscription closed by server, server is shutting down"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopped routine: routine="CA root pruning"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.controller-runtime: error received from watch: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices error="rpc error: code = Canceled desc = context canceled"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.controller-runtime: error received from watch: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices error="rpc error: code = Canceled desc = context canceled"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.controller-runtime: error received from watch: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices error="rpc error: code = Canceled desc = context canceled"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.controller-runtime: error received from watch: controller=consul.io/exported-services-v1-compat managed_type=multicluster.v2.ComputedExportedServices error="rpc error: code = Canceled desc = context canceled"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopped routine: routine="raft log verifier"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopped routine: routine="streaming peering resources"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [INFO] agent.leader: stopped routine: routine="CA signing expiration metric"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.540Z [WARN] agent.server.api_gateway_controller: error watching controller subscription: controller=apiGatewayController topic=JWTProvider err="subscription closed by server, server is shutting down"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopped routine: routine="intermediate cert renew watch"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.leader: stopping routine: routine="CA root pruning"
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.server.autopilot: reconciliation now disabled
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [WARN] agent.server.serf.wan: serf: Shutdown without a Leave
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.router.manager: shutting down
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.541Z [INFO] agent.router.manager: shutting down
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent: consul server down
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent: shutdown complete
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent.dns: Stopping server: protocol=DNS address=0.0.0.0:8600 network=tcp
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent.dns: Stopping server: protocol=DNS address=0.0.0.0:8600 network=udp
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent: Stopping server: address=[::]:8500 network=tcp protocol=http
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent: Waiting for endpoints to shut down
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent: Endpoints down
Jan 23 10:27:36 [HOSTNAME] consul[246585]: 2025-01-23T10:27:36.542Z [INFO] agent: Exit code: code=1
Jan 23 10:27:36 [HOSTNAME] systemd[1]: consul.service: Main process exited, code=exited, status=1/FAILURE
Jan 23 10:27:36 [HOSTNAME] systemd[1]: consul.service: Failed with result 'timeout'.
Jan 23 10:27:36 [HOSTNAME] systemd[1]: Failed to start consul.service - "HashiCorp Consul - A service mesh solution".
Jan 23 10:27:36 [HOSTNAME] systemd[1]: consul.service: Consumed 3.002s CPU time.
Jan 23 10:27:36 [HOSTNAME] systemd[1]: consul.service: Scheduled restart job, restart counter is at 1.
Jan 23 10:27:36 [HOSTNAME] systemd[1]: Starting consul.service - "HashiCorp Consul - A service mesh solution"...
Jan 23 10:27:36 [HOSTNAME] consul[246845]: ==> Starting Consul agent...
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Version: '1.20.2'
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Build Date: '2025-01-03 14:38:40 +0000 UTC'
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Node ID: '8cc1142f-5350-aac4-6d14-84013c8e056e'
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Node name: '[HOSTNAME]'
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Datacenter: 'dc-1' (Segment: '<all>')
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Server: true (Bootstrap: true)
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: 8503, DNS: 8600)
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Gossip Encryption: false
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Auto-Encrypt-TLS: false
Jan 23 10:27:36 [HOSTNAME] consul[246845]: ACL Enabled: false
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Reporting Enabled: false
Jan 23 10:27:36 [HOSTNAME] consul[246845]: ACL Default Policy: allow
Jan 23 10:27:36 [HOSTNAME] consul[246845]: HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
Jan 23 10:27:36 [HOSTNAME] consul[246845]: gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
Jan 23 10:27:36 [HOSTNAME] consul[246845]: Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2
Jan 23 10:27:36 [HOSTNAME] consul[246845]: ==> Log data will now stream in as it occurs:
Jan 23 10:27:36 [HOSTNAME] consul[246845]: 2025-01-23T10:27:36.930Z [WARN] agent: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Jan 23 10:27:36 [HOSTNAME] consul[246845]: 2025-01-23T10:27:36.930Z [WARN] agent: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Jan 23 10:27:36 [HOSTNAME] consul[246845]: 2025-01-23T10:27:36.930Z [WARN] agent: bootstrap = true: do not enable unless necessary
Jan 23 10:27:36 [HOSTNAME] consul[246845]: 2025-01-23T10:27:36.957Z [WARN] agent.auto_config: skipping file /etc/consul.d/consul.env, extension must be .hcl or .json, or config format must be set
Jan 23 10:27:36 [HOSTNAME] consul[246845]: 2025-01-23T10:27:36.957Z [WARN] agent.auto_config: BootstrapExpect is set to 1; this is the same as Bootstrap mode.
Jan 23 10:27:36 [HOSTNAME] consul[246845]: 2025-01-23T10:27:36.957Z [WARN] agent.auto_config: bootstrap = true: do not enable unless necessary
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.152Z [INFO] agent.server.raft: starting restore from snapshot: id=14398-196659-1737574284566 last-index=196659 last-term=14398 size-in-bytes=40226
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.156Z [INFO] agent.server.raft: snapshot restore progress: id=14398-196659-1737574284566 last-index=196659 last-term=14398 size-in-bytes=40226 read-bytes=40226 percent-complete="100.00%"
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.156Z [INFO] agent.server.raft: restored from snapshot: id=14398-196659-1737574284566 last-index=196659 last-term=14398 size-in-bytes=40226
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.258Z [INFO] agent.server.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:8cc1142f-5350-aac4-6d14-84013c8e056e Address:127.0.0.1:8300}]"
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.258Z [INFO] agent.server.raft: entering follower state: follower="Node at 127.0.0.1:8300 [Follower]" leader-address= leader-id=
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.260Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: [HOSTNAME].dc-1 127.0.0.1
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.260Z [WARN] agent.server.serf.wan: serf: Failed to re-join any previously known node
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.260Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: [HOSTNAME] 127.0.0.1
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.260Z [INFO] agent.router: Initializing LAN area manager
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.260Z [WARN] agent.server.serf.lan: serf: Failed to re-join any previously known node
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.261Z [INFO] agent.server: Handled event for server in area: event=member-join server=[HOSTNAME].dc-1 area=wan
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.261Z [INFO] agent.server: Adding LAN server: server="[HOSTNAME] (Addr: tcp/127.0.0.1:8300) (DC: dc-1)"
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.261Z [INFO] agent.server.autopilot: reconciliation now disabled
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.262Z [INFO] agent.server.cert-manager: initialized server certificate management
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.268Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.268Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/artist
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/executive
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/album
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/festival
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/exportedservices
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/namespaceexportedservices
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/partitionexportedservices
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/multicluster/v2/computedexportedservices
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/album
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/link
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/hcp/v2/telemetrystate
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/recordlabel
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v1/concept
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/demo/v2/artist
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent.http: Registered resource endpoint: endpoint=/api/internal/v1/tombstone
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.269Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.270Z [INFO] agent: Started gRPC listeners: port_name=grpc_tls address=[::]:8503 network=tcp
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.270Z [INFO] agent: started state syncer
Jan 23 10:27:37 [HOSTNAME] consul[246845]: 2025-01-23T10:27:37.270Z [INFO] agent: Consul agent running!