Consul agent start fails with exit code 1

Consul
1 
consul agent -config-dir ~/consul-config/server -ui-dir ~/consul-ui -bootstrap true -client=0.0.0.0

While starting consul with following command, i get following logs

==> WARNING: LAN keyring exists but -encrypt given, using keyring
==> WARNING: WAN keyring exists but -encrypt given, using keyring
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Starting Consul agent RPC...
==> Consul agent running!
         Node name: '192.168.1.5'
        Datacenter: 'dc1'
            Server: true (bootstrap: true)
       Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600, RPC: 8400)
      Cluster Addr: 192.168.1.5 (LAN: 8301, WAN: 8302)
    Gossip encrypt: true, RPC-TLS: false, TLS-Incoming: false
             Atlas: <disabled>

==> Log data will now stream in as it occurs:

    2020/05/18 12:38:30 [INFO] serf: EventMemberJoin: 192.168.1.5 192.168.1.5
    2020/05/18 12:38:30 [INFO] serf: EventMemberJoin: 192.168.1.5.dc1 192.168.1.5
    2020/05/18 12:38:30 [INFO] raft: Node at 192.168.1.5:8300 [Follower] entering Follower state
    2020/05/18 12:38:30 [WARN] serf: Failed to re-join any previously known node
    2020/05/18 12:38:30 [INFO] consul: adding LAN server 192.168.1.5 (Addr: 192.168.1.5:8300) (DC: dc1)
    2020/05/18 12:38:30 [WARN] serf: Failed to re-join any previously known node
    2020/05/18 12:38:30 [INFO] consul: adding WAN server 192.168.1.5.dc1 (Addr: 192.168.1.5:8300) (DC: dc1)
    2020/05/18 12:38:30 [ERR] agent: failed to sync remote state: No cluster leader
==> Newer Consul version available: 1.7.3
    2020/05/18 12:38:32 [WARN] raft: Heartbeat timeout reached, starting election
    2020/05/18 12:38:32 [INFO] raft: Node at 192.168.1.5:8300 [Candidate] entering Candidate state
    2020/05/18 12:38:32 [DEBUG] raft: Votes needed: 1
    2020/05/18 12:38:32 [DEBUG] raft: Vote granted from 192.168.1.5:8300. Tally: 1
    2020/05/18 12:38:32 [INFO] raft: Election won. Tally: 1
    2020/05/18 12:38:32 [INFO] raft: Node at 192.168.1.5:8300 [Leader] entering Leader state
    2020/05/18 12:38:32 [INFO] consul: cluster leadership acquired
    2020/05/18 12:38:32 [INFO] consul: New leader elected: 192.168.1.5
    2020/05/18 12:38:32 [INFO] raft: Disabling EnableSingleNode (bootstrap)
    2020/05/18 12:38:32 [ERR] raft: Got unrecognized log type: &raft.Log{Index:0x1, Term:0x1, Type:0x5, Data:[]uint8{0x81, 0xa7, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x73, 0x91, 0x83, 0xa7, 0x41, 0x64, 0x64, 0x72, 0x65, 0x73, 0x73, 0xb0, 0x31, 0x39, 0x32, 0x2e, 0x31, 0x36, 0x38, 0x2e, 0x31, 0x2e, 0x35, 0x3a, 0x38, 0x33, 0x30, 0x30, 0xa2, 0x49, 0x44, 0xda, 0x0, 0x24, 0x65, 0x33, 0x35, 0x65, 0x61, 0x33, 0x39, 0x37, 0x2d, 0x36, 0x64, 0x30, 0x38, 0x2d, 0x62, 0x62, 0x64, 0x66, 0x2d, 0x38, 0x61, 0x64, 0x30, 0x2d, 0x61, 0x66, 0x36, 0x30, 0x65, 0x38, 0x34, 0x31, 0x39, 0x30, 0x31, 0x35, 0xa8, 0x53, 0x75, 0x66, 0x66, 0x72, 0x61, 0x67, 0x65, 0x0}, peer:""}
    2020/05/18 12:38:32 [DEBUG] raft: Node 192.168.1.5:8300 updated peer set (2): [192.168.1.5:8300]
    2020/05/18 12:38:32 [DEBUG] raft: Node 192.168.1.5:8300 updated peer set (2): [192.168.1.5:8300]
panic: failed to apply request: []byte{0x13, 0x81, 0xa8, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x69, 0x65, 0x73, 0x91, 0x89, 0xab, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x0, 0xab, 0x44, 0x61, 0x74, 0x61, 0x63, 0x65, 0x6e, 0x74, 0x65, 0x72, 0x73, 0xc0, 0xab, 0x44, 0x65, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0xda, 0x0, 0x2b, 0x42, 0x75, 0x69, 0x6c, 0x74, 0x69, 0x6e, 0x20, 0x50, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x74, 0x68, 0x61, 0x74, 0x20, 0x67, 0x72, 0x61, 0x6e, 0x74, 0x73, 0x20, 0x75, 0x6e, 0x6c, 0x69, 0x6d, 0x69, 0x74, 0x65, 0x64, 0x20, 0x61, 0x63, 0x63, 0x65, 0x73, 0x73, 0xa4, 0x48, 0x61, 0x73, 0x68, 0xda, 0x0, 0x20, 0xb3, 0x2, 0x10, 0xb7, 0xab, 0xa9, 0xfa, 0xcd, 0x1c, 0x57, 0x89, 0x1e, 0x3d, 0xf2, 0x76, 0x69, 0x17, 0x4a, 0x8, 0xb6, 0x90, 0xcb, 0x29, 0x5, 0xe0, 0x79, 0x75, 0x35, 0xf7, 0x5e, 0xba, 0x69, 0xa2, 0x49, 0x44, 0xda, 0x0, 0x24, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x2d, 0x30, 0x30, 0x30, 0x30, 0x2d, 0x30, 0x30, 0x30, 0x30, 0x2d, 0x30, 0x30, 0x30, 0x30, 0x2d, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x31, 0xab, 0x4d, 0x6f, 0x64, 0x69, 0x66, 0x79, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x0, 0xa4, 0x4e, 0x61, 0x6d, 0x65, 0xb1, 0x67, 0x6c, 0x6f, 0x62, 0x61, 0x6c, 0x2d, 0x6d, 0x61, 0x6e, 0x61, 0x67, 0x65, 0x6d, 0x65, 0x6e, 0x74, 0xa5, 0x52, 0x75, 0x6c, 0x65, 0x73, 0xda, 0x1, 0x54, 0xa, 0x61, 0x63, 0x6c, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x61, 0x67, 0x65, 0x6e, 0x74, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x22, 0x22, 0x20, 0x7b, 0xa, 0x9, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x7d, 0xa, 0x65, 0x76, 0x65, 0x6e, 0x74, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x22, 0x22, 0x20, 0x7b, 0xa, 0x9, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x7d, 0xa, 0x6b, 0x65, 0x79, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x22, 0x22, 0x20, 0x7b, 0xa, 0x9, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x7d, 0xa, 0x6b, 0x65, 0x79, 0x72, 0x69, 0x6e, 0x67, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x6e, 0x6f, 0x64, 0x65, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x22, 0x22, 0x20, 0x7b, 0xa, 0x9, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x7d, 0xa, 0x6f, 0x70, 0x65, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x71, 0x75, 0x65, 0x72, 0x79, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x22, 0x22, 0x20, 0x7b, 0xa, 0x9, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x7d, 0xa, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x22, 0x22, 0x20, 0x7b, 0xa, 0x9, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x9, 0x69, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x7d, 0xa, 0x73, 0x65, 0x73, 0x73, 0x69, 0x6f, 0x6e, 0x5f, 0x70, 0x72, 0x65, 0x66, 0x69, 0x78, 0x20, 0x22, 0x22, 0x20, 0x7b, 0xa, 0x9, 0x70, 0x6f, 0x6c, 0x69, 0x63, 0x79, 0x20, 0x3d, 0x20, 0x22, 0x77, 0x72, 0x69, 0x74, 0x65, 0x22, 0xa, 0x7d, 0xa6, 0x53, 0x79, 0x6e, 0x74, 0x61, 0x78, 0x0}

goroutine 16 [running]:
panic(0xc75f20, 0xc820448a50)
        /goroot/src/runtime/panic.go:464 +0x3e6
github.com/hashicorp/consul/consul.(*consulFSM).Apply(0xc820155180, 0xc820421fc0, 0x0, 0x0)
        /gopath/src/github.com/hashicorp/consul/consul/fsm.go:101 +0x3fd
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.(*Raft).runFSM(0xc820110540)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft/raft.go:564 +0xeb1
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.(*Raft).(github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.runFSM)-fm()
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft/raft.go:253 +0x20
github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc820110540, 0xc820137020)
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft/state.go:152 +0x44
created by github.com/hashicorp/consul/vendor/github.com/hashicorp/raft.(*raftState).goFunc
        /gopath/src/github.com/hashicorp/consul/vendor/github.com/hashicorp/raft/state.go:153 +0x4d

Here is my config file

{
“bootstrap”: true,
“server”: true,
“log_level”: “DEBUG”,
“enable_syslog”: true,
“datacenter”: “dc1”,
“addresses” : {
“http”: “0.0.0.0”
},
“bind_addr”: “p.p.p.p”,
“node_name”: “p.p.p.p”,
“data_dir”: “/home/user/consuldata”,
“ui_dir”: “/home/user/consul-ui”,
“acl_datacenter”: “dc1”,
“acl_master_token”: “4f3fa35a-h3ab-45a2-914f-f82a34fab0c45”,
“acl_default_policy”: “deny”,
“encrypt”: “pXoaLOJ816mO+da8y8zrsg==”
}

Please help to resolve this issue.

2

Hi @chaudharydevender95,

Thanks for posting, and welcome to the forums! I’ve got a few questions to help run this down.

  • What version of Consul are you running? How many servers are you configuring?
  • Have you gone through the Bootstrap Guide we posted?
  • What are the non-bootstrapped servers reporting in their logs?

Unfortunately, I don’t know the go panics by heart, but I think that some adjustments to your set-up may be the best route to go. Can you try the guide and see if that panic is consistently repeatable?

Looking forward to hearing back from you!