# Ceph Volume is blocking task from starting **URL:** \ **Category:** Nomad\ **Created:** [January 30, 2023, 6:57pm UTC](https://discuss.hashicorp.com/t/ceph-volume-is-blocking-task-from-starting/49507 "2023-01-30T18:57:10Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![acziryak](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/acziryak/32/31559_2.png) [@acziryak](https://discuss.hashicorp.com/u/acziryak)\ **Post date:** [January 30, 2023, 6:57pm UTC](https://discuss.hashicorp.com/t/ceph-volume-is-blocking-task-from-starting/49507/1 "2023-01-30T18:57:10Z")
I’m failing to deploy a container that is attempting to use a volume that’s using a `ceph_csi` plugin: ```auto failed to setup alloc: pre-run hook "csi_hook" failed: node plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = failed to establish the connection: failed to get connection: connecting failed: rados: ret=-110, Connection timed out ``` I see this in the nomad logs: ```auto Jan 28 00:22:01 ind-test-nomad-worker11 nomad[208094]: 2023-01-28T00:22:01.022-0500 [WARN] client.ceph-csi: finished client unary call: grpc.code=Internal duration=50m0.016328065s grpc.service=csi.v1.Node grpc.method=NodeStageVolume Jan 28 00:22:01 ind-test-nomad-worker11 nomad[208094]: 2023-01-28T00:22:01.022-0500 [ERROR] client.alloc_runner: prerun failed: alloc_id=b6c05535-82b8-f5d7-a65f-0960daf0b087 error="pre-run hook \"csi_hook\" failed: node plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = failed to establish the connection: failed to get connection: connecting failed: rados: ret=-110, Connection timed out" ``` I see this in the ceph node container logs: ```auto I0128 05:30:42.057159 7 utils.go:195] ID: 5090 Req-ID: 0001-0024-1e35f6bc-1257-45b6-aa9d-16f9ecd30652-0000000000000024-dc31d910-9ecc-11ed-b9a0-9238a2ead1d6 GRPC call: /csi.v1.Node/NodeStageVolume I0128 05:30:42.057365 7 utils.go:206] ID: 5090 Req-ID: 0001-0024-1e35f6bc-1257-45b6-aa9d-16f9ecd30652-0000000000000024-dc31d910-9ecc-11ed-b9a0-9238a2ead1d6 GRPC request: {"secrets":" ***stripped***","staging_target_path":"/local/csi/staging/prometheus-us-ind-test/rw-file-system-single-node-writer","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"clusterID":" *********************","imageFeatures":"layering","imageName":"csi-vol-dc31d910-9ecc-11ed-b9a0-9238a2ead1d6","journalPool":"ind-nonprod2","pool":"ind-nonprod2"},"volume_id":"0001-0024-1e35f6bc-1257-45b6-aa9d-16f9ecd30652-0000000000000024-dc31d910-9ecc-11ed-b9a0-9238a2ead1d6"} I0128 05:30:42.057659 7 rbd_util.go:1279] ID: 5090 Req-ID: 0001-0024-1e35f6bc-1257-45b6-aa9d-16f9ecd30652-0000000000000024-dc31d910-9ecc-11ed-b9a0-9238a2ead1d6 setting disableInUseChecks: false image features: [layering] mounter: rbd ``` FWIW, I am able to manually mount this via ```auto rbd device map ind-nonprod2/test_image --id ind-nonprod2 --keyfile ceph_secret.txt ``` The thing that’s _really_ leaving me scratching my head is that the allocation isn’t even starting, there is no task for the job. Presumably this is because the volume is not registered even though it says it’s registered in the nomad UI. I’m sure I’m missing something here, but I’m not sure where to start. ---
**Author:** ![acziryak](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/acziryak/32/31559_2.png) [@acziryak](https://discuss.hashicorp.com/u/acziryak)\ **Post date:** [February 3, 2023, 5:53pm UTC](https://discuss.hashicorp.com/t/ceph-volume-is-blocking-task-from-starting/49507/2 "2023-02-03T17:53:11Z")
Let me back up a bit. Here are the logs from trying to do the `rbd device map...` command inside of the node container: > **Logs** > > ```auto > 2023-02-03T17:49:11.903+0000 7f8a94117700 2 Event(0x55c0b8ec92e0 nevent=5000 time_id=1).set_owner center_id=0 owner=140233166386944 > 2023-02-03T17:49:11.903+0000 7f8a93916700 2 Event(0x55c0b8ecb390 nevent=5000 time_id=1).set_owner center_id=1 owner=140233157994240 > 2023-02-03T17:49:11.903+0000 7f8a93115700 2 Event(0x55c0b8ecc380 nevent=5000 time_id=1).set_owner center_id=2 owner=140233149601536 > 2023-02-03T17:49:11.903+0000 7f8a95e0d380 1 Processor -- start > 2023-02-03T17:49:11.903+0000 7f8a95e0d380 1 -- start start > 2023-02-03T17:49:11.903+0000 7f8a95e0d380 1 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect > 2023-02-03T17:49:11.903+0000 7f8a95e0d380 1 -- --> v1:10.2.64.130:6789/0 -- auth(proto 0 37 bytes epoch 0) v1 -- 0x55c0b8df4ad0 con 0x55c0b8f60f50 > 2023-02-03T17:49:11.903+0000 7f8a95e0d380 1 -- --> v1:10.2.64.132:6789/0 -- auth(proto 0 37 bytes epoch 0) v1 -- 0x55c0b8de8b40 con 0x55c0b8f5d2d0 > 2023-02-03T17:49:11.903+0000 7f8a95e0d380 1 -- --> v2:10.2.64.130:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c0b8e47470 con 0x55c0b8f5a950 > 2023-02-03T17:49:11.903+0000 7f8a95e0d380 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x55c0b8e47470 type=5 mon_getmap magic: 0 v1 > 2023-02-03T17:49:11.903+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:11.903+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:11.903+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:11.903+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:11.903+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:11.903+0000 7f8a93916700 1 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 > 2023-02-03T17:49:11.903+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:11.903+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:12.107+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:12.107+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:12.107+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:12.107+0000 7f8a93916700 1 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 > 2023-02-03T17:49:12.107+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:12.107+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:12.107+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:12.107+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:12.507+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:12.507+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:12.507+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:12.507+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:12.507+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:12.507+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:12.507+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:12.507+0000 7f8a93916700 1 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 > 2023-02-03T17:49:13.311+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:13.311+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:13.311+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:13.311+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:13.311+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:13.311+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:13.311+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:13.311+0000 7f8a93916700 1 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 -- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 msgr2=0x55c0b8f5cd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop > 2023-02-03T17:49:14.907+0000 7f8a91912700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 -- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 legacy=0x55c0b8f5f6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down > 2023-02-03T17:49:14.907+0000 7f8a91912700 2 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop > 2023-02-03T17:49:14.907+0000 7f8a91912700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:14.907+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers > 2023-02-03T17:49:14.907+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state (warped) reseting crypto and compression handlers > 2023-02-03T17:49:14.907+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:14.907+0000 7f8a93916700 5 --2- >> v2:10.2.64.130:3300/0 conn(0x55c0b8f5a950 0x55c0b8f5cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:14.907+0000 7f8a93115700 5 --1- >> v1:10.2.64.132:6789/0 conn(0x55c0b8f5d2d0 0x55c0b8f5f6b0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 -- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 legacy=0x55c0b8f63400 unknown :-1 s=STATE_CONNECTING l=0).mark_down > 2023-02-03T17:49:14.907+0000 7f8a91912700 2 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop > 2023-02-03T17:49:14.907+0000 7f8a91912700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:14.907+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers > 2023-02-03T17:49:14.907+0000 7f8a94117700 5 --1- >> v1:10.2.64.130:6789/0 conn(0x55c0b8f60f50 0x55c0b8f63400 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect > 2023-02-03T17:49:14.907+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:14.907+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:14.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:14.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:14.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:14.907+0000 7f8a93115700 1 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 > 2023-02-03T17:49:14.907+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:14.907+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 -- --> v1:10.2.64.128:6789/0 -- auth(proto 0 37 bytes epoch 0) v1 -- 0x55c0b8de8930 con 0x7f8a7c008850 > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 -- --> v1:10.2.64.131:6789/0 -- auth(proto 0 37 bytes epoch 0) v1 -- 0x55c0b8df4ad0 con 0x7f8a7c002060 > 2023-02-03T17:49:14.907+0000 7f8a91912700 1 -- --> v2:10.2.64.131:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c0b8e47470 con 0x7f8a7c005cb0 > 2023-02-03T17:49:14.907+0000 7f8a91912700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x55c0b8e47470 type=5 mon_getmap magic: 0 v1 > 2023-02-03T17:49:15.107+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:15.107+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:15.107+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:15.107+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:15.107+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:15.107+0000 7f8a93115700 1 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 > 2023-02-03T17:49:15.107+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:15.107+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:15.511+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:15.511+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:15.511+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:15.511+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:15.511+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:15.511+0000 7f8a93115700 1 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 > 2023-02-03T17:49:15.511+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:15.511+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:16.311+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:16.311+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:16.311+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:16.311+0000 7f8a93115700 1 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 > 2023-02-03T17:49:16.311+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:16.311+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:16.311+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:16.311+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 -- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 msgr2=0x7f8a7c008160 unknown :-1 s=STATE_CONNECTING l=0).mark_down > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop > 2023-02-03T17:49:17.907+0000 7f8a91912700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 -- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 legacy=0x7f8a7c0044a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down > 2023-02-03T17:49:17.907+0000 7f8a91912700 2 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop > 2023-02-03T17:49:17.907+0000 7f8a91912700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:17.907+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers > 2023-02-03T17:49:17.907+0000 7f8a93916700 5 --1- >> v1:10.2.64.131:6789/0 conn(0x7f8a7c002060 0x7f8a7c0044a0 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:17.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state (warped) reseting crypto and compression handlers > 2023-02-03T17:49:17.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:17.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.131:3300/0 conn(0x7f8a7c005cb0 0x7f8a7c008160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 -- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 legacy=0x7f8a7c00ad00 unknown :-1 s=STATE_CONNECTING l=0).mark_down > 2023-02-03T17:49:17.907+0000 7f8a91912700 2 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).stop > 2023-02-03T17:49:17.907+0000 7f8a91912700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=START_CONNECT pgs=0 cs=0 l=0).reset_recv_state > 2023-02-03T17:49:17.907+0000 7f8a94117700 1 -- reap_dead start > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect > 2023-02-03T17:49:17.907+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:17.907+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:17.907+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:17.907+0000 7f8a93916700 1 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 -- reap_dead delete 0x55c0b8f5a950 > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 -- reap_dead delete 0x55c0b8f5d2d0 > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 -- reap_dead delete 0x55c0b8f60f50 > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 -- reap_dead delete 0x7f8a7c002060 > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 -- reap_dead delete 0x7f8a7c005cb0 > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 -- reap_dead delete 0x7f8a7c008850 > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_recv_state (warped) reseting security handlers > 2023-02-03T17:49:17.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 --1- >> v1:10.2.64.128:6789/0 conn(0x7f8a7c008850 0x7f8a7c00ad00 :-1 s=CLOSED pgs=0 cs=0 l=0).reset_security > 2023-02-03T17:49:17.907+0000 7f8a94117700 1 -- reap_dead start > 2023-02-03T17:49:17.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:17.907+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:17.907+0000 7f8a93115700 1 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:17.907+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:17.907+0000 7f8a94117700 1 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 -- --> v2:10.2.64.128:3300/0 -- mon_getmap magic: 0 v1 -- 0x55c0b8e47470 con 0x7f8a7c00e120 > 2023-02-03T17:49:17.907+0000 7f8a91912700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x55c0b8e47470 type=5 mon_getmap magic: 0 v1 > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 -- --> v2:10.2.64.129:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7c00d2b0 con 0x7f8a7c00d6b0 > 2023-02-03T17:49:17.907+0000 7f8a91912700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x7f8a7c00d2b0 type=5 mon_getmap magic: 0 v1 > 2023-02-03T17:49:17.907+0000 7f8a91912700 1 -- --> v2:10.2.64.132:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a7c00d3f0 con 0x7f8a7c00cc30 > 2023-02-03T17:49:17.907+0000 7f8a91912700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).send_message enqueueing message m=0x7f8a7c00d3f0 type=5 mon_getmap magic: 0 v1 > 2023-02-03T17:49:18.111+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:18.111+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:18.111+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:18.111+0000 7f8a93916700 1 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 > 2023-02-03T17:49:18.111+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:18.111+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:18.111+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:18.111+0000 7f8a94117700 1 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 > 2023-02-03T17:49:18.111+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:18.111+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:18.111+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:18.111+0000 7f8a93115700 1 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 > 2023-02-03T17:49:18.515+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:18.515+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:18.515+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:18.515+0000 7f8a93916700 1 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 > 2023-02-03T17:49:18.515+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:18.515+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:18.515+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:18.515+0000 7f8a94117700 1 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 > 2023-02-03T17:49:18.515+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:18.515+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:18.515+0000 7f8a93115700 5 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:18.515+0000 7f8a93115700 1 --2- >> v2:10.2.64.128:3300/0 conn(0x7f8a7c00e120 0x7f8a7c00c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 > 2023-02-03T17:49:19.315+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:19.315+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > 2023-02-03T17:49:19.315+0000 7f8a93916700 5 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_compression > 2023-02-03T17:49:19.315+0000 7f8a93916700 1 --2- >> v2:10.2.64.129:3300/0 conn(0x7f8a7c00d6b0 0x7f8a7c00da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 > 2023-02-03T17:49:19.315+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_recv_state > 2023-02-03T17:49:19.315+0000 7f8a94117700 5 --2- >> v2:10.2.64.132:3300/0 conn(0x7f8a7c00cc30 0x7f8a7c0148a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).reset_security > > ``` Maybe it’s my untrained eye, but there’s nothing in there that shows me any failure. Additionally, there’s no `ping` or `traceroute` inside of the ceph-csi container that I can use to do network troubleshooting. I have to conclude that it’s silently failing somewhere, but where? ---
**Author:** ![acziryak](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/acziryak/32/31559_2.png) [@acziryak](https://discuss.hashicorp.com/u/acziryak)\ **Post date:** [February 17, 2023, 6:46pm UTC](https://discuss.hashicorp.com/t/ceph-volume-is-blocking-task-from-starting/49507/3 "2023-02-17T18:46:55Z")
I’ve created a new volume, and am now getting this error: ```auto failed to setup alloc: pre-run hook "csi_hook" failed: node plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = mount failed: exit status 1 Mounting command: mount Mounting arguments: -t ext4 -o _netdev,noatime,defaults /dev/rbd0 /local/csi/staging/agnostic2-us-ind-test/rw-file-system-single-node-writer/0001-0024-1e35f6bc-1257-45b6-aa9d-16f9ecd30652-0000000000000024-9f0760b0-aee6-11ed-96f6-ae765b511692 Output: mount: only root can use "--options" option (effective UID is 100000) ``` Is this an issue with namespace mapping? ---
**Author:** ![acziryak](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/acziryak/32/31559_2.png) [@acziryak](https://discuss.hashicorp.com/u/acziryak)\ **Post date:** [February 17, 2023, 6:55pm UTC](https://discuss.hashicorp.com/t/ceph-volume-is-blocking-task-from-starting/49507/4 "2023-02-17T18:55:18Z")
Also, is there some place that Nomad keeps metadata for volumes? I went to delete the volume that was previously giving me trouble, but when I go to recreate it with the same name and ID, it immediately fails. Other volumes are able to be created, but this one is not able to be recreated. I can’t find it in the UI or with CLI tools, but I have a suspicion that there is a cache that needs busting here… ---
**Author:** ![mmeier86](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/mmeier86/32/24773_2.png) [@mmeier86](https://discuss.hashicorp.com/u/mmeier86)\ **Post date:** [February 18, 2023, 9:43pm UTC](https://discuss.hashicorp.com/t/ceph-volume-is-blocking-task-from-starting/49507/5 "2023-02-18T21:43:54Z")
Let’s start with the recreation issue. Did you also delete the volume in Ceph itself? The ID is part of the volume name in Ceph. Perhaps that’s where the error is coming from? For the mounting issue. If I remember correctly, Ceph CSI plugin jobs need to run as privileged containers. What about the Ceph auth you are using? When testing the manual mount, did you use exactly the same credentials as in the Nomad volume file? I am afraid I am not seeing anything obvious in the logs. What about the Ceph auth config? Did you happen to restrict the user you are using for Nomad to a specific network? ---
**Author:** ![acziryak](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/acziryak/32/31559_2.png) [@acziryak](https://discuss.hashicorp.com/u/acziryak)\ **Post date:** [February 20, 2023, 5:41pm UTC](https://discuss.hashicorp.com/t/ceph-volume-is-blocking-task-from-starting/49507/6 "2023-02-20T17:41:50Z")
The recreation issue was solved somwhere between restarting the CSI node & controller containers, and a `nomad system gc` and `nomad system reconcile summaries`. Now that I have that volume again, the job hangs when starting. I get no logs from the controller, the node, the job, or the nomad daemon about why it won’t start. There’s just… nothing. Also, in testing the container WITHOUT the volume, it starts up immediately and healthily.