# Nomad - remove down or left nodes from cluster **URL:** \ **Category:** Nomad\ **Created:** [January 18, 2022, 1:44pm UTC](https://discuss.hashicorp.com/t/nomad-remove-down-or-left-nodes-from-cluster/34393 "2022-01-18T13:44:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![voxsim](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/voxsim/32/25215_2.png) [@voxsim](https://discuss.hashicorp.com/u/voxsim)\ **Post date:** [January 18, 2022, 1:44pm UTC](https://discuss.hashicorp.com/t/nomad-remove-down-or-left-nodes-from-cluster/34393/1 "2022-01-18T13:44:47Z")
We set up a Nomad cluster with a Consul cluster following the best practices provided in the documentation on top of Google cloud infrastructure. So far so good. We did some tests, but then we have a lot of empty clients when we go on Topology in the Nomad UI Interface ![Screenshot 2022-01-18 at 14.41.53](https://global.discourse-cdn.com/hashicorp/original/3X/e/3/e3bcd522c40e2b30512e246886ab5ad1f7e1d89e.png) And also we have a lot of down clients ![Screenshot 2022-01-18 at 14.42.36](https://global.discourse-cdn.com/hashicorp/original/3X/b/8/b8c37db1febb71c223068906e37764a33ebc4717.png) The same goes for left servers ![Screenshot 2022-01-18 at 14.44.12](https://global.discourse-cdn.com/hashicorp/original/3X/4/b/4b999f12fc9fbf7b2b23feac091426014f06e7e4.png) Can we do something in order to remove them from the UI? Thank you 🙏 EDIT: We enabled `leave_on_interrupt` and `leave_on_terminate` but we still have this problem. EDIT 2: We have the same problem when we run `nomad node status` ```auto organizations/gcp/bendingspoons.com on  master [$] via 💠 default on ☁️ sv@bendingspoons.com ❯ nomad node status ID DC Name Class Drain Eligibility Status d71b3262 us-central1 nomad-nomad-client-4l6l false eligible ready cb4d299a us-central1 nomad-nomad-client-1q54 false eligible ready a06f390c us-central1 nomad-nomad-client-xxlv false eligible ready 196f2053 us-central1 nomad-nomad-client-lg7r false eligible down 5908adc2 us-central1 nomad-nomad-client-dslk false eligible down 93c3cf1e us-central1 nomad-nomad-client-25r8 false eligible down c5ec4fd3 us-central1 nomad-nomad-client-bwb3 false eligible down ae6ae3a6 us-central1 nomad-nomad-client-jtvb false ineligible down a63d21eb us-central1 nomad-nomad-client-45rf false ineligible down bacb8c9e us-central1 nomad-nomad-client-qxb0 false ineligible down f3f29acf us-central1 nomad-nomad-client-3gzp false ineligible down e679e47e us-central1 nomad-nomad-client-sm0k false ineligible down 7c3a13eb us-central1 nomad-nomad-client-vznm false ineligible down a10d0c66 us-central1 nomad-nomad-client-89j7 false ineligible down f45f3eb7 us-central1 nomad-nomad-client-2117 false ineligible down 3444c520 us-central1 nomad-nomad-client-bht1 false ineligible down 6c00709e us-central1 nomad-nomad-client-4mlq false ineligible down bdccc266 us-central1 nomad-nomad-client-wrqc false ineligible down 89193843 us-central1 nomad-nomad-client-2d28 false ineligible down 2acca668 us-central1 nomad-nomad-client-8j8n false ineligible down f5593a7b us-central1 nomad-nomad-client-mrm0 false ineligible down b9e95158 us-central1 nomad-nomad-client-v5dz false ineligible down b42c0d2f us-central1 nomad-nomad-client-7nnx false ineligible down 5ff00f7a us-central1 nomad-nomad-client-8gfs false ineligible down 2cd75c27 us-central1 nomad-nomad-client-g77h false ineligible down bd6d77fb us-central1 nomad-nomad-client-ps8l false ineligible down 88606107 us-central1 nomad-nomad-client-xrb6 false ineligible down 2b7ac743 us-central1 nomad-nomad-client-4dkx false ineligible down 67b723cf us-central1 nomad-nomad-client-mjz0 false eligible down 2c31618a us-central1 nomad-nomad-client-6sz1 false eligible down ``` ---
**Author:** ![jrasell](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/jrasell/32/5500_2.png) [@jrasell](https://discuss.hashicorp.com/u/jrasell)\ **Post date:** [February 2, 2022, 8:51am UTC](https://discuss.hashicorp.com/t/nomad-remove-down-or-left-nodes-from-cluster/34393/2 "2022-02-02T08:51:17Z")
Hi @voxsim. Terminal resources such as nodes are cleaned from the Nomad state via an internal garbage collector which runs according to a periodic schedule. This can be triggered manually via the [nomad system gc command](https://www.nomadproject.io/docs/commands/system/gc) or via the [/v1/system/gc](https://www.nomadproject.io/api-docs/system#force-gc) API endpoint. If you wish Nomad to be more aggressive with its periodic garbage collection, you can set the [node\_gc\_threshold](https://www.nomadproject.io/docs/configuration/server#node_gc_threshold) server configuration option. How long have the servers been in their `left` state for and what triggered this? Thanks, jrasell and the Nomad team ---
**Author:** ![shantanugadgil](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/shantanugadgil/32/923_2.png) [@shantanugadgil](https://discuss.hashicorp.com/u/shantanugadgil)\ **Post date:** [November 15, 2022, 6:41pm UTC](https://discuss.hashicorp.com/t/nomad-remove-down-or-left-nodes-from-cluster/34393/3 "2022-11-15T18:41:05Z")
@jrasell@voxsim I stumbled upon this discussion when searching for “disconnected clients”. I noticed today that I have a few nodes which are in `disconnected` state for more than 3 (three) months! ```bash $ nomad node status -verbose | grep 0265be10-283b-7a2e-9c65-b6a0898501bf 0265be10-283b-7a2e-9c65-b6a0898501bf my-aws-region my-node-name my-node-class 10.72.95.238 1.3.1 false eligible disconnected $ nomad node status 0265be10-283b-7a2e-9c65-b6a0898501bf error fetching node stats: Unexpected response code: 404 (No path to node) ID = 0265be10-283b-7a2e-9c65-b6a0898501bf Name = my-node-name Class = my-node-class DC = my-aws-region Drain = false Eligibility = eligible Status = disconnected CSI Controllers = CSI Drivers = Host Volumes = Host Networks = loopback CSI Volumes = Driver Status = docker,exec,java,raw_exec Node Events Time Subsystem Message 2022-07-22T07:29:39Z Cluster Node heartbeat missed 2022-07-21T09:06:25Z Driver: java Healthy 2022-07-21T09:05:26Z Cluster Node registered Allocated Resources CPU Memory Disk 0/4472 MHz 0 B/15 GiB 0 B/95 GiB Allocation Resource Utilization CPU Memory 0/4472 MHz 0 B/15 GiB error fetching node stats: actual resource usage not present Allocations ID Node ID Task Group Version Desired Status Created Modified 035284df 0265be10 0 run pending 3mo26d ago 3mo26d ago 2f1266d2 0265be10 0 run pending 3mo26d ago 3mo26d ago ac69f8c4 0265be10 0 run pending 3mo26d ago 3mo26d ago cbcbe500 0265be10 0 run pending 3mo26d ago 3mo26d ago 72d36385 0265be10 0 run pending 3mo26d ago 3mo26d ago e36d7448 0265be10 0 run pending 3mo26d ago 3mo26d ago ``` The node doesn’t go away even if I run multiple `nomad system gc` commands. What made it actually work (surprisingly) was when I marked the non-existent node to `drain`. Once the drain was complete; again, this node DOES NOT exist in the infrastructure, and the drain showed the following … ```bash $ nomad_oregon node drain -enable -deadline 1m 0943d50b-c28a-63d9-dbd2-9bd8769d020b 2022-11-15T18:22:51Z: Ctrl-C to stop monitoring: will not cancel the node drain 2022-11-15T18:22:51Z: Node "0943d50b-c28a-63d9-dbd2-9bd8769d020b" drain strategy set 2022-11-15T18:22:51Z: Drain complete for node 0943d50b-c28a-63d9-dbd2-9bd8769d020b 2022-11-15T18:22:51Z: Alloc "453938c2-ae48-e3cf-bdea-92f05768acd8" marked for migration 2022-11-15T18:22:51Z: Alloc "5c53b51d-45b8-5f81-9eae-bf0061c3be5c" marked for migration 2022-11-15T18:22:51Z: Alloc "267472e4-bf2b-4836-386e-8802f268fed2" marked for migration 2022-11-15T18:22:51Z: Alloc "a11f4abd-aa1c-2859-0d53-2335ad676b17" marked for migration 2022-11-15T18:22:51Z: Alloc "a73b282a-91b6-89ea-b5ba-5ae8e11c61b8" marked for migration 2022-11-15T18:22:51Z: Alloc "a4c12833-7f67-d7fa-7e19-6cc59f732c64" marked for migration 2022-11-15T18:22:51Z: Alloc "267472e4-bf2b-4836-386e-8802f268fed2" draining 2022-11-15T18:22:51Z: Alloc "5c53b51d-45b8-5f81-9eae-bf0061c3be5c" draining 2022-11-15T18:22:52Z: Alloc "453938c2-ae48-e3cf-bdea-92f05768acd8" draining 2022-11-15T18:22:52Z: Alloc "a4c12833-7f67-d7fa-7e19-6cc59f732c64" draining 2022-11-15T18:22:52Z: Alloc "a11f4abd-aa1c-2859-0d53-2335ad676b17" draining 2022-11-15T18:22:52Z: Alloc "a73b282a-91b6-89ea-b5ba-5ae8e11c61b8" draining 2022-11-15T18:23:20Z: Alloc "267472e4-bf2b-4836-386e-8802f268fed2" status pending -> lost 2022-11-15T18:23:21Z: Alloc "a4c12833-7f67-d7fa-7e19-6cc59f732c64" status pending -> lost 2022-11-15T18:23:21Z: Alloc "a11f4abd-aa1c-2859-0d53-2335ad676b17" status pending -> lost 2022-11-15T18:23:21Z: Alloc "a73b282a-91b6-89ea-b5ba-5ae8e11c61b8" status pending -> lost 2022-11-15T18:23:21Z: Alloc "5c53b51d-45b8-5f81-9eae-bf0061c3be5c" status pending -> lost 2022-11-15T18:23:21Z: Alloc "453938c2-ae48-e3cf-bdea-92f05768acd8" status pending -> lost 2022-11-15T18:23:21Z: All allocations on node "0943d50b-c28a-63d9-dbd2-9bd8769d020b" have stopped ``` … and then the node went from `disconnected` to `down`. I was then able to make the node go away from the list by doing a `nomad system gc` ---
**Author:** ![muratyarali777](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/muratyarali777/32/24799_2.png) [@muratyarali777](https://discuss.hashicorp.com/u/muratyarali777)\ **Post date:** [June 8, 2023, 8:59am UTC](https://discuss.hashicorp.com/t/nomad-remove-down-or-left-nodes-from-cluster/34393/4 "2023-06-08T08:59:01Z")
“nomad system gc” command is worked for me