Hi again 
We have nomad server that connect to consul server with consul client in same node,
here is the config and command to run it :
advertise {
http = "10.11.X.X"
rpc = "10.11.X.X"
serf = "10.11.X.X"
}
server {
enabled = true
enable_event_broker = true
raft_protocol = 3
num_schedulers = 128
event_buffer_size = 100
raft_multiplier = 2
heartbeat_grace = "1h"
}
leave_on_interrupt = true
leave_on_terminate = true
telemetry {
publish_allocation_metrics = true
publish_node_metrics = true
prometheus_metrics = true
disable_hostname = true
}
limits {
http_max_conns_per_client = 0
rpc_max_conns_per_client = 0
rpc_handshake_timeout = "20s"
}
/usr/bin/nomad agent \
-server \
-bind="0.0.0.0" \
-bootstrap-expect=5 \
-encrypt="{{redacted}}" \
-data-dir=/nomad/data \
-config=/nomad/config \
-dc="dc1" \
-node="10.11.X.X" \
-consul-address="127.0.0.1:8500" \
-consul-auto-advertise \
-consul-checks-use-advertise \
-retry-join="10.11.X.X" \
-retry-join="10.11.X.Y" \
-retry-join="10.11.X.Z" \
-retry-join="10.11.X.A" \
-retry-join="10.11.X.B" \
-consul-server-auto-join
and we have a golang service that communicate to nomad server using nomad api library, let’s say the service can spawn a job with unique name and config each job
we than do loadtest to our server
when our job hit: ~6000 job
our server have a symptop leader change per ~1 minutes, we check this using script
2021-04-20 09:36:39 [INFO] [leader-check] 10.11.X.X.global
2021-04-20 09:37:39 [INFO] [leader-check] 10.11.X.Y.global
2021-04-20 09:38:39 [INFO] [leader-check] 10.11.X.Y.global
2021-04-20 09:39:40 [INFO] [leader-check] 10.11.X.Z.global
2021-04-20 09:40:40 [INFO] [leader-check] 10.11.X.Z.global
2021-04-20 09:41:40 [INFO] [leader-check] 10.11.X.A.global
2021-04-20 09:42:43 [INFO] [leader-check] 10.11.X.A.global
2021-04-20 09:43:43 [INFO] [leader-check] 10.11.X.A.global
2021-04-20 09:44:43 [INFO] [leader-check] 10.11.X.A.global
2021-04-20 09:45:49 [INFO] [leader-check] 10.11.X.Z.global
2021-04-20 09:46:49 [INFO] [leader-check] 10.11.X.X.global
2021-04-20 09:47:49 [INFO] [leader-check] 10.11.X.Z.global
after go deeper the leader lost each ~20 seconds, and raft do leader election,
and the we stuck at our job hit : ~15000 job
the leader lost is too frequent and the job count didn’t increase or just increase 1 job per ~2-5 minutes
we use spec :
CPU: 2.5 Core
Memory: 12GB
Disk: 24 GB ( IOPS : 12000 )
but the average for CPU and memory Usage was :
Server CPU Memory
10.11.X.X 122m 1062Mi
10.11.X.Y 61m 1072Mi
10.11.X.Z 125m 1921Mi
10.11.X.A 1132m 2004Mi
10.11.X.B 124m 2102Mi
so based on resource usage above, the nomad server usage didn’t hit the limit we set
the nomad client node/instance already idle as we already calculate
Any advise how to achieve high scheduling throughput, 100-200 Job per minutes ? our target job: ~100.000 job
Thanks for your attentions and help 
