2022-10-12T12:43:41.655Z [DEBUG] policy_eval.worker: fetching current count: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:43:41.656Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:43:41.656Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:43:41.656Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:42:41.656485378 +0000 UTC m=+29464.038987567 2022-10-12 12:43:41.656485378 +0000 UTC m=+29524.038987567}" 2022-10-12T12:43:41.945Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=16 2022-10-12T12:43:41.945Z [INFO] policy_eval.worker: scaling target: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target from=16 to=100 reason="capped count from 16407 to 100 to stay within limits" meta="map[nomad_autoscaler.count.capped:true nomad_autoscaler.count.original:16407 nomad_autoscaler.reason_history:[scaling up because metric is 16407] nomad_policy_id:787ac75b-962e-1a9e-d64c-06d485acb008]" 2022-10-12T12:43:41.948Z [INFO] policy_eval.worker: scaling action skipped: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target reason="skipping scaling group workers/workers due to active deployment" 2022-10-12T12:43:41.948Z [DEBUG] policy_eval.broker: ack eval: eval_id=03246341-acab-e300-7a5e-4a9c5b671ff1 token=b638b05c-8dfa-7b88-5f04-768eb7f450f5 eval_id=03246341-acab-e300-7a5e-4a9c5b671ff1 token=b638b05c-8dfa-7b88-5f04-768eb7f450f5 2022-10-12T12:43:41.948Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:43:41.948Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:43:41.948Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:44:11.251Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=06793ada-fd2d-6631-170d-969d2eb89c8e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:44:11.251Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=06793ada-fd2d-6631-170d-969d2eb89c8e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:44:11.251Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=06793ada-fd2d-6631-170d-969d2eb89c8e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=c84a3965-5444-aee6-284e-6717a2385012 2022-10-12T12:44:11.251Z [DEBUG] policy_eval.worker: received policy for evaluation: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:44:11.252Z [DEBUG] policy_eval.worker: fetching current count: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:44:11.252Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:44:11.252Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:44:11.252Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:43:11.252102624 +0000 UTC m=+29493.634604803 2022-10-12 12:44:11.252102624 +0000 UTC m=+29553.634604803}" 2022-10-12T12:44:11.532Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:44:11.532Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:44:11.532Z [DEBUG] policy_eval.worker: no checks need to be executed: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:44:11.532Z [DEBUG] policy_eval.broker: ack eval: eval_id=06793ada-fd2d-6631-170d-969d2eb89c8e token=c84a3965-5444-aee6-284e-6717a2385012 eval_id=06793ada-fd2d-6631-170d-969d2eb89c8e token=c84a3965-5444-aee6-284e-6717a2385012 2022-10-12T12:44:11.532Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:44:11.532Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:44:11.532Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:44:11.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=89decdee-b41a-6c61-2c8b-fb6b4f6f80ea policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:44:11.653Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=89decdee-b41a-6c61-2c8b-fb6b4f6f80ea policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:44:11.653Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=89decdee-b41a-6c61-2c8b-fb6b4f6f80ea policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=4eadd97f-5072-041e-af78-fe3be43dbb60 2022-10-12T12:44:11.653Z [DEBUG] policy_eval.worker: received policy for evaluation: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:44:11.653Z [DEBUG] policy_eval.worker: fetching current count: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:44:11.653Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:44:11.654Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:44:11.654Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:43:11.654037147 +0000 UTC m=+29494.036539326 2022-10-12 12:44:11.654037147 +0000 UTC m=+29554.036539326}" 2022-10-12T12:44:11.933Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=16 2022-10-12T12:44:11.933Z [INFO] policy_eval.worker: scaling target: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target from=16 to=100 reason="capped count from 24404 to 100 to stay within limits" meta="map[nomad_autoscaler.count.capped:true nomad_autoscaler.count.original:24404 nomad_autoscaler.reason_history:[scaling up because metric is 24404] nomad_policy_id:787ac75b-962e-1a9e-d64c-06d485acb008]" 2022-10-12T12:44:11.938Z [INFO] policy_eval.worker: scaling action skipped: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target reason="skipping scaling group workers/workers due to active deployment" 2022-10-12T12:44:11.939Z [DEBUG] policy_eval.broker: ack eval: eval_id=89decdee-b41a-6c61-2c8b-fb6b4f6f80ea token=4eadd97f-5072-041e-af78-fe3be43dbb60 eval_id=89decdee-b41a-6c61-2c8b-fb6b4f6f80ea token=4eadd97f-5072-041e-af78-fe3be43dbb60 2022-10-12T12:44:11.939Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:44:11.939Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:44:11.939Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:44:34.780Z [DEBUG] policy_manager.policy_handler: scaling policy has been placed into cooldown: policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 cooldown=13.505117234s 2022-10-12T12:44:37.626Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection 2022-10-12T12:44:41.253Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=c43d77b2-7885-6cd1-0dfe-c4358cf4321e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:44:41.253Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=c43d77b2-7885-6cd1-0dfe-c4358cf4321e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:44:41.253Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=c43d77b2-7885-6cd1-0dfe-c4358cf4321e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=5d3b34ef-823e-e2c2-bfe5-88fe18e11a8e 2022-10-12T12:44:41.253Z [DEBUG] policy_eval.worker: received policy for evaluation: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:44:41.253Z [DEBUG] policy_eval.worker: fetching current count: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:44:41.253Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:44:41.253Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:44:41.253Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:43:41.253788356 +0000 UTC m=+29523.636290535 2022-10-12 12:44:41.253788356 +0000 UTC m=+29583.636290535}" 2022-10-12T12:44:41.551Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:44:41.551Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:44:41.551Z [DEBUG] policy_eval.worker: no checks need to be executed: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:44:41.551Z [DEBUG] policy_eval.broker: ack eval: eval_id=c43d77b2-7885-6cd1-0dfe-c4358cf4321e token=5d3b34ef-823e-e2c2-bfe5-88fe18e11a8e eval_id=c43d77b2-7885-6cd1-0dfe-c4358cf4321e token=5d3b34ef-823e-e2c2-bfe5-88fe18e11a8e 2022-10-12T12:44:41.551Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:44:41.551Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:44:41.551Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:44:41.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=17b7da62-a399-6c98-d7f8-60adee765e30 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:44:41.654Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=17b7da62-a399-6c98-d7f8-60adee765e30 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:44:41.654Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=17b7da62-a399-6c98-d7f8-60adee765e30 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=eed21d99-f01f-4321-7559-9d0e4f34860e 2022-10-12T12:44:41.654Z [DEBUG] policy_eval.worker: received policy for evaluation: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:44:41.654Z [DEBUG] policy_eval.worker: fetching current count: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:44:41.654Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:44:41.654Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:44:41.654Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:43:41.654130359 +0000 UTC m=+29524.036632538 2022-10-12 12:44:41.654130359 +0000 UTC m=+29584.036632538}" 2022-10-12T12:44:41.941Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=16 2022-10-12T12:44:41.942Z [INFO] policy_eval.worker: scaling target: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target from=16 to=100 reason="capped count from 24404 to 100 to stay within limits" meta="map[nomad_autoscaler.count.capped:true nomad_autoscaler.count.original:24404 nomad_autoscaler.reason_history:[scaling up because metric is 24404] nomad_policy_id:787ac75b-962e-1a9e-d64c-06d485acb008]" 2022-10-12T12:44:41.959Z [INFO] policy_eval.worker: scaling action skipped: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target reason="skipping scaling group workers/workers due to active deployment" 2022-10-12T12:44:41.959Z [DEBUG] policy_eval.broker: ack eval: eval_id=17b7da62-a399-6c98-d7f8-60adee765e30 token=eed21d99-f01f-4321-7559-9d0e4f34860e eval_id=17b7da62-a399-6c98-d7f8-60adee765e30 token=eed21d99-f01f-4321-7559-9d0e4f34860e 2022-10-12T12:44:41.959Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:44:41.959Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:44:41.959Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:44:48.313Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 is initializing, node 725bd410-572a-1260-9a7a-1bb5a369afde is initializing, node 171aa5f2-7638-356e-9231-beaeb9d2ffcc is initializing, node dac51581-ec47-d653-e577-25ad4b30aa1d is initializing, node b779c284-396f-deff-3de9-02df51ccf024 is initializing, node 5b79ee9c-b72b-6391-6602-3e479aa505c7 is initializing, node e79e037a-3200-89b0-6a87-b65688b52403 is initializing, node 11c4df5a-606d-4222-7def-f512740f890a is initializing" 2022-10-12T12:45:11.251Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=8035b26a-8eb0-755e-28b3-72b9c6ee6718 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:45:11.252Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=8035b26a-8eb0-755e-28b3-72b9c6ee6718 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:45:11.252Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=8035b26a-8eb0-755e-28b3-72b9c6ee6718 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=cc711233-5ead-e7d0-755e-2ccd5b509f80 2022-10-12T12:45:11.252Z [DEBUG] policy_eval.worker: received policy for evaluation: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:45:11.252Z [DEBUG] policy_eval.worker: fetching current count: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:45:11.253Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:45:11.253Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:45:11.253Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:44:11.253445569 +0000 UTC m=+29553.635947748 2022-10-12 12:45:11.253445569 +0000 UTC m=+29613.635947748}" 2022-10-12T12:45:11.542Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:45:11.542Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:45:11.542Z [DEBUG] policy_eval.worker: no checks need to be executed: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:45:11.542Z [DEBUG] policy_eval.broker: ack eval: eval_id=8035b26a-8eb0-755e-28b3-72b9c6ee6718 token=cc711233-5ead-e7d0-755e-2ccd5b509f80 eval_id=8035b26a-8eb0-755e-28b3-72b9c6ee6718 token=cc711233-5ead-e7d0-755e-2ccd5b509f80 2022-10-12T12:45:11.542Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:45:11.542Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:45:11.542Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:45:11.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=1c7d3c08-9fcd-c174-e243-5c1da034424d policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:45:11.653Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=1c7d3c08-9fcd-c174-e243-5c1da034424d policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:45:11.653Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=1c7d3c08-9fcd-c174-e243-5c1da034424d policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=d8568c56-533d-f4ef-5d8a-6ab5e502559a 2022-10-12T12:45:11.653Z [DEBUG] policy_eval.worker: received policy for evaluation: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:45:11.653Z [DEBUG] policy_eval.worker: fetching current count: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:45:11.654Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:45:11.654Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:45:11.654Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:44:11.654569101 +0000 UTC m=+29554.037071280 2022-10-12 12:45:11.654569101 +0000 UTC m=+29614.037071280}" 2022-10-12T12:45:11.934Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=86 2022-10-12T12:45:11.934Z [INFO] policy_eval.worker: scaling target: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target from=86 to=100 reason="capped count from 32039 to 100 to stay within limits" meta="map[nomad_autoscaler.count.capped:true nomad_autoscaler.count.original:32039 nomad_autoscaler.reason_history:[scaling up because metric is 32039] nomad_policy_id:787ac75b-962e-1a9e-d64c-06d485acb008]" 2022-10-12T12:45:11.939Z [INFO] policy_eval.worker: scaling action skipped: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target reason="skipping scaling group workers/workers due to active deployment" 2022-10-12T12:45:11.940Z [DEBUG] policy_eval.broker: ack eval: eval_id=1c7d3c08-9fcd-c174-e243-5c1da034424d token=d8568c56-533d-f4ef-5d8a-6ab5e502559a eval_id=1c7d3c08-9fcd-c174-e243-5c1da034424d token=d8568c56-533d-f4ef-5d8a-6ab5e502559a 2022-10-12T12:45:11.940Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:45:11.940Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:45:11.940Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:45:13.385Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster token="" 2022-10-12T12:45:13.385Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster token="" 2022-10-12T12:45:13.385Z [DEBUG] policy_eval.broker: eval dequeued: queue=cluster eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 token=5261ed5e-4f6d-2f87-ed53-d03d66b2b87c 2022-10-12T12:45:13.385Z [DEBUG] policy_eval.worker: received policy for evaluation: id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg 2022-10-12T12:45:13.385Z [DEBUG] policy_eval.worker: fetching current count: id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg 2022-10-12T12:45:13.678Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_allocated_percentage id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster source=prometheus strategy=pass-through target=aws-asg 2022-10-12T12:45:13.679Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_allocated_percentage id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster source=prometheus strategy=pass-through target=aws-asg query="ceil((sum(nomad_client_allocs_cpu_allocated{namespace="dev"}) + sum(nomad_nomad_blocked_evals_job_cpu{namespace="dev"} OR on() vector(0)))/6000)" source=prometheus 2022-10-12T12:45:13.679Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(nomad_client_allocs_cpu_allocated{namespace="dev"}) + sum(nomad_nomad_blocked_evals_job_cpu{namespace="dev"} OR on() vector(0)))/6000)" range="{2022-10-12 12:44:13.679098869 +0000 UTC m=+29556.061601057 2022-10-12 12:45:13.679098869 +0000 UTC m=+29616.061601057}" 2022-10-12T12:45:13.962Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_allocated_percentage id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster source=prometheus strategy=pass-through target=aws-asg count=50 2022-10-12T12:45:13.966Z [INFO] policy_eval.worker: scaling target: id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg from=50 to=13 reason="scaling down because metric is 13" meta=map[nomad_policy_id:c4d837ab-f45c-303a-ea9a-8309a1b02590] 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0048c3ccfe361ac85 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-008c5445a668a0009 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-00d3ea14bdaa33368 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-00e27b94b28d68e77 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-01a91664bdc5ff74b 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-021f5e026d92ca302 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-025f4777ced8206f2 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0277c7c4aa6b851e8 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-02d1c87d1c9b9cd7f 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-02f4f4ac605601407 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0332a5f1841464dc8 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-03599434bb1d2a8e6 2022-10-12T12:45:14.056Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0393bde26d6537bde 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0450cdcd3275fe3e9 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-050f40991e0f63e4b 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-062ceb795d424f426 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-064b0bdc7ee71e110 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0652692e78c262a0e 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-07668bfc4e809a341 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-07cb628df21338953 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0891e445a96077226 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-08f8d0daa89a0195c 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-091536ac62050fa60 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-09482f7fa83ce6892 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0952776df290dfa4c 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-09656f256b66ef696 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-09b5a53430d79b0e8 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0a2327e4f7a83812e 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0a45c001395e83782 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0a6362b62575d7873 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0a7b6c78c4d1c922e 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0aabd322999a88e95 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0b64095156fdbf479 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0b8014eb4d689dddb 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0c0b2ebfde7ccb336 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0c20873c099129aa2 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0c23276747d0b6ba5 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0c2b7ea53cd0c18c0 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0c7360fd270204faa 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0c9b5583a1aef4a0f 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0c9cd156ec9634e2d 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0ce390876fdeeb6cc 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0d3c9fe9d14851924 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0d719e71ac3bb07c4 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0e1e5560caa1b8719 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0e2038e99168d2fd0 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0e9582e3bc8e6686b 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0f4d1d9f2e29c1260 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0f8fa6c54bd93b48e 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: found healthy instance: action=scale_in asg_name=test-cloud-nomad-client instance_id=i-0fe4c87879964c43f 2022-10-12T12:45:14.057Z [DEBUG] internal_plugin.aws-asg: performing node pool filtering: node_class=hashistack-client 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=5ac68c24-bb30-44f2-5829-581b1623895a datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=47e6591c-1d9b-0231-df05-089262080217 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b8c7d206-607b-7d51-945a-75179b50485c datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4592c999-9240-e62d-edd3-5c3faa37129e datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=29b86c48-da66-2229-be80-361e607a99ce datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b7b16595-e281-4f34-7189-c653c447f9ea datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=71d147c8-2031-02b5-feea-daee12267e19 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=725bd410-572a-1260-9a7a-1bb5a369afde datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b779c284-396f-deff-3de9-02df51ccf024 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=e79e037a-3200-89b0-6a87-b65688b52403 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=7c995302-018d-fc7f-9b66-8cb242122581 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.139Z [DEBUG] internal_plugin.aws-asg: found node: node_id=11c4df5a-606d-4222-7def-f512740f890a datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=5fa178f9-6812-3163-5075-92364269a17b datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=d36869a3-7293-0111-b111-f4c5fbe933f3 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=37dc2bc7-6af4-d7e4-940a-48e9771de4f0 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=0c595d30-34f5-adbe-8974-8a366de58715 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=745dd1b2-fb27-3f5b-80f2-b14a515b8a1c datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=37c57d6e-f64f-9aad-ffc1-e40185aca485 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=a73ca864-b9e7-7faf-fc75-44862199a7ad datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=f06cee3e-8708-6447-4c68-1dd2356405b9 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=c6927a14-71e6-9b8a-c134-7100bd21f28f datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b374de37-15a5-8709-e20f-7c30467c702b datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=c7cc829d-4771-12c5-eb30-64b15677f9f5 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=2c719dde-b52f-8285-2ec9-dc4cf90146e8 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=c45602aa-fb1e-6f1e-887f-515196fad962 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=93bbcf76-8e7e-8d37-e3dc-2261a475ed59 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=d014b2fa-128a-9487-04a2-0cc5c294f5ce datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=db0ce600-00a8-1e59-9c9c-f00890a929b0 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b3debf58-fc86-e899-203d-70a19ec050e2 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4ba70c07-f999-bf4c-775b-bc50e22c9ffb datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4dd5ae28-6f37-be09-6c7d-c75e6dec0825 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=a3bdd682-e8be-42d6-014b-5482acd8974f datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=0ff453e0-a827-7ba0-d79e-8201b53bc19a datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=09b536c5-5114-8f9e-e093-94d5f0362cfb datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=d6932990-cb83-a52a-17dc-f67a5240308f datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=2bdfd394-92b2-b5f9-e81d-20f7b95eb4c1 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=f13b949c-ea41-eaf8-05f5-ecf45d7a39db datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4a00a586-a6a6-ef0f-46cb-7135b8028e50 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=9ec34a5f-5e9d-a8a1-23ba-37b16879732e datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=d375809f-9d05-844f-28a0-a5063abc5aad datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=851761a3-9164-2045-f6d3-815819ee3405 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=5360553c-f666-7852-5598-3c19fc577a9a datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=066c3e49-5ec1-b101-4d02-2368f1d6a99f datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=c78e6814-ad09-0fca-733b-cef6db0f1be2 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=30a60e16-7483-beee-7cff-5d73ffb54d64 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=18a13a31-67c4-6470-0f2d-10d935400e19 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b7a0debb-087d-6993-a081-3736ce7cc493 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=61fe4666-0994-e597-c794-3365d3d4de5c datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=7af70277-f359-7bb2-ac80-2e14d4192df8 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=04593d9f-5e3d-72cc-5012-07431ffb165e datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=199acbd9-cc1b-ed12-2c59-fc56360befbb datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4ae93005-03d8-77cb-1e4e-0c06831c9d20 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=e3ce1c32-be16-1203-ec0e-94087b6e6563 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=3d953de8-3b37-5639-084a-845d75f46241 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=a8bd3cf6-e045-72b1-256e-ab4ac915d4f2 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b311cf75-b5a7-4367-26d0-538178dbf792 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=ed82a930-8a39-f9cd-2abb-51b84f255010 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.140Z [DEBUG] internal_plugin.aws-asg: found node: node_id=811fbb4b-7f39-9ea3-b6a1-803dab671ecb datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=16855350-921c-6d31-6fa4-3a9813ea610b datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=55b6a5d4-3cef-e557-c854-424c1d4510c1 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=500cc242-28bf-af72-bdf7-e5abea0da4bb datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4b76fbf1-7a15-2f60-01ab-739636670155 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=6772afc2-4453-205f-f4d1-fac7a35fbc00 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=9cc458c9-05ef-a1b8-1622-ebb1ae76eac3 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=9730cb55-3bdc-8f07-3c1b-5eca26d2d5bf datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=e71b5db8-76d5-cf9c-5ba6-aa8b4aa88813 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=0b335350-d3a7-f4ea-8c61-f2c696c77951 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=dd9f3690-f267-542d-7029-b92f13f22dba datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=61244ea2-0b3e-61ed-5461-f34bc2778585 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=27f63427-3be3-304b-039e-2103f32c31e9 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=7132481e-0bb5-e632-0b8e-32987dc65894 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 datacenter=dc1 node_class=hashistack-client status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=859d6bfb-596f-a70b-c67d-de154ba98e7e datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=00711bd6-3be1-2ec4-8211-88610932b476 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=8fc443dc-e163-db36-acc4-2518eb125b51 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=0852c1a5-d642-c2a2-8404-9170b5d25f2a datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=0ccc5e20-3c71-0802-8ace-7947282a05ef datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=456675ec-e3d6-30f7-c83e-dceb10dd4108 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.141Z [DEBUG] internal_plugin.aws-asg: found node: node_id=2754a985-a4d6-5acf-7148-7d3d25d2153a datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=0db9deec-dddb-8ed7-fd67-e680a4e6df61 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=0eb5d71d-35b5-462d-4e14-6495eb8facee datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=43e3b76c-bf1a-31f0-d2a3-6795ea550664 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=693007d3-3017-06df-e160-f91852f0472a datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=eb0c4821-835e-fcbe-3674-77d3b9a4974e datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=eb0f67b9-5ef6-56cb-3c39-9015813f3d78 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=dc018632-7a8e-137b-ac2f-3dacd87b87ed datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=961c9db5-ff80-766f-e1d7-864ebeff1a40 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.142Z [DEBUG] internal_plugin.aws-asg: found node: node_id=31f41af5-93ef-7c8f-0a58-77f680819f27 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=928a963b-291c-fb57-f61c-03f144c43465 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=42d2fa0b-0d48-b910-6476-837337de226b datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=293f53f1-ad51-b18c-7a98-715bb7bfb8d3 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=eb9fda1f-b9c1-5a65-855f-7d0282ce6946 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=154a5bf1-1526-96e0-41b1-7aebbb34d1ca datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=b05d8f45-f2d8-de0d-6d29-e1e7bcc1cba7 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=018672a3-e6da-6ae3-4a8d-259f72105d2f datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=4bc29eaa-91ac-3978-96d5-b7ee8924c12d datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=3738350c-8ec9-7bea-fb00-0f19e7b1af92 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=653a00b9-12ad-09db-f854-91ee69d928bb datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=e7256aa8-57f8-7ca4-e6d8-c06b549c5568 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=990b071f-d1b2-7030-d6b2-69e3f093a2ae datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=1c202fd2-cd50-22f3-c19d-d0b977cef065 datacenter=dc1 node_class=hashistack-client status=down eligibility=ineligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=018f2cdc-c531-0660-7c67-514f4f2626d7 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=570f8212-3cdb-153d-7d45-2c3871729695 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=9d953be2-8baa-77a1-764c-7019f9b81d7b datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=437d2161-d241-1f1d-291f-70390faf2426 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=e03ced15-0ea0-c00c-4286-a96d9def9072 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=9bb15289-6982-84ff-cb3a-8f08b6492433 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=cd440dd0-4903-2e40-86b9-fd8f716c6249 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=2acaa089-bba3-aea8-90da-68d2ff972873 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=ffda06c2-436a-924e-9a51-72cd6653bf93 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=ed7c3425-2e20-c0a7-3746-0d0f44cff154 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=8eaef983-0d5a-48b8-2631-5851a57c974f datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=70d698e4-9837-49ee-c50d-b91678e9ee8e datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=a5effb96-7504-6893-3cbf-0a9a4b3692e8 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=e2a4b724-20c8-174a-ab21-01d53e5e30c3 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=486d2577-8552-80cf-0586-ccf79dcce9dc datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=bf251908-76c5-b371-27ff-c72f975ef424 datacenter=dc1 node_class=hashistack-client status=down eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: found node: node_id=af4ef312-1b6e-6488-0d16-1a4644b461e3 datacenter=dc1 node_class=webapp status=ready eligibility=eligible draining=false 2022-10-12T12:45:14.143Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 2022-10-12T12:45:14.144Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=5ac68c24-bb30-44f2-5829-581b1623895a 2022-10-12T12:45:14.144Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 2022-10-12T12:45:14.144Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=47e6591c-1d9b-0231-df05-089262080217 2022-10-12T12:45:14.144Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da 2022-10-12T12:45:14.144Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=b8c7d206-607b-7d51-945a-75179b50485c 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=4592c999-9240-e62d-edd3-5c3faa37129e 2022-10-12T12:45:14.145Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=29b86c48-da66-2229-be80-361e607a99ce 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=b7b16595-e281-4f34-7189-c653c447f9ea 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=71d147c8-2031-02b5-feea-daee12267e19 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=725bd410-572a-1260-9a7a-1bb5a369afde 2022-10-12T12:45:14.146Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=b779c284-396f-deff-3de9-02df51ccf024 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=e79e037a-3200-89b0-6a87-b65688b52403 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=7c995302-018d-fc7f-9b66-8cb242122581 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=11c4df5a-606d-4222-7def-f512740f890a 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 2022-10-12T12:45:14.147Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=5fa178f9-6812-3163-5075-92364269a17b 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=d36869a3-7293-0111-b111-f4c5fbe933f3 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=37dc2bc7-6af4-d7e4-940a-48e9771de4f0 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=0c595d30-34f5-adbe-8974-8a366de58715 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=745dd1b2-fb27-3f5b-80f2-b14a515b8a1c 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=37c57d6e-f64f-9aad-ffc1-e40185aca485 2022-10-12T12:45:14.148Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=a73ca864-b9e7-7faf-fc75-44862199a7ad 2022-10-12T12:45:14.152Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=b374de37-15a5-8709-e20f-7c30467c702b 2022-10-12T12:45:14.152Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=c7cc829d-4771-12c5-eb30-64b15677f9f5 2022-10-12T12:45:14.152Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=d014b2fa-128a-9487-04a2-0cc5c294f5ce 2022-10-12T12:45:14.153Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 2022-10-12T12:45:14.153Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=ed82a930-8a39-f9cd-2abb-51b84f255010 2022-10-12T12:45:14.153Z [DEBUG] internal_plugin.aws-asg: node passed filter criteria: node_id=16855350-921c-6d31-6fa4-3a9813ea610b 2022-10-12T12:45:14.157Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 remote_id=i-09482f7fa83ce6892 2022-10-12T12:45:14.160Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=5ac68c24-bb30-44f2-5829-581b1623895a remote_id=i-0c23276747d0b6ba5 2022-10-12T12:45:14.164Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 remote_id=i-0e9582e3bc8e6686b 2022-10-12T12:45:14.170Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=47e6591c-1d9b-0231-df05-089262080217 remote_id=i-0b8014eb4d689dddb 2022-10-12T12:45:14.174Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da remote_id=i-0b64095156fdbf479 2022-10-12T12:45:14.178Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 remote_id=i-008c5445a668a0009 2022-10-12T12:45:14.189Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef remote_id=i-0652692e78c262a0e 2022-10-12T12:45:14.206Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a remote_id=i-0e1e5560caa1b8719 2022-10-12T12:45:14.213Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 remote_id=i-07668bfc4e809a341 2022-10-12T12:45:14.230Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 remote_id=i-0393bde26d6537bde 2022-10-12T12:45:14.236Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 remote_id=i-0952776df290dfa4c 2022-10-12T12:45:14.240Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a remote_id=i-0c7360fd270204faa 2022-10-12T12:45:14.243Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=b8c7d206-607b-7d51-945a-75179b50485c remote_id=i-0aabd322999a88e95 2022-10-12T12:45:14.247Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=4592c999-9240-e62d-edd3-5c3faa37129e remote_id=i-0d3c9fe9d14851924 2022-10-12T12:45:14.253Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=29b86c48-da66-2229-be80-361e607a99ce remote_id=i-062ceb795d424f426 2022-10-12T12:45:14.258Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 remote_id=i-0d719e71ac3bb07c4 2022-10-12T12:45:14.264Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 remote_id=i-09656f256b66ef696 2022-10-12T12:45:14.277Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f remote_id=i-00e27b94b28d68e77 2022-10-12T12:45:14.283Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=b7b16595-e281-4f34-7189-c653c447f9ea remote_id=i-0c9cd156ec9634e2d 2022-10-12T12:45:14.287Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 remote_id=i-0c2b7ea53cd0c18c0 2022-10-12T12:45:14.291Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=71d147c8-2031-02b5-feea-daee12267e19 remote_id=i-0048c3ccfe361ac85 2022-10-12T12:45:14.306Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 remote_id=i-0a6362b62575d7873 2022-10-12T12:45:14.313Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e remote_id=i-050f40991e0f63e4b 2022-10-12T12:45:14.316Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=725bd410-572a-1260-9a7a-1bb5a369afde remote_id=i-0c20873c099129aa2 2022-10-12T12:45:14.319Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc remote_id=i-0277c7c4aa6b851e8 2022-10-12T12:45:14.323Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 remote_id=i-0a45c001395e83782 2022-10-12T12:45:14.326Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d remote_id=i-08f8d0daa89a0195c 2022-10-12T12:45:14.334Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=b779c284-396f-deff-3de9-02df51ccf024 remote_id=i-03599434bb1d2a8e6 2022-10-12T12:45:14.354Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 remote_id=i-0a7b6c78c4d1c922e 2022-10-12T12:45:14.360Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=e79e037a-3200-89b0-6a87-b65688b52403 remote_id=i-0c9b5583a1aef4a0f 2022-10-12T12:45:14.374Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=7c995302-018d-fc7f-9b66-8cb242122581 remote_id=i-02d1c87d1c9b9cd7f 2022-10-12T12:45:14.386Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=11c4df5a-606d-4222-7def-f512740f890a remote_id=i-0c0b2ebfde7ccb336 2022-10-12T12:45:14.390Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a remote_id=i-01a91664bdc5ff74b 2022-10-12T12:45:14.394Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 remote_id=i-064b0bdc7ee71e110 2022-10-12T12:45:14.397Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 remote_id=i-021f5e026d92ca302 2022-10-12T12:45:14.400Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd remote_id=i-0a2327e4f7a83812e 2022-10-12T12:45:14.404Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=5fa178f9-6812-3163-5075-92364269a17b remote_id=i-0e2038e99168d2fd0 2022-10-12T12:45:14.406Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=d36869a3-7293-0111-b111-f4c5fbe933f3 remote_id=i-0450cdcd3275fe3e9 2022-10-12T12:45:14.411Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=37dc2bc7-6af4-d7e4-940a-48e9771de4f0 remote_id=i-02f4f4ac605601407 2022-10-12T12:45:14.415Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=0c595d30-34f5-adbe-8974-8a366de58715 remote_id=i-0891e445a96077226 2022-10-12T12:45:14.431Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=745dd1b2-fb27-3f5b-80f2-b14a515b8a1c remote_id=i-0f8fa6c54bd93b48e 2022-10-12T12:45:14.437Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=37c57d6e-f64f-9aad-ffc1-e40185aca485 remote_id=i-025f4777ced8206f2 2022-10-12T12:45:14.456Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=a73ca864-b9e7-7faf-fc75-44862199a7ad remote_id=i-0332a5f1841464dc8 2022-10-12T12:45:14.478Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=b374de37-15a5-8709-e20f-7c30467c702b remote_id=i-00d3ea14bdaa33368 2022-10-12T12:45:14.496Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=c7cc829d-4771-12c5-eb30-64b15677f9f5 remote_id=i-0fe4c87879964c43f 2022-10-12T12:45:14.503Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=d014b2fa-128a-9487-04a2-0cc5c294f5ce remote_id=i-091536ac62050fa60 2022-10-12T12:45:14.513Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 remote_id=i-09b5a53430d79b0e8 2022-10-12T12:45:14.526Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=ed82a930-8a39-f9cd-2abb-51b84f255010 remote_id=i-07cb628df21338953 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: identified remote provider ID for node: node_id=16855350-921c-6d31-6fa4-3a9813ea610b remote_id=i-0ce390876fdeeb6cc 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 remote_id=i-09482f7fa83ce6892 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=5ac68c24-bb30-44f2-5829-581b1623895a remote_id=i-0c23276747d0b6ba5 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 remote_id=i-0e9582e3bc8e6686b 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=47e6591c-1d9b-0231-df05-089262080217 remote_id=i-0b8014eb4d689dddb 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da remote_id=i-0b64095156fdbf479 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 remote_id=i-008c5445a668a0009 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef remote_id=i-0652692e78c262a0e 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a remote_id=i-0e1e5560caa1b8719 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 remote_id=i-07668bfc4e809a341 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 remote_id=i-0393bde26d6537bde 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 remote_id=i-0952776df290dfa4c 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a remote_id=i-0c7360fd270204faa 2022-10-12T12:45:14.542Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=b8c7d206-607b-7d51-945a-75179b50485c remote_id=i-0aabd322999a88e95 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=4592c999-9240-e62d-edd3-5c3faa37129e remote_id=i-0d3c9fe9d14851924 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=29b86c48-da66-2229-be80-361e607a99ce remote_id=i-062ceb795d424f426 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 remote_id=i-0d719e71ac3bb07c4 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 remote_id=i-09656f256b66ef696 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f remote_id=i-00e27b94b28d68e77 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=b7b16595-e281-4f34-7189-c653c447f9ea remote_id=i-0c9cd156ec9634e2d 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 remote_id=i-0c2b7ea53cd0c18c0 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=71d147c8-2031-02b5-feea-daee12267e19 remote_id=i-0048c3ccfe361ac85 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 remote_id=i-0a6362b62575d7873 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e remote_id=i-050f40991e0f63e4b 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=725bd410-572a-1260-9a7a-1bb5a369afde remote_id=i-0c20873c099129aa2 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc remote_id=i-0277c7c4aa6b851e8 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 remote_id=i-0a45c001395e83782 2022-10-12T12:45:14.549Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d remote_id=i-08f8d0daa89a0195c 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=b779c284-396f-deff-3de9-02df51ccf024 remote_id=i-03599434bb1d2a8e6 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 remote_id=i-0a7b6c78c4d1c922e 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=e79e037a-3200-89b0-6a87-b65688b52403 remote_id=i-0c9b5583a1aef4a0f 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=7c995302-018d-fc7f-9b66-8cb242122581 remote_id=i-02d1c87d1c9b9cd7f 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=11c4df5a-606d-4222-7def-f512740f890a remote_id=i-0c0b2ebfde7ccb336 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a remote_id=i-01a91664bdc5ff74b 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 remote_id=i-064b0bdc7ee71e110 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 remote_id=i-021f5e026d92ca302 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd remote_id=i-0a2327e4f7a83812e 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=5fa178f9-6812-3163-5075-92364269a17b remote_id=i-0e2038e99168d2fd0 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=d36869a3-7293-0111-b111-f4c5fbe933f3 remote_id=i-0450cdcd3275fe3e9 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=37dc2bc7-6af4-d7e4-940a-48e9771de4f0 remote_id=i-02f4f4ac605601407 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=0c595d30-34f5-adbe-8974-8a366de58715 remote_id=i-0891e445a96077226 2022-10-12T12:45:14.550Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=745dd1b2-fb27-3f5b-80f2-b14a515b8a1c remote_id=i-0f8fa6c54bd93b48e 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=37c57d6e-f64f-9aad-ffc1-e40185aca485 remote_id=i-025f4777ced8206f2 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=a73ca864-b9e7-7faf-fc75-44862199a7ad remote_id=i-0332a5f1841464dc8 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=b374de37-15a5-8709-e20f-7c30467c702b remote_id=i-00d3ea14bdaa33368 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=c7cc829d-4771-12c5-eb30-64b15677f9f5 remote_id=i-0fe4c87879964c43f 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=d014b2fa-128a-9487-04a2-0cc5c294f5ce remote_id=i-091536ac62050fa60 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 remote_id=i-09b5a53430d79b0e8 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=ed82a930-8a39-f9cd-2abb-51b84f255010 remote_id=i-07cb628df21338953 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: node is part of the policy target: node_id=16855350-921c-6d31-6fa4-3a9813ea610b remote_id=i-0ce390876fdeeb6cc 2022-10-12T12:45:14.551Z [DEBUG] internal_plugin.aws-asg: performing node selection: selector_strategy=least_busy 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 remote_id=i-09482f7fa83ce6892 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=5ac68c24-bb30-44f2-5829-581b1623895a remote_id=i-0c23276747d0b6ba5 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 remote_id=i-0e9582e3bc8e6686b 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d remote_id=i-08f8d0daa89a0195c 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a remote_id=i-0c7360fd270204faa 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef remote_id=i-0652692e78c262a0e 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 remote_id=i-008c5445a668a0009 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a remote_id=i-0e1e5560caa1b8719 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 remote_id=i-07668bfc4e809a341 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 remote_id=i-0393bde26d6537bde 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 remote_id=i-0952776df290dfa4c 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da remote_id=i-0b64095156fdbf479 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=b8c7d206-607b-7d51-945a-75179b50485c remote_id=i-0aabd322999a88e95 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=4592c999-9240-e62d-edd3-5c3faa37129e remote_id=i-0d3c9fe9d14851924 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=29b86c48-da66-2229-be80-361e607a99ce remote_id=i-062ceb795d424f426 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 remote_id=i-0d719e71ac3bb07c4 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 remote_id=i-09656f256b66ef696 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f remote_id=i-00e27b94b28d68e77 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=b7b16595-e281-4f34-7189-c653c447f9ea remote_id=i-0c9cd156ec9634e2d 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 remote_id=i-0c2b7ea53cd0c18c0 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=71d147c8-2031-02b5-feea-daee12267e19 remote_id=i-0048c3ccfe361ac85 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 remote_id=i-0a6362b62575d7873 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e remote_id=i-050f40991e0f63e4b 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=725bd410-572a-1260-9a7a-1bb5a369afde remote_id=i-0c20873c099129aa2 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc remote_id=i-0277c7c4aa6b851e8 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 remote_id=i-0a45c001395e83782 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=47e6591c-1d9b-0231-df05-089262080217 remote_id=i-0b8014eb4d689dddb 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=b779c284-396f-deff-3de9-02df51ccf024 remote_id=i-03599434bb1d2a8e6 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 remote_id=i-0a7b6c78c4d1c922e 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=e79e037a-3200-89b0-6a87-b65688b52403 remote_id=i-0c9b5583a1aef4a0f 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=7c995302-018d-fc7f-9b66-8cb242122581 remote_id=i-02d1c87d1c9b9cd7f 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=11c4df5a-606d-4222-7def-f512740f890a remote_id=i-0c0b2ebfde7ccb336 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a remote_id=i-01a91664bdc5ff74b 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 remote_id=i-064b0bdc7ee71e110 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 remote_id=i-021f5e026d92ca302 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd remote_id=i-0a2327e4f7a83812e 2022-10-12T12:45:14.820Z [DEBUG] internal_plugin.aws-asg: node selected for removal: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 remote_id=i-09b5a53430d79b0e8 2022-10-12T12:45:14.820Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 deadline=5m0s 2022-10-12T12:45:14.820Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f deadline=5m0s 2022-10-12T12:45:14.821Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 deadline=5m0s 2022-10-12T12:45:14.821Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=5ac68c24-bb30-44f2-5829-581b1623895a deadline=5m0s 2022-10-12T12:45:14.821Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 deadline=5m0s 2022-10-12T12:45:14.821Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d deadline=5m0s 2022-10-12T12:45:14.822Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a deadline=5m0s 2022-10-12T12:45:14.822Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef deadline=5m0s 2022-10-12T12:45:14.822Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=b7b16595-e281-4f34-7189-c653c447f9ea deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 deadline=5m0s 2022-10-12T12:45:14.824Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=71d147c8-2031-02b5-feea-daee12267e19 deadline=5m0s 2022-10-12T12:45:14.826Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 deadline=5m0s 2022-10-12T12:45:14.826Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e deadline=5m0s 2022-10-12T12:45:14.828Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=725bd410-572a-1260-9a7a-1bb5a369afde deadline=5m0s 2022-10-12T12:45:14.838Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc deadline=5m0s 2022-10-12T12:45:14.838Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 deadline=5m0s 2022-10-12T12:45:14.838Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=47e6591c-1d9b-0231-df05-089262080217 deadline=5m0s 2022-10-12T12:45:14.838Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=b779c284-396f-deff-3de9-02df51ccf024 deadline=5m0s 2022-10-12T12:45:14.838Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 deadline=5m0s 2022-10-12T12:45:14.838Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=e79e037a-3200-89b0-6a87-b65688b52403 deadline=5m0s 2022-10-12T12:45:14.845Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=7c995302-018d-fc7f-9b66-8cb242122581 deadline=5m0s 2022-10-12T12:45:14.845Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=11c4df5a-606d-4222-7def-f512740f890a deadline=5m0s 2022-10-12T12:45:14.846Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a deadline=5m0s 2022-10-12T12:45:14.846Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 deadline=5m0s 2022-10-12T12:45:14.846Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 deadline=5m0s 2022-10-12T12:45:14.846Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=b8c7d206-607b-7d51-945a-75179b50485c deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=4592c999-9240-e62d-edd3-5c3faa37129e deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=29b86c48-da66-2229-be80-361e607a99ce deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 deadline=5m0s 2022-10-12T12:45:14.823Z [INFO] internal_plugin.aws-asg: triggering drain on node: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 deadline=5m0s 2022-10-12T12:45:16.126Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a2cab557-2a69-7ad6-554c-a7789d197484" status pending -> running" 2022-10-12T12:45:16.399Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a92f8ca4-95b2-dd53-e97d-da908028e5aa" status pending -> running" 2022-10-12T12:45:19.115Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a msg="Alloc "5cab3637-6449-d195-20b2-21abef7e1401" status pending -> running" 2022-10-12T12:45:19.117Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a msg="Alloc "794f1278-bc3b-bcb6-9c66-ff25755d684b" status pending -> running" 2022-10-12T12:45:19.907Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "998888eb-4b75-584d-c918-7faba8946bbc" status pending -> running" 2022-10-12T12:45:20.179Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "1ff4d72b-3f2e-0e3b-faf7-c7ba3a660807" status pending -> running" 2022-10-12T12:45:22.441Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 msg="Alloc "20a08c72-4293-1eaa-9ebb-1e150d6bac92" status pending -> running" 2022-10-12T12:45:22.909Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 msg="Alloc "a774895b-fd66-4836-d41e-8ca0fac235cc" status pending -> running" 2022-10-12T12:45:27.044Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="Alloc "1aedab96-7ceb-b5bd-5e15-7c80f99eb848" status pending -> running" 2022-10-12T12:45:27.044Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="Alloc "9b483e18-84ed-bfab-c971-64c4133333f9" status pending -> running" 2022-10-12T12:45:37.627Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection 2022-10-12T12:45:41.251Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=2b1bc5c7-e7f6-24e5-c224-a0b09bb6b7ec policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:45:41.251Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=2b1bc5c7-e7f6-24e5-c224-a0b09bb6b7ec policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:45:41.251Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=2b1bc5c7-e7f6-24e5-c224-a0b09bb6b7ec policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=79f8958f-927c-535c-bc52-26c6288cd7c8 2022-10-12T12:45:41.251Z [DEBUG] policy_eval.worker: received policy for evaluation: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:45:41.251Z [DEBUG] policy_eval.worker: fetching current count: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:45:41.251Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:45:41.252Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:45:41.252Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:44:41.252390694 +0000 UTC m=+29583.634892883 2022-10-12 12:45:41.252390694 +0000 UTC m=+29643.634892883}" 2022-10-12T12:45:41.529Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:45:41.529Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:45:41.529Z [DEBUG] policy_eval.worker: no checks need to be executed: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:45:41.529Z [DEBUG] policy_eval.broker: ack eval: eval_id=2b1bc5c7-e7f6-24e5-c224-a0b09bb6b7ec token=79f8958f-927c-535c-bc52-26c6288cd7c8 eval_id=2b1bc5c7-e7f6-24e5-c224-a0b09bb6b7ec token=79f8958f-927c-535c-bc52-26c6288cd7c8 2022-10-12T12:45:41.529Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:45:41.529Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:45:41.529Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:45:41.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=c5d97fed-34d4-f8f4-25be-a69e7aa69468 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:45:41.654Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=c5d97fed-34d4-f8f4-25be-a69e7aa69468 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:45:41.654Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=c5d97fed-34d4-f8f4-25be-a69e7aa69468 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=5d4e623e-b419-1fa6-8b38-b9224fd9dae3 2022-10-12T12:45:41.654Z [DEBUG] policy_eval.worker: received policy for evaluation: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:45:41.654Z [DEBUG] policy_eval.worker: fetching current count: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:45:41.654Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:45:41.655Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:45:41.655Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:44:41.655463205 +0000 UTC m=+29584.037965384 2022-10-12 12:45:41.655463205 +0000 UTC m=+29644.037965384}" 2022-10-12T12:45:41.935Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:45:41.935Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:45:41.935Z [DEBUG] policy_eval.worker: no checks need to be executed: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:45:41.935Z [DEBUG] policy_eval.broker: ack eval: eval_id=c5d97fed-34d4-f8f4-25be-a69e7aa69468 token=5d4e623e-b419-1fa6-8b38-b9224fd9dae3 eval_id=c5d97fed-34d4-f8f4-25be-a69e7aa69468 token=5d4e623e-b419-1fa6-8b38-b9224fd9dae3 2022-10-12T12:45:41.935Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:45:41.935Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:45:41.935Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:45:43.062Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:45:59.651Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="Alloc "01a079f5-940c-1e13-b5cb-6f88c1896a93" marked for migration" 2022-10-12T12:45:59.674Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="Alloc "01a079f5-940c-1e13-b5cb-6f88c1896a93" draining" 2022-10-12T12:46:01.197Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="Alloc "01a079f5-940c-1e13-b5cb-6f88c1896a93" status running -> complete" 2022-10-12T12:46:11.253Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=3248a5cf-b06d-c9f4-0bee-979a7fdedbcb policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:46:11.253Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=3248a5cf-b06d-c9f4-0bee-979a7fdedbcb policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:46:11.253Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=3248a5cf-b06d-c9f4-0bee-979a7fdedbcb policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=53bc8431-580e-1027-c305-36a1d59bdee4 2022-10-12T12:46:11.277Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:46:11.278Z [DEBUG] policy_eval.worker: fetching current count: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:46:11.278Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:46:11.279Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:46:11.279Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:45:11.279558234 +0000 UTC m=+29613.662060433 2022-10-12 12:46:11.279558234 +0000 UTC m=+29673.662060433}" 2022-10-12T12:46:11.574Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:46:11.576Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:46:11.577Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:46:11.577Z [DEBUG] policy_eval.broker: ack eval: eval_id=3248a5cf-b06d-c9f4-0bee-979a7fdedbcb token=53bc8431-580e-1027-c305-36a1d59bdee4 eval_id=3248a5cf-b06d-c9f4-0bee-979a7fdedbcb token=53bc8431-580e-1027-c305-36a1d59bdee4 2022-10-12T12:46:11.578Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:46:11.579Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:46:11.579Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:46:11.655Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=2553fdd9-aee1-4814-4c94-576c1a5947ea policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:46:11.655Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=2553fdd9-aee1-4814-4c94-576c1a5947ea policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:46:11.655Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=2553fdd9-aee1-4814-4c94-576c1a5947ea policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=ce24a18f-0c81-0b15-76c1-981d404a9dfd 2022-10-12T12:46:11.655Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:46:11.655Z [DEBUG] policy_eval.worker: fetching current count: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:46:11.655Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:46:11.656Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:46:11.656Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:45:11.656328939 +0000 UTC m=+29614.038831158 2022-10-12 12:46:11.656328939 +0000 UTC m=+29674.038831158}" 2022-10-12T12:46:11.962Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:46:11.962Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:46:11.962Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:46:11.964Z [DEBUG] policy_eval.broker: ack eval: eval_id=2553fdd9-aee1-4814-4c94-576c1a5947ea token=ce24a18f-0c81-0b15-76c1-981d404a9dfd eval_id=2553fdd9-aee1-4814-4c94-576c1a5947ea token=ce24a18f-0c81-0b15-76c1-981d404a9dfd 2022-10-12T12:46:11.964Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:46:11.964Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:46:11.964Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:46:13.121Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:46:37.627Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection 2022-10-12T12:46:41.252Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=f0805e26-66ef-3c99-329f-6a9ac09ec9e9 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:46:41.252Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=f0805e26-66ef-3c99-329f-6a9ac09ec9e9 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:46:41.252Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=f0805e26-66ef-3c99-329f-6a9ac09ec9e9 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=994e69ec-c40e-6eca-f92b-843319e4374e 2022-10-12T12:46:41.252Z [DEBUG] policy_eval.worker: received policy for evaluation: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:46:41.252Z [DEBUG] policy_eval.worker: fetching current count: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:46:41.252Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:46:41.252Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:46:41.252Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:45:41.252708749 +0000 UTC m=+29643.635210928 2022-10-12 12:46:41.252708749 +0000 UTC m=+29703.635210928}" 2022-10-12T12:46:41.533Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:46:41.533Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:46:41.533Z [DEBUG] policy_eval.worker: no checks need to be executed: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:46:41.533Z [DEBUG] policy_eval.broker: ack eval: eval_id=f0805e26-66ef-3c99-329f-6a9ac09ec9e9 token=994e69ec-c40e-6eca-f92b-843319e4374e eval_id=f0805e26-66ef-3c99-329f-6a9ac09ec9e9 token=994e69ec-c40e-6eca-f92b-843319e4374e 2022-10-12T12:46:41.533Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:46:41.533Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:46:41.534Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:46:41.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=024cd713-7f55-20cb-7979-6416d00bc9b8 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:46:41.653Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=024cd713-7f55-20cb-7979-6416d00bc9b8 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:46:41.653Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=024cd713-7f55-20cb-7979-6416d00bc9b8 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=cd35de37-8131-020b-e2f5-cb61a005b3fa 2022-10-12T12:46:41.654Z [DEBUG] policy_eval.worker: received policy for evaluation: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:46:41.654Z [DEBUG] policy_eval.worker: fetching current count: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:46:41.654Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:46:41.654Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:46:41.654Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:45:41.654554148 +0000 UTC m=+29644.037056327 2022-10-12 12:46:41.654554148 +0000 UTC m=+29704.037056327}" 2022-10-12T12:46:41.933Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:46:41.933Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:46:41.933Z [DEBUG] policy_eval.worker: no checks need to be executed: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:46:41.933Z [DEBUG] policy_eval.broker: ack eval: eval_id=024cd713-7f55-20cb-7979-6416d00bc9b8 token=cd35de37-8131-020b-e2f5-cb61a005b3fa eval_id=024cd713-7f55-20cb-7979-6416d00bc9b8 token=cd35de37-8131-020b-e2f5-cb61a005b3fa 2022-10-12T12:46:41.933Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:46:41.933Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:46:41.933Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:46:43.072Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:47:11.252Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=c19f5125-d8a6-8c5b-c71d-59b85c6e5095 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:47:11.252Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=c19f5125-d8a6-8c5b-c71d-59b85c6e5095 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:47:11.252Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=c19f5125-d8a6-8c5b-c71d-59b85c6e5095 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=83752821-83a8-182a-0daa-1cec1f077a37 2022-10-12T12:47:11.252Z [DEBUG] policy_eval.worker: received policy for evaluation: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:47:11.252Z [DEBUG] policy_eval.worker: fetching current count: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:47:11.253Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:47:11.253Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:47:11.253Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:46:11.253270209 +0000 UTC m=+29673.635772398 2022-10-12 12:47:11.253270209 +0000 UTC m=+29733.635772398}" 2022-10-12T12:47:11.547Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:47:11.548Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:47:11.550Z [DEBUG] policy_eval.worker: no checks need to be executed: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:47:11.550Z [DEBUG] policy_eval.broker: ack eval: eval_id=c19f5125-d8a6-8c5b-c71d-59b85c6e5095 token=83752821-83a8-182a-0daa-1cec1f077a37 eval_id=c19f5125-d8a6-8c5b-c71d-59b85c6e5095 token=83752821-83a8-182a-0daa-1cec1f077a37 2022-10-12T12:47:11.550Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:47:11.550Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:47:11.550Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:47:11.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=a4d14825-bcfe-2d08-23b5-8f8a698fdaa4 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:47:11.654Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=a4d14825-bcfe-2d08-23b5-8f8a698fdaa4 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:47:11.654Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=a4d14825-bcfe-2d08-23b5-8f8a698fdaa4 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=af716a26-d7de-893f-e30a-4d97abc07ff7 2022-10-12T12:47:11.654Z [DEBUG] policy_eval.worker: received policy for evaluation: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:47:11.654Z [DEBUG] policy_eval.worker: fetching current count: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:47:11.654Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:47:11.654Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:47:11.654Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:46:11.654255368 +0000 UTC m=+29674.036757557 2022-10-12 12:47:11.654255368 +0000 UTC m=+29734.036757557}" 2022-10-12T12:47:11.929Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:47:11.929Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:47:11.929Z [DEBUG] policy_eval.worker: no checks need to be executed: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:47:11.929Z [DEBUG] policy_eval.broker: ack eval: eval_id=a4d14825-bcfe-2d08-23b5-8f8a698fdaa4 token=af716a26-d7de-893f-e30a-4d97abc07ff7 eval_id=a4d14825-bcfe-2d08-23b5-8f8a698fdaa4 token=af716a26-d7de-893f-e30a-4d97abc07ff7 2022-10-12T12:47:11.929Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:47:11.929Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:47:11.929Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:47:13.078Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:47:37.627Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection 2022-10-12T12:47:41.261Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=abd61227-3d46-1ce3-92ad-2d89c685deb9 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:47:41.261Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=abd61227-3d46-1ce3-92ad-2d89c685deb9 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:47:41.262Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=abd61227-3d46-1ce3-92ad-2d89c685deb9 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=d1f91cc5-e93c-945a-03cb-871d8cffa845 2022-10-12T12:47:41.262Z [DEBUG] policy_eval.worker: received policy for evaluation: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:47:41.262Z [DEBUG] policy_eval.worker: fetching current count: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:47:41.262Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:47:41.264Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:47:41.265Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:46:41.265360545 +0000 UTC m=+29703.647862734 2022-10-12 12:47:41.265360545 +0000 UTC m=+29763.647862734}" 2022-10-12T12:47:41.550Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:47:41.550Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:47:41.550Z [DEBUG] policy_eval.worker: no checks need to be executed: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:47:41.550Z [DEBUG] policy_eval.broker: ack eval: eval_id=abd61227-3d46-1ce3-92ad-2d89c685deb9 token=d1f91cc5-e93c-945a-03cb-871d8cffa845 eval_id=abd61227-3d46-1ce3-92ad-2d89c685deb9 token=d1f91cc5-e93c-945a-03cb-871d8cffa845 2022-10-12T12:47:41.551Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:47:41.551Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:47:41.551Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:47:41.667Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=eac99709-4f8a-1d63-d1ae-09666ca97b68 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:47:41.667Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=eac99709-4f8a-1d63-d1ae-09666ca97b68 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:47:41.667Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=eac99709-4f8a-1d63-d1ae-09666ca97b68 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=7635ed09-138e-96ab-4068-400e2ce2e0a2 2022-10-12T12:47:41.667Z [DEBUG] policy_eval.worker: received policy for evaluation: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:47:41.667Z [DEBUG] policy_eval.worker: fetching current count: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:47:41.668Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:47:41.668Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:47:41.668Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:46:41.668268761 +0000 UTC m=+29704.050770940 2022-10-12 12:47:41.668268761 +0000 UTC m=+29764.050770940}" 2022-10-12T12:47:41.944Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:47:41.945Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:47:41.945Z [DEBUG] policy_eval.worker: no checks need to be executed: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:47:41.945Z [DEBUG] policy_eval.broker: ack eval: eval_id=eac99709-4f8a-1d63-d1ae-09666ca97b68 token=7635ed09-138e-96ab-4068-400e2ce2e0a2 eval_id=eac99709-4f8a-1d63-d1ae-09666ca97b68 token=7635ed09-138e-96ab-4068-400e2ce2e0a2 2022-10-12T12:47:41.945Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:47:41.945Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:47:41.945Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:47:50.237Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:48:11.252Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=138494e6-da2f-d861-9151-709e1a30455e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:48:11.252Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=138494e6-da2f-d861-9151-709e1a30455e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:48:11.252Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=138494e6-da2f-d861-9151-709e1a30455e policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=4db40b06-2dcc-742d-11ec-d63bd71cf934 2022-10-12T12:48:11.253Z [DEBUG] policy_eval.worker: received policy for evaluation: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:48:11.253Z [DEBUG] policy_eval.worker: fetching current count: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:48:11.253Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:48:11.253Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:48:11.254Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:47:11.254075062 +0000 UTC m=+29733.636577251 2022-10-12 12:48:11.254075062 +0000 UTC m=+29793.636577251}" 2022-10-12T12:48:11.557Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:48:11.557Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:48:11.557Z [DEBUG] policy_eval.worker: no checks need to be executed: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:48:11.557Z [DEBUG] policy_eval.broker: ack eval: eval_id=138494e6-da2f-d861-9151-709e1a30455e token=4db40b06-2dcc-742d-11ec-d63bd71cf934 eval_id=138494e6-da2f-d861-9151-709e1a30455e token=4db40b06-2dcc-742d-11ec-d63bd71cf934 2022-10-12T12:48:11.561Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:48:11.562Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:48:11.572Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:48:11.654Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=ce04eb35-b13a-bc9c-e865-055038222b98 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:48:11.654Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=ce04eb35-b13a-bc9c-e865-055038222b98 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:48:11.654Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=ce04eb35-b13a-bc9c-e865-055038222b98 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=e1500efe-5976-d970-8128-81b87833d842 2022-10-12T12:48:11.663Z [DEBUG] policy_eval.worker: received policy for evaluation: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:48:11.676Z [DEBUG] policy_eval.worker: fetching current count: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:48:11.678Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:48:11.693Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:48:11.718Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:47:11.718066794 +0000 UTC m=+29734.100569093 2022-10-12 12:48:11.718066794 +0000 UTC m=+29794.100569093}" 2022-10-12T12:48:12.018Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:48:12.039Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:48:12.040Z [DEBUG] policy_eval.worker: no checks need to be executed: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:48:12.055Z [DEBUG] policy_eval.broker: ack eval: eval_id=ce04eb35-b13a-bc9c-e865-055038222b98 token=e1500efe-5976-d970-8128-81b87833d842 eval_id=ce04eb35-b13a-bc9c-e865-055038222b98 token=e1500efe-5976-d970-8128-81b87833d842 2022-10-12T12:48:12.058Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:48:12.060Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:48:12.065Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:48:13.679Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:48:37.629Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection 2022-10-12T12:48:41.254Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=6db893ba-4b98-a32d-75cc-0f9c15ed02d6 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:48:41.254Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=6db893ba-4b98-a32d-75cc-0f9c15ed02d6 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:48:41.254Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=6db893ba-4b98-a32d-75cc-0f9c15ed02d6 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=2e651d65-bbb5-ba1c-e787-36d10f1d12e8 2022-10-12T12:48:41.256Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:48:41.256Z [DEBUG] policy_eval.worker: fetching current count: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:48:41.257Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:48:41.279Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:48:41.280Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:47:41.280256801 +0000 UTC m=+29763.662759060 2022-10-12 12:48:41.280256801 +0000 UTC m=+29823.662759060}" 2022-10-12T12:48:41.572Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:48:41.584Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:48:41.584Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:48:41.602Z [DEBUG] policy_eval.broker: ack eval: eval_id=6db893ba-4b98-a32d-75cc-0f9c15ed02d6 token=2e651d65-bbb5-ba1c-e787-36d10f1d12e8 eval_id=6db893ba-4b98-a32d-75cc-0f9c15ed02d6 token=2e651d65-bbb5-ba1c-e787-36d10f1d12e8 2022-10-12T12:48:41.602Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:48:41.603Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:48:41.603Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:48:41.662Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=18a35ef2-17dc-7a45-3777-2763e93aa1bc policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:48:41.662Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=18a35ef2-17dc-7a45-3777-2763e93aa1bc policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:48:41.662Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=18a35ef2-17dc-7a45-3777-2763e93aa1bc policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=9d425196-7473-bdd5-7749-b4e3eaa79ce0 2022-10-12T12:48:41.662Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:48:41.662Z [DEBUG] policy_eval.worker: fetching current count: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:48:41.663Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:48:41.663Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:48:41.663Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:47:41.663781757 +0000 UTC m=+29764.046283996 2022-10-12 12:48:41.663781757 +0000 UTC m=+29824.046283996}" 2022-10-12T12:48:41.959Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:48:41.961Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:48:41.962Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:48:41.962Z [DEBUG] policy_eval.broker: ack eval: eval_id=18a35ef2-17dc-7a45-3777-2763e93aa1bc token=9d425196-7473-bdd5-7749-b4e3eaa79ce0 eval_id=18a35ef2-17dc-7a45-3777-2763e93aa1bc token=9d425196-7473-bdd5-7749-b4e3eaa79ce0 2022-10-12T12:48:41.963Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:48:41.963Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:48:41.968Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:48:43.339Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:49:11.252Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=44dfc083-40a7-57f3-14ab-8082b08950a3 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:49:11.252Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=44dfc083-40a7-57f3-14ab-8082b08950a3 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:49:11.252Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=44dfc083-40a7-57f3-14ab-8082b08950a3 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=053eb13b-5dda-d23d-1346-6462fc3255d5 2022-10-12T12:49:11.252Z [DEBUG] policy_eval.worker: received policy for evaluation: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:49:11.252Z [DEBUG] policy_eval.worker: fetching current count: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:49:11.253Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:49:11.253Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:49:11.254Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:48:11.254136979 +0000 UTC m=+29793.636639178 2022-10-12 12:49:11.254136979 +0000 UTC m=+29853.636639178}" 2022-10-12T12:49:11.553Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:49:11.553Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:49:11.553Z [DEBUG] policy_eval.worker: no checks need to be executed: id=d151b4b6-49ae-f7cb-979a-5a35ccec2090 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:49:11.553Z [DEBUG] policy_eval.broker: ack eval: eval_id=44dfc083-40a7-57f3-14ab-8082b08950a3 token=053eb13b-5dda-d23d-1346-6462fc3255d5 eval_id=44dfc083-40a7-57f3-14ab-8082b08950a3 token=053eb13b-5dda-d23d-1346-6462fc3255d5 2022-10-12T12:49:11.553Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:49:11.553Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:49:11.553Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:49:11.661Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=33b359b0-975e-63dc-a6a2-6460f1219125 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:49:11.661Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=33b359b0-975e-63dc-a6a2-6460f1219125 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:49:11.661Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=33b359b0-975e-63dc-a6a2-6460f1219125 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=0af0973d-2c81-5e97-c52e-e8fb4746a1b5 2022-10-12T12:49:11.661Z [DEBUG] policy_eval.worker: received policy for evaluation: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:49:11.661Z [DEBUG] policy_eval.worker: fetching current count: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:49:11.661Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:49:11.661Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:49:11.661Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:48:11.661733465 +0000 UTC m=+29794.044235644 2022-10-12 12:49:11.661733465 +0000 UTC m=+29854.044235644}" 2022-10-12T12:49:11.959Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:49:11.959Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:49:11.959Z [DEBUG] policy_eval.worker: no checks need to be executed: id=879db42c-dbb1-e701-f64b-3335941fc044 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:49:11.959Z [DEBUG] policy_eval.broker: ack eval: eval_id=33b359b0-975e-63dc-a6a2-6460f1219125 token=0af0973d-2c81-5e97-c52e-e8fb4746a1b5 eval_id=33b359b0-975e-63dc-a6a2-6460f1219125 token=0af0973d-2c81-5e97-c52e-e8fb4746a1b5 2022-10-12T12:49:11.959Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:49:11.959Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:49:11.959Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:49:13.180Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:49:37.629Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection 2022-10-12T12:49:41.264Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=9bbd9957-ead8-cd23-a7ed-5ce1452e23f7 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:49:41.264Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=9bbd9957-ead8-cd23-a7ed-5ce1452e23f7 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:49:41.264Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=9bbd9957-ead8-cd23-a7ed-5ce1452e23f7 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=f3466a7c-9bb5-f788-83d4-9b3cc51eb332 2022-10-12T12:49:41.265Z [DEBUG] policy_eval.worker: received policy for evaluation: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:49:41.265Z [DEBUG] policy_eval.worker: fetching current count: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:49:41.265Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:49:41.265Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:49:41.265Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:48:41.265884031 +0000 UTC m=+29823.648386210 2022-10-12 12:49:41.265884031 +0000 UTC m=+29883.648386210}" 2022-10-12T12:49:41.553Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:49:41.554Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:49:41.554Z [DEBUG] policy_eval.worker: no checks need to be executed: id=95c7a741-1cad-02be-4744-a09e0bfa0d90 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:49:41.554Z [DEBUG] policy_eval.broker: ack eval: eval_id=9bbd9957-ead8-cd23-a7ed-5ce1452e23f7 token=f3466a7c-9bb5-f788-83d4-9b3cc51eb332 eval_id=9bbd9957-ead8-cd23-a7ed-5ce1452e23f7 token=f3466a7c-9bb5-f788-83d4-9b3cc51eb332 2022-10-12T12:49:41.554Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:49:41.554Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:49:41.554Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:49:41.655Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=535dc33b-8e70-edbb-2e0f-d3b12c324e7f policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:49:41.655Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=535dc33b-8e70-edbb-2e0f-d3b12c324e7f policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:49:41.655Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=535dc33b-8e70-edbb-2e0f-d3b12c324e7f policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=3894cd7f-1228-fd85-fc06-6f7294a87c69 2022-10-12T12:49:41.655Z [DEBUG] policy_eval.worker: received policy for evaluation: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:49:41.655Z [DEBUG] policy_eval.worker: fetching current count: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:49:41.656Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:49:41.656Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:49:41.656Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:48:41.656792854 +0000 UTC m=+29824.039295033 2022-10-12 12:49:41.656792854 +0000 UTC m=+29884.039295033}" 2022-10-12T12:49:41.947Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:49:41.948Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:49:41.948Z [DEBUG] policy_eval.worker: no checks need to be executed: id=2be86b14-e410-c5ef-6d0e-fda94ed9d0fb policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:49:41.948Z [DEBUG] policy_eval.broker: ack eval: eval_id=535dc33b-8e70-edbb-2e0f-d3b12c324e7f token=3894cd7f-1228-fd85-fc06-6f7294a87c69 eval_id=535dc33b-8e70-edbb-2e0f-d3b12c324e7f token=3894cd7f-1228-fd85-fc06-6f7294a87c69 2022-10-12T12:49:41.949Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:49:41.949Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:49:41.949Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:49:43.199Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:50:11.252Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=bd3767dc-2388-fe9b-2bbb-00b59307fed1 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:50:11.252Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=bd3767dc-2388-fe9b-2bbb-00b59307fed1 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:50:11.252Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=bd3767dc-2388-fe9b-2bbb-00b59307fed1 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=27e69aab-3fe5-6d4d-1f74-b94509f6a329 2022-10-12T12:50:11.252Z [DEBUG] policy_eval.worker: received policy for evaluation: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:50:11.252Z [DEBUG] policy_eval.worker: fetching current count: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:50:11.252Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:50:11.252Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:50:11.252Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:49:11.252692426 +0000 UTC m=+29853.635194605 2022-10-12 12:50:11.252692426 +0000 UTC m=+29913.635194605}" 2022-10-12T12:50:11.533Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:50:11.533Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:50:11.533Z [DEBUG] policy_eval.worker: no checks need to be executed: id=bf884d53-ac5b-c8ae-eac4-1e4ee5333d45 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:50:11.533Z [DEBUG] policy_eval.broker: ack eval: eval_id=bd3767dc-2388-fe9b-2bbb-00b59307fed1 token=27e69aab-3fe5-6d4d-1f74-b94509f6a329 eval_id=bd3767dc-2388-fe9b-2bbb-00b59307fed1 token=27e69aab-3fe5-6d4d-1f74-b94509f6a329 2022-10-12T12:50:11.533Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:50:11.533Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:50:11.533Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:50:11.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e38ba173-7a58-8d50-4ccd-3b082d4897af policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:50:11.654Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=e38ba173-7a58-8d50-4ccd-3b082d4897af policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:50:11.654Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=e38ba173-7a58-8d50-4ccd-3b082d4897af policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=935b4a0d-94d0-ffc2-94a5-6cf16861d7a0 2022-10-12T12:50:11.654Z [DEBUG] policy_eval.worker: received policy for evaluation: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:50:11.654Z [DEBUG] policy_eval.worker: fetching current count: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:50:11.654Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:50:11.654Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:50:11.654Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:49:11.654867004 +0000 UTC m=+29854.037369183 2022-10-12 12:50:11.654867004 +0000 UTC m=+29914.037369183}" 2022-10-12T12:50:11.962Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=100 2022-10-12T12:50:11.962Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=worker_scaling id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target from=100 to=100 2022-10-12T12:50:11.962Z [DEBUG] policy_eval.worker: no checks need to be executed: id=bd28eb86-d50c-312c-f6d4-ae576ff08e03 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:50:11.962Z [DEBUG] policy_eval.broker: ack eval: eval_id=e38ba173-7a58-8d50-4ccd-3b082d4897af token=935b4a0d-94d0-ffc2-94a5-6cf16861d7a0 eval_id=e38ba173-7a58-8d50-4ccd-3b082d4897af token=935b4a0d-94d0-ffc2-94a5-6cf16861d7a0 2022-10-12T12:50:11.962Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:50:11.962Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:50:11.962Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:50:13.069Z [WARN] internal_plugin.aws-asg: node pool status readiness check failed: error="node 18a728a6-26dc-f161-bab4-0fce3e63fba1 is draining, node 5ac68c24-bb30-44f2-5829-581b1623895a is draining, node 76f875dd-8abf-d363-cfc4-7ff548f864b7 is draining, node 47e6591c-1d9b-0231-df05-089262080217 is draining, node 75fed26e-3dd5-235f-b78f-82ec4d7f89da is draining, node 05ceef1a-7bcf-7e7a-8794-88ba16dde885 is draining, node 7ae96fbf-6acf-58a6-a4fc-8217659265ef is draining, node 5e7e6984-7f6b-b383-d860-f6779cb6097a is draining, node f883811d-4da7-0c55-b59d-49dbb6059c38 is draining, node aa989f2d-52ba-86f5-954f-947b2a59bb06 is draining" 2022-10-12T12:50:13.385Z [DEBUG] policy_eval.broker: nack eval: eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee token=5261ed5e-4f6d-2f87-ed53-d03d66b2b87c 2022-10-12T12:50:13.385Z [WARN] policy_eval.broker: eval delivery limit reached: eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 token=5261ed5e-4f6d-2f87-ed53-d03d66b2b87c count=1 limit=1 2022-10-12T12:50:16.334Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="Alloc "4800219a-0d4f-e82d-f277-7e6f07c986d4" marked for migration" 2022-10-12T12:50:16.405Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="Alloc "325f6c32-9798-d5c7-0c4e-2359c959bed6" marked for migration" 2022-10-12T12:50:16.405Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="Alloc "d12838eb-23c6-bb07-5ca9-41867f91683e" marked for migration" 2022-10-12T12:50:16.533Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a msg="Alloc "5cab3637-6449-d195-20b2-21abef7e1401" marked for migration" 2022-10-12T12:50:16.534Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a msg="Alloc "794f1278-bc3b-bcb6-9c66-ff25755d684b" marked for migration" 2022-10-12T12:50:16.539Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="Alloc "a84f1e6b-58bd-87df-8d6a-278cdd422ce3" marked for migration" 2022-10-12T12:50:16.543Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="Alloc "ec56cc6a-a4d8-4eed-ab97-20789fb8e7d0" marked for migration" 2022-10-12T12:50:16.599Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 msg="Drain complete for node 7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3" 2022-10-12T12:50:16.621Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd msg="Alloc "259f1ad3-ea89-dd3e-52c2-1653125c7ad8" marked for migration" 2022-10-12T12:50:16.624Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd msg="Alloc "7d506de4-6999-36a6-519d-fdc9e1dc91aa" marked for migration" 2022-10-12T12:50:16.624Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="Alloc "537b6bad-cdfe-75d7-f6dd-58b78fa3c623" marked for migration" 2022-10-12T12:50:16.624Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="Alloc "7cfe67b2-d07f-3110-85a1-109dcad31fca" marked for migration" 2022-10-12T12:50:16.628Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="Drain complete for node 85afe4c6-a56e-a6af-3fee-a9c9a7980f44" 2022-10-12T12:50:16.647Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="Alloc "18f15905-8ddc-c718-98d9-3ceef4f177ce" marked for migration" 2022-10-12T12:50:16.653Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="Alloc "54200208-86b7-7832-b758-7d67a21506a2" marked for migration" 2022-10-12T12:50:16.647Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="Drain complete for node bb7294e1-d61f-9582-6c49-6bbbdd4f86e4" 2022-10-12T12:50:16.630Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a2cab557-2a69-7ad6-554c-a7789d197484" marked for migration" 2022-10-12T12:50:16.656Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a92f8ca4-95b2-dd53-e97d-da908028e5aa" marked for migration" 2022-10-12T12:50:16.657Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 msg="Alloc "1775f750-4a5c-b2e8-c8f4-8a16044d0b0c" marked for migration" 2022-10-12T12:50:16.658Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 msg="Alloc "f8c8ae2c-469b-2c75-d59e-5c862b6b7bf3" marked for migration" 2022-10-12T12:50:16.643Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 msg="Alloc "20a08c72-4293-1eaa-9ebb-1e150d6bac92" marked for migration" 2022-10-12T12:50:16.658Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 msg="Alloc "a774895b-fd66-4836-d41e-8ca0fac235cc" marked for migration" 2022-10-12T12:50:16.643Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="Alloc "cb3ff2ea-c927-018d-c0b9-438d4be0754e" marked for migration" 2022-10-12T12:50:16.659Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="Alloc "dba816ac-bfd0-031d-1733-b56abaedf76e" marked for migration" 2022-10-12T12:50:16.643Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="Alloc "d0bed79f-ffef-bafd-84c9-d3e7f98b7b04" marked for migration" 2022-10-12T12:50:16.660Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="Alloc "dbdb1e18-ad4a-2f81-8fa6-d0801e1b4ff7" marked for migration" 2022-10-12T12:50:16.644Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="Drain complete for node 5e7e6984-7f6b-b383-d860-f6779cb6097a" 2022-10-12T12:50:16.644Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="Drain complete for node 11c4df5a-606d-4222-7def-f512740f890a" 2022-10-12T12:50:16.664Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="Alloc "1f5d9d18-ff48-4857-7e31-50c8626387c4" marked for migration" 2022-10-12T12:50:16.647Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="Alloc "9af8e510-918e-527b-2c2b-e7896b77e6f2" marked for migration" 2022-10-12T12:50:16.648Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="Drain complete for node 99157452-ba8f-2c4e-73c3-a297a89914a2" 2022-10-12T12:50:16.651Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a msg="Drain complete for node 728f43fa-8d77-7878-5bd9-7b66290dce5a" 2022-10-12T12:50:16.651Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="Drain complete for node e5ce4421-98ca-71b8-4395-2debd69a5424" 2022-10-12T12:50:16.657Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Drain complete for node 5ac68c24-bb30-44f2-5829-581b1623895a" 2022-10-12T12:50:16.664Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="Drain complete for node 1da27d6e-4649-6203-acd8-c9ac90ce998f" 2022-10-12T12:50:16.664Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7c995302-018d-fc7f-9b66-8cb242122581 msg="Alloc "31ceee79-2c0b-000e-09c0-6b9b8d6ad399" marked for migration" 2022-10-12T12:50:16.628Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e msg="Alloc "54d699a6-3b93-b902-df11-9959346c23eb" marked for migration" 2022-10-12T12:50:16.638Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="Alloc "3209981e-3777-d11b-6c3c-3c498e6dde27" marked for migration" 2022-10-12T12:50:16.675Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="Alloc "537b6bad-cdfe-75d7-f6dd-58b78fa3c623" draining" 2022-10-12T12:50:16.697Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="Alloc "cb3ff2ea-c927-018d-c0b9-438d4be0754e" draining" 2022-10-12T12:50:16.697Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="Alloc "dba816ac-bfd0-031d-1733-b56abaedf76e" draining" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="Alloc "50c71b2b-f6b0-18ad-3869-56d5c712fa9b" marked for migration" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="Alloc "b09c0db2-5de2-921c-df6e-1477729e7f41" marked for migration" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="Alloc "0d380697-e251-1c8d-d3b4-f2a5a6cfb6e2" marked for migration" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="Alloc "a8268eb0-5908-532d-29d2-1b89ed06598e" marked for migration" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7c995302-018d-fc7f-9b66-8cb242122581 msg="Alloc "f19ba0a3-d0e5-9bb3-9640-4af5b3a61806" marked for migration" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e msg="Alloc "650a1490-f6cf-b413-96b0-40d334f52edb" marked for migration" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="Alloc "b83cc2c1-798d-26f6-dcad-805a6716a833" marked for migration" 2022-10-12T12:50:16.701Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="Alloc "7cfe67b2-d07f-3110-85a1-109dcad31fca" draining" 2022-10-12T12:50:16.701Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="Drain complete for node 5b79ee9c-b72b-6391-6602-3e479aa505c7" 2022-10-12T12:50:16.705Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a2cab557-2a69-7ad6-554c-a7789d197484" draining" 2022-10-12T12:50:16.705Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a92f8ca4-95b2-dd53-e97d-da908028e5aa" draining" 2022-10-12T12:50:16.707Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="Drain complete for node aa989f2d-52ba-86f5-954f-947b2a59bb06" 2022-10-12T12:50:16.707Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="Alloc "3c2a3f24-e3f0-d722-ca2b-dc9f782ba876" marked for migration" 2022-10-12T12:50:16.707Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="Alloc "54142198-2062-c051-d160-25758db284c4" marked for migration" 2022-10-12T12:50:16.708Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd msg="Drain complete for node b494a874-6a2d-bb07-5cfe-4c9fda8edefd" 2022-10-12T12:50:16.709Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="Alloc "74364d89-cbd2-4fea-980a-6f1601756d7b" marked for migration" 2022-10-12T12:50:16.709Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="Alloc "75dc5a8a-ff13-42e3-9deb-aab39065ea4d" marked for migration" 2022-10-12T12:50:16.709Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="Drain complete for node 725bd410-572a-1260-9a7a-1bb5a369afde" 2022-10-12T12:50:16.709Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="Alloc "74364d89-cbd2-4fea-980a-6f1601756d7b" draining" 2022-10-12T12:50:16.709Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="Alloc "75dc5a8a-ff13-42e3-9deb-aab39065ea4d" draining" 2022-10-12T12:50:16.733Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b8c7d206-607b-7d51-945a-75179b50485c msg="Alloc "45536096-13f0-e55a-045d-ed80821ea2c8" marked for migration" 2022-10-12T12:50:16.733Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b8c7d206-607b-7d51-945a-75179b50485c msg="Alloc "7179ac4a-31ac-fb5e-706b-f578343296f8" marked for migration" 2022-10-12T12:50:16.734Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="Alloc "3209981e-3777-d11b-6c3c-3c498e6dde27" draining" 2022-10-12T12:50:16.734Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="Alloc "b83cc2c1-798d-26f6-dcad-805a6716a833" draining" 2022-10-12T12:50:16.734Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="Alloc "18f15905-8ddc-c718-98d9-3ceef4f177ce" draining" 2022-10-12T12:50:16.734Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="Alloc "54200208-86b7-7832-b758-7d67a21506a2" draining" 2022-10-12T12:50:16.740Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="Alloc "4800219a-0d4f-e82d-f277-7e6f07c986d4" draining" 2022-10-12T12:50:16.741Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef msg="Alloc "37360026-c618-b5c3-4f56-db0be138a908" marked for migration" 2022-10-12T12:50:16.741Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef msg="Alloc "430f3843-4537-b630-60b7-722605603f69" marked for migration" 2022-10-12T12:50:16.760Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "1ff4d72b-3f2e-0e3b-faf7-c7ba3a660807" marked for migration" 2022-10-12T12:50:16.760Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "998888eb-4b75-584d-c918-7faba8946bbc" marked for migration" 2022-10-12T12:50:16.760Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "1ff4d72b-3f2e-0e3b-faf7-c7ba3a660807" draining" 2022-10-12T12:50:16.760Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "998888eb-4b75-584d-c918-7faba8946bbc" draining" 2022-10-12T12:50:16.761Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="Alloc "54b6068f-2b37-f331-824b-536226329574" marked for migration" 2022-10-12T12:50:16.761Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="Alloc "ad1d8768-6dfb-c384-0091-6d2128647c87" marked for migration" 2022-10-12T12:50:16.764Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="Alloc "026a6893-b2bd-a6d7-38b5-88fd5d18cf01" marked for migration" 2022-10-12T12:50:16.764Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="Alloc "f279acd7-b120-cedd-d432-18c23792ce67" marked for migration" 2022-10-12T12:50:16.806Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="Drain complete for node 18a728a6-26dc-f161-bab4-0fce3e63fba1" 2022-10-12T12:50:16.813Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="Drain complete for node f883811d-4da7-0c55-b59d-49dbb6059c38" 2022-10-12T12:50:16.818Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e msg="Drain complete for node 4ea7bfca-b9bf-67b4-1183-a19758d9626e" 2022-10-12T12:50:16.824Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="Alloc "d0bed79f-ffef-bafd-84c9-d3e7f98b7b04" draining" 2022-10-12T12:50:16.827Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="Alloc "dbdb1e18-ad4a-2f81-8fa6-d0801e1b4ff7" draining" 2022-10-12T12:50:16.824Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="Alloc "30a2f5aa-9971-ba0a-22f9-39266fc6501a" marked for migration" 2022-10-12T12:50:16.827Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="Alloc "59c07429-9b5c-7202-eea1-9a316e8701b2" marked for migration" 2022-10-12T12:50:16.827Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="Drain complete for node 05ceef1a-7bcf-7e7a-8794-88ba16dde885" 2022-10-12T12:50:16.824Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="Alloc "0d380697-e251-1c8d-d3b4-f2a5a6cfb6e2" draining" 2022-10-12T12:50:16.828Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="Alloc "a8268eb0-5908-532d-29d2-1b89ed06598e" draining" 2022-10-12T12:50:16.826Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="Alloc "3c2a3f24-e3f0-d722-ca2b-dc9f782ba876" draining" 2022-10-12T12:50:16.828Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="Alloc "54142198-2062-c051-d160-25758db284c4" draining" 2022-10-12T12:50:16.826Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="Drain complete for node b779c284-396f-deff-3de9-02df51ccf024" 2022-10-12T12:50:16.828Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="Alloc "1f5d9d18-ff48-4857-7e31-50c8626387c4" draining" 2022-10-12T12:50:16.828Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="Alloc "50c71b2b-f6b0-18ad-3869-56d5c712fa9b" draining" 2022-10-12T12:50:16.835Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="Alloc "325f6c32-9798-d5c7-0c4e-2359c959bed6" draining" 2022-10-12T12:50:16.835Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="Alloc "d12838eb-23c6-bb07-5ca9-41867f91683e" draining" 2022-10-12T12:50:16.847Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="Alloc "9af8e510-918e-527b-2c2b-e7896b77e6f2" draining" 2022-10-12T12:50:16.847Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="Alloc "b09c0db2-5de2-921c-df6e-1477729e7f41" draining" 2022-10-12T12:50:16.847Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="Alloc "66f7a070-ca0e-e273-fb94-a9cb381bb30b" marked for migration" 2022-10-12T12:50:16.847Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="Alloc "d022cadd-c62c-02fc-3f92-213dff347b8b" marked for migration" 2022-10-12T12:50:16.847Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="Alloc "73bd9591-1dcf-80a3-dfb5-d0cf53db7426" marked for migration" 2022-10-12T12:50:16.847Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="Alloc "94b51b82-0354-ad16-27f4-22917f8a0c88" marked for migration" 2022-10-12T12:50:16.848Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="Alloc "73bd9591-1dcf-80a3-dfb5-d0cf53db7426" draining" 2022-10-12T12:50:16.848Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="Alloc "94b51b82-0354-ad16-27f4-22917f8a0c88" draining" 2022-10-12T12:50:16.850Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="Alloc "026a6893-b2bd-a6d7-38b5-88fd5d18cf01" draining" 2022-10-12T12:50:16.850Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="Alloc "f279acd7-b120-cedd-d432-18c23792ce67" draining" 2022-10-12T12:50:16.850Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="Alloc "54b6068f-2b37-f331-824b-536226329574" draining" 2022-10-12T12:50:16.850Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="Alloc "ad1d8768-6dfb-c384-0091-6d2128647c87" draining" 2022-10-12T12:50:16.851Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="Alloc "01c4e20f-9a7c-a013-5607-e457ea1227c1" marked for migration" 2022-10-12T12:50:16.851Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="Alloc "e4251841-deb1-f419-4a98-19475d5542f6" marked for migration" 2022-10-12T12:50:16.854Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="Drain complete for node 171aa5f2-7638-356e-9231-beaeb9d2ffcc" 2022-10-12T12:50:16.859Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="Drain complete for node 75fed26e-3dd5-235f-b78f-82ec4d7f89da" 2022-10-12T12:50:16.861Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="Alloc "a4dff6b8-78f9-6ccc-fbe2-d3971c151b02" marked for migration" 2022-10-12T12:50:16.863Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="Alloc "c6afa0a5-73c3-90d1-864d-2d0b7be7f3d7" marked for migration" 2022-10-12T12:50:16.865Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="Alloc "30a2f5aa-9971-ba0a-22f9-39266fc6501a" draining" 2022-10-12T12:50:16.865Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="Alloc "59c07429-9b5c-7202-eea1-9a316e8701b2" draining" 2022-10-12T12:50:16.866Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="Alloc "206a8b39-8a78-aaa3-c2d5-b7151285a1b6" marked for migration" 2022-10-12T12:50:16.866Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="Alloc "5c71aeda-fb31-f34f-e058-9dc63c7cbd21" marked for migration" 2022-10-12T12:50:16.875Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=29b86c48-da66-2229-be80-361e607a99ce msg="Alloc "82a1c070-65b5-71e1-358b-3767f89ca46e" marked for migration" 2022-10-12T12:50:16.875Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=29b86c48-da66-2229-be80-361e607a99ce msg="Alloc "9e9121ac-fc99-0f21-1034-6d1fcb6c7d90" marked for migration" 2022-10-12T12:50:16.876Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="Drain complete for node b7b16595-e281-4f34-7189-c653c447f9ea" 2022-10-12T12:50:16.876Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4592c999-9240-e62d-edd3-5c3faa37129e msg="Alloc "6b8def25-2cbb-26d8-a30c-be3e9c82dc04" marked for migration" 2022-10-12T12:50:16.876Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4592c999-9240-e62d-edd3-5c3faa37129e msg="Alloc "f757a022-38fb-3d53-cf01-a6e060fc8835" marked for migration" 2022-10-12T12:50:16.878Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Drain complete for node dac51581-ec47-d653-e577-25ad4b30aa1d" 2022-10-12T12:50:16.879Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=29b86c48-da66-2229-be80-361e607a99ce msg="Drain complete for node 29b86c48-da66-2229-be80-361e607a99ce" 2022-10-12T12:50:16.882Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="Alloc "036ca648-d933-36db-24d5-a70ab013a13b" marked for migration" 2022-10-12T12:50:16.883Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="Alloc "09747377-d5d7-35d2-ccb1-0a3a440f3bc1" marked for migration" 2022-10-12T12:50:16.883Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="Drain complete for node 61219e70-61ef-042e-79e3-0eaed861b7d9" 2022-10-12T12:50:16.883Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="Alloc "a84f1e6b-58bd-87df-8d6a-278cdd422ce3" draining" 2022-10-12T12:50:16.883Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="Alloc "ec56cc6a-a4d8-4eed-ab97-20789fb8e7d0" draining" 2022-10-12T12:50:16.883Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a msg="Alloc "625e12d3-fe7d-8bf8-70e3-6a26d568f402" marked for migration" 2022-10-12T12:50:16.883Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a msg="Alloc "97527da1-06ef-6d4e-cc74-b550a7fc8b8d" marked for migration" 2022-10-12T12:50:16.884Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="Drain complete for node 2bf175c7-72a4-ce76-514e-cb6d1bbc5407" 2022-10-12T12:50:16.884Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=71d147c8-2031-02b5-feea-daee12267e19 msg="Drain complete for node 71d147c8-2031-02b5-feea-daee12267e19" 2022-10-12T12:50:16.886Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=7c995302-018d-fc7f-9b66-8cb242122581 msg="Drain complete for node 7c995302-018d-fc7f-9b66-8cb242122581" 2022-10-12T12:50:16.886Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="Alloc "1aedab96-7ceb-b5bd-5e15-7c80f99eb848" marked for migration" 2022-10-12T12:50:16.886Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="Alloc "9b483e18-84ed-bfab-c971-64c4133333f9" marked for migration" 2022-10-12T12:50:16.886Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="Drain complete for node 4adf69e2-c74d-fce5-27ca-177af5441565" 2022-10-12T12:50:16.888Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="Drain complete for node 47e6591c-1d9b-0231-df05-089262080217" 2022-10-12T12:50:16.892Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="Drain complete for node e79e037a-3200-89b0-6a87-b65688b52403" 2022-10-12T12:50:16.894Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=4592c999-9240-e62d-edd3-5c3faa37129e msg="Drain complete for node 4592c999-9240-e62d-edd3-5c3faa37129e" 2022-10-12T12:50:16.894Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b8c7d206-607b-7d51-945a-75179b50485c msg="Drain complete for node b8c7d206-607b-7d51-945a-75179b50485c" 2022-10-12T12:50:16.895Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="Drain complete for node 42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4" 2022-10-12T12:50:16.900Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef msg="Drain complete for node 7ae96fbf-6acf-58a6-a4fc-8217659265ef" 2022-10-12T12:50:16.900Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="Alloc "a4dff6b8-78f9-6ccc-fbe2-d3971c151b02" draining" 2022-10-12T12:50:16.900Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="Alloc "c6afa0a5-73c3-90d1-864d-2d0b7be7f3d7" draining" 2022-10-12T12:50:16.900Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="Alloc "01c4e20f-9a7c-a013-5607-e457ea1227c1" draining" 2022-10-12T12:50:16.900Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="Alloc "e4251841-deb1-f419-4a98-19475d5542f6" draining" 2022-10-12T12:50:16.902Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 msg="Drain complete for node 76f875dd-8abf-d363-cfc4-7ff548f864b7" 2022-10-12T12:50:16.902Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="Alloc "206a8b39-8a78-aaa3-c2d5-b7151285a1b6" draining" 2022-10-12T12:50:16.902Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="Alloc "5c71aeda-fb31-f34f-e058-9dc63c7cbd21" draining" 2022-10-12T12:50:17.686Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="Alloc "59c07429-9b5c-7202-eea1-9a316e8701b2" status running -> complete" 2022-10-12T12:50:17.693Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="Alloc "b83cc2c1-798d-26f6-dcad-805a6716a833" status running -> complete" 2022-10-12T12:50:17.695Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="Alloc "cb3ff2ea-c927-018d-c0b9-438d4be0754e" status running -> complete" 2022-10-12T12:50:17.695Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="Alloc "dba816ac-bfd0-031d-1733-b56abaedf76e" status running -> complete" 2022-10-12T12:50:17.698Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da msg="All allocations on node "75fed26e-3dd5-235f-b78f-82ec4d7f89da" have stopped" 2022-10-12T12:50:17.698Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=75fed26e-3dd5-235f-b78f-82ec4d7f89da 2022-10-12T12:50:17.699Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="Alloc "1f5d9d18-ff48-4857-7e31-50c8626387c4" status running -> complete" 2022-10-12T12:50:17.730Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b8c7d206-607b-7d51-945a-75179b50485c msg="Alloc "45536096-13f0-e55a-045d-ed80821ea2c8" status running -> complete" 2022-10-12T12:50:17.730Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b8c7d206-607b-7d51-945a-75179b50485c msg="Alloc "7179ac4a-31ac-fb5e-706b-f578343296f8" status running -> complete" 2022-10-12T12:50:17.730Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b8c7d206-607b-7d51-945a-75179b50485c msg="All allocations on node "b8c7d206-607b-7d51-945a-75179b50485c" have stopped" 2022-10-12T12:50:17.730Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=b8c7d206-607b-7d51-945a-75179b50485c 2022-10-12T12:50:17.826Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="Alloc "325f6c32-9798-d5c7-0c4e-2359c959bed6" status running -> complete" 2022-10-12T12:50:17.828Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="Alloc "54b6068f-2b37-f331-824b-536226329574" status running -> complete" 2022-10-12T12:50:17.828Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="Alloc "ad1d8768-6dfb-c384-0091-6d2128647c87" status running -> complete" 2022-10-12T12:50:17.828Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 msg="All allocations on node "42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4" have stopped" 2022-10-12T12:50:17.828Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=42b3f7f5-50f9-b2d6-86d4-21c9f6e7ddb4 2022-10-12T12:50:17.857Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="Alloc "66f7a070-ca0e-e273-fb94-a9cb381bb30b" draining" 2022-10-12T12:50:17.857Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="Alloc "d022cadd-c62c-02fc-3f92-213dff347b8b" draining" 2022-10-12T12:50:17.891Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="Alloc "036ca648-d933-36db-24d5-a70ab013a13b" draining" 2022-10-12T12:50:17.891Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="Alloc "09747377-d5d7-35d2-ccb1-0a3a440f3bc1" draining" 2022-10-12T12:50:17.893Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="Alloc "9af8e510-918e-527b-2c2b-e7896b77e6f2" status running -> complete" 2022-10-12T12:50:17.920Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="Alloc "01c4e20f-9a7c-a013-5607-e457ea1227c1" status running -> complete" 2022-10-12T12:50:17.920Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="Alloc "e4251841-deb1-f419-4a98-19475d5542f6" status running -> complete" 2022-10-12T12:50:17.920Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b779c284-396f-deff-3de9-02df51ccf024 msg="All allocations on node "b779c284-396f-deff-3de9-02df51ccf024" have stopped" 2022-10-12T12:50:17.920Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=b779c284-396f-deff-3de9-02df51ccf024 2022-10-12T12:50:17.921Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="Alloc "537b6bad-cdfe-75d7-f6dd-58b78fa3c623" status running -> complete" 2022-10-12T12:50:17.921Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="Alloc "7cfe67b2-d07f-3110-85a1-109dcad31fca" status running -> complete" 2022-10-12T12:50:17.921Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 msg="All allocations on node "2bf175c7-72a4-ce76-514e-cb6d1bbc5407" have stopped" 2022-10-12T12:50:17.921Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=2bf175c7-72a4-ce76-514e-cb6d1bbc5407 2022-10-12T12:50:17.936Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="Alloc "b09c0db2-5de2-921c-df6e-1477729e7f41" status running -> complete" 2022-10-12T12:50:17.937Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=47e6591c-1d9b-0231-df05-089262080217 msg="All allocations on node "47e6591c-1d9b-0231-df05-089262080217" have stopped" 2022-10-12T12:50:17.937Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=47e6591c-1d9b-0231-df05-089262080217 2022-10-12T12:50:17.939Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="Alloc "50c71b2b-f6b0-18ad-3869-56d5c712fa9b" status running -> complete" 2022-10-12T12:50:17.939Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=11c4df5a-606d-4222-7def-f512740f890a msg="All allocations on node "11c4df5a-606d-4222-7def-f512740f890a" have stopped" 2022-10-12T12:50:17.939Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=11c4df5a-606d-4222-7def-f512740f890a 2022-10-12T12:50:17.940Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="Alloc "73bd9591-1dcf-80a3-dfb5-d0cf53db7426" status running -> complete" 2022-10-12T12:50:17.940Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="Alloc "94b51b82-0354-ad16-27f4-22917f8a0c88" status running -> complete" 2022-10-12T12:50:17.940Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f msg="All allocations on node "1da27d6e-4649-6203-acd8-c9ac90ce998f" have stopped" 2022-10-12T12:50:17.940Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=1da27d6e-4649-6203-acd8-c9ac90ce998f 2022-10-12T12:50:17.999Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a2cab557-2a69-7ad6-554c-a7789d197484" status running -> complete" 2022-10-12T12:50:17.999Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="Alloc "a92f8ca4-95b2-dd53-e97d-da908028e5aa" status running -> complete" 2022-10-12T12:50:17.999Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=5ac68c24-bb30-44f2-5829-581b1623895a msg="All allocations on node "5ac68c24-bb30-44f2-5829-581b1623895a" have stopped" 2022-10-12T12:50:17.999Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=5ac68c24-bb30-44f2-5829-581b1623895a 2022-10-12T12:50:18.002Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="Alloc "66f7a070-ca0e-e273-fb94-a9cb381bb30b" status running -> complete" 2022-10-12T12:50:18.002Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="Alloc "d022cadd-c62c-02fc-3f92-213dff347b8b" status running -> complete" 2022-10-12T12:50:18.002Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=e79e037a-3200-89b0-6a87-b65688b52403 msg="All allocations on node "e79e037a-3200-89b0-6a87-b65688b52403" have stopped" 2022-10-12T12:50:18.002Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=e79e037a-3200-89b0-6a87-b65688b52403 2022-10-12T12:50:18.002Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="Alloc "30a2f5aa-9971-ba0a-22f9-39266fc6501a" status running -> complete" 2022-10-12T12:50:18.002Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 msg="All allocations on node "05ceef1a-7bcf-7e7a-8794-88ba16dde885" have stopped" 2022-10-12T12:50:18.002Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=05ceef1a-7bcf-7e7a-8794-88ba16dde885 2022-10-12T12:50:18.005Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="Alloc "d12838eb-23c6-bb07-5ca9-41867f91683e" status running -> complete" 2022-10-12T12:50:18.005Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a msg="All allocations on node "5e7e6984-7f6b-b383-d860-f6779cb6097a" have stopped" 2022-10-12T12:50:18.005Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=5e7e6984-7f6b-b383-d860-f6779cb6097a 2022-10-12T12:50:18.005Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="Alloc "d0bed79f-ffef-bafd-84c9-d3e7f98b7b04" status running -> complete" 2022-10-12T12:50:18.005Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="Alloc "dbdb1e18-ad4a-2f81-8fa6-d0801e1b4ff7" status running -> complete" 2022-10-12T12:50:18.007Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 msg="All allocations on node "4adf69e2-c74d-fce5-27ca-177af5441565" have stopped" 2022-10-12T12:50:18.007Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=4adf69e2-c74d-fce5-27ca-177af5441565 2022-10-12T12:50:18.057Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="Alloc "a4dff6b8-78f9-6ccc-fbe2-d3971c151b02" status running -> complete" 2022-10-12T12:50:18.057Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="Alloc "c6afa0a5-73c3-90d1-864d-2d0b7be7f3d7" status running -> complete" 2022-10-12T12:50:18.057Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 msg="All allocations on node "99157452-ba8f-2c4e-73c3-a297a89914a2" have stopped" 2022-10-12T12:50:18.057Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=99157452-ba8f-2c4e-73c3-a297a89914a2 2022-10-12T12:50:18.057Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="Alloc "036ca648-d933-36db-24d5-a70ab013a13b" status running -> complete" 2022-10-12T12:50:18.057Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="Alloc "09747377-d5d7-35d2-ccb1-0a3a440f3bc1" status running -> complete" 2022-10-12T12:50:18.057Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 msg="All allocations on node "61219e70-61ef-042e-79e3-0eaed861b7d9" have stopped" 2022-10-12T12:50:18.057Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=61219e70-61ef-042e-79e3-0eaed861b7d9 2022-10-12T12:50:18.451Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="Alloc "4800219a-0d4f-e82d-f277-7e6f07c986d4" status running -> complete" 2022-10-12T12:50:18.451Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 msg="All allocations on node "5b79ee9c-b72b-6391-6602-3e479aa505c7" have stopped" 2022-10-12T12:50:18.451Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=5b79ee9c-b72b-6391-6602-3e479aa505c7 2022-10-12T12:50:19.046Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "998888eb-4b75-584d-c918-7faba8946bbc" status running -> complete" 2022-10-12T12:50:19.056Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="Alloc "026a6893-b2bd-a6d7-38b5-88fd5d18cf01" status running -> complete" 2022-10-12T12:50:19.056Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="Alloc "f279acd7-b120-cedd-d432-18c23792ce67" status running -> complete" 2022-10-12T12:50:19.056Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc msg="All allocations on node "171aa5f2-7638-356e-9231-beaeb9d2ffcc" have stopped" 2022-10-12T12:50:19.056Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=171aa5f2-7638-356e-9231-beaeb9d2ffcc 2022-10-12T12:50:19.287Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="Alloc "74364d89-cbd2-4fea-980a-6f1601756d7b" status running -> complete" 2022-10-12T12:50:19.288Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="Alloc "75dc5a8a-ff13-42e3-9deb-aab39065ea4d" status running -> complete" 2022-10-12T12:50:19.288Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=725bd410-572a-1260-9a7a-1bb5a369afde msg="All allocations on node "725bd410-572a-1260-9a7a-1bb5a369afde" have stopped" 2022-10-12T12:50:19.288Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=725bd410-572a-1260-9a7a-1bb5a369afde 2022-10-12T12:50:19.348Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="Alloc "1ff4d72b-3f2e-0e3b-faf7-c7ba3a660807" status running -> complete" 2022-10-12T12:50:19.348Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d msg="All allocations on node "dac51581-ec47-d653-e577-25ad4b30aa1d" have stopped" 2022-10-12T12:50:19.348Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=dac51581-ec47-d653-e577-25ad4b30aa1d 2022-10-12T12:50:20.032Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="Alloc "a84f1e6b-58bd-87df-8d6a-278cdd422ce3" status running -> complete" 2022-10-12T12:50:20.035Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="Alloc "ec56cc6a-a4d8-4eed-ab97-20789fb8e7d0" status running -> complete" 2022-10-12T12:50:20.040Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 msg="All allocations on node "f883811d-4da7-0c55-b59d-49dbb6059c38" have stopped" 2022-10-12T12:50:20.043Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=f883811d-4da7-0c55-b59d-49dbb6059c38 2022-10-12T12:50:20.097Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=29b86c48-da66-2229-be80-361e607a99ce msg="Alloc "82a1c070-65b5-71e1-358b-3767f89ca46e" status running -> complete" 2022-10-12T12:50:20.100Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=29b86c48-da66-2229-be80-361e607a99ce msg="Alloc "9e9121ac-fc99-0f21-1034-6d1fcb6c7d90" status running -> complete" 2022-10-12T12:50:20.106Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=29b86c48-da66-2229-be80-361e607a99ce msg="All allocations on node "29b86c48-da66-2229-be80-361e607a99ce" have stopped" 2022-10-12T12:50:20.111Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=29b86c48-da66-2229-be80-361e607a99ce 2022-10-12T12:50:20.741Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="Alloc "3209981e-3777-d11b-6c3c-3c498e6dde27" status running -> complete" 2022-10-12T12:50:20.741Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 msg="All allocations on node "aa989f2d-52ba-86f5-954f-947b2a59bb06" have stopped" 2022-10-12T12:50:20.741Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=aa989f2d-52ba-86f5-954f-947b2a59bb06 2022-10-12T12:50:23.522Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a msg="Alloc "5cab3637-6449-d195-20b2-21abef7e1401" status running -> complete" 2022-10-12T12:50:23.522Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a msg="Alloc "794f1278-bc3b-bcb6-9c66-ff25755d684b" status running -> complete" 2022-10-12T12:50:23.522Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a msg="All allocations on node "c9acf6e1-12ef-6236-269b-e662fc9b1a3a" have stopped" 2022-10-12T12:50:23.780Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="Alloc "18f15905-8ddc-c718-98d9-3ceef4f177ce" status running -> complete" 2022-10-12T12:50:23.780Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="Alloc "54200208-86b7-7832-b758-7d67a21506a2" status running -> complete" 2022-10-12T12:50:23.780Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 msg="All allocations on node "85afe4c6-a56e-a6af-3fee-a9c9a7980f44" have stopped" 2022-10-12T12:50:23.780Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=85afe4c6-a56e-a6af-3fee-a9c9a7980f44 2022-10-12T12:50:23.781Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef msg="Alloc "37360026-c618-b5c3-4f56-db0be138a908" status running -> complete" 2022-10-12T12:50:23.781Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef msg="Alloc "430f3843-4537-b630-60b7-722605603f69" status running -> complete" 2022-10-12T12:50:23.781Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef msg="All allocations on node "7ae96fbf-6acf-58a6-a4fc-8217659265ef" have stopped" 2022-10-12T12:50:23.781Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=7ae96fbf-6acf-58a6-a4fc-8217659265ef 2022-10-12T12:50:23.788Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e msg="Alloc "54d699a6-3b93-b902-df11-9959346c23eb" status running -> complete" 2022-10-12T12:50:23.788Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e msg="Alloc "650a1490-f6cf-b413-96b0-40d334f52edb" status running -> complete" 2022-10-12T12:50:23.788Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e msg="All allocations on node "4ea7bfca-b9bf-67b4-1183-a19758d9626e" have stopped" 2022-10-12T12:50:23.788Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=4ea7bfca-b9bf-67b4-1183-a19758d9626e 2022-10-12T12:50:23.797Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd msg="Alloc "259f1ad3-ea89-dd3e-52c2-1653125c7ad8" status running -> complete" 2022-10-12T12:50:23.797Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd msg="Alloc "7d506de4-6999-36a6-519d-fdc9e1dc91aa" status running -> complete" 2022-10-12T12:50:23.797Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd msg="All allocations on node "b494a874-6a2d-bb07-5cfe-4c9fda8edefd" have stopped" 2022-10-12T12:50:23.797Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=b494a874-6a2d-bb07-5cfe-4c9fda8edefd 2022-10-12T12:50:24.038Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="Alloc "206a8b39-8a78-aaa3-c2d5-b7151285a1b6" status running -> complete" 2022-10-12T12:50:24.038Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="Alloc "5c71aeda-fb31-f34f-e058-9dc63c7cbd21" status running -> complete" 2022-10-12T12:50:24.038Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 msg="All allocations on node "bb7294e1-d61f-9582-6c49-6bbbdd4f86e4" have stopped" 2022-10-12T12:50:24.038Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=bb7294e1-d61f-9582-6c49-6bbbdd4f86e4 2022-10-12T12:50:24.040Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="Alloc "0d380697-e251-1c8d-d3b4-f2a5a6cfb6e2" status running -> complete" 2022-10-12T12:50:24.040Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="Alloc "a8268eb0-5908-532d-29d2-1b89ed06598e" status running -> complete" 2022-10-12T12:50:24.040Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 msg="All allocations on node "e5ce4421-98ca-71b8-4395-2debd69a5424" have stopped" 2022-10-12T12:50:24.040Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=e5ce4421-98ca-71b8-4395-2debd69a5424 2022-10-12T12:50:24.044Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="Alloc "3c2a3f24-e3f0-d722-ca2b-dc9f782ba876" status running -> complete" 2022-10-12T12:50:24.044Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="Alloc "54142198-2062-c051-d160-25758db284c4" status running -> complete" 2022-10-12T12:50:24.044Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=b7b16595-e281-4f34-7189-c653c447f9ea msg="All allocations on node "b7b16595-e281-4f34-7189-c653c447f9ea" have stopped" 2022-10-12T12:50:24.044Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=b7b16595-e281-4f34-7189-c653c447f9ea 2022-10-12T12:50:32.049Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 msg="Alloc "1775f750-4a5c-b2e8-c8f4-8a16044d0b0c" status running -> complete" 2022-10-12T12:50:32.050Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 msg="Alloc "f8c8ae2c-469b-2c75-d59e-5c862b6b7bf3" status running -> complete" 2022-10-12T12:50:32.079Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 msg="All allocations on node "7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3" have stopped" 2022-10-12T12:50:32.086Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=7cede70b-e8b8-d0dc-cc07-8cf514b3f0b3 2022-10-12T12:50:32.160Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7c995302-018d-fc7f-9b66-8cb242122581 msg="Alloc "31ceee79-2c0b-000e-09c0-6b9b8d6ad399" status running -> complete" 2022-10-12T12:50:32.160Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=7c995302-018d-fc7f-9b66-8cb242122581 msg="Alloc "f19ba0a3-d0e5-9bb3-9640-4af5b3a61806" status running -> complete" 2022-10-12T12:50:32.160Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=7c995302-018d-fc7f-9b66-8cb242122581 msg="All allocations on node "7c995302-018d-fc7f-9b66-8cb242122581" have stopped" 2022-10-12T12:50:32.160Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=7c995302-018d-fc7f-9b66-8cb242122581 2022-10-12T12:50:32.235Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="Alloc "1aedab96-7ceb-b5bd-5e15-7c80f99eb848" status running -> complete" 2022-10-12T12:50:32.235Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="Alloc "9b483e18-84ed-bfab-c971-64c4133333f9" status running -> complete" 2022-10-12T12:50:32.235Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 msg="All allocations on node "18a728a6-26dc-f161-bab4-0fce3e63fba1" have stopped" 2022-10-12T12:50:32.235Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=18a728a6-26dc-f161-bab4-0fce3e63fba1 2022-10-12T12:50:32.318Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a msg="Alloc "625e12d3-fe7d-8bf8-70e3-6a26d568f402" status running -> complete" 2022-10-12T12:50:32.318Z [DEBUG] internal_plugin.aws-asg: received node drain message: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a msg="Alloc "97527da1-06ef-6d4e-cc74-b550a7fc8b8d" status running -> complete" 2022-10-12T12:50:32.318Z [INFO] internal_plugin.aws-asg: received node drain message: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a msg="All allocations on node "728f43fa-8d77-7878-5bd9-7b66290dce5a" have stopped" 2022-10-12T12:50:32.320Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=728f43fa-8d77-7878-5bd9-7b66290dce5a 2022-10-12T12:50:37.626Z [DEBUG] internal_plugin.nomad-target: triggering run of handler garbage collection 2022-10-12T12:50:41.251Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=e57aecde-ea9e-2048-5919-ee7659d0afea policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:50:41.251Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=e57aecde-ea9e-2048-5919-ee7659d0afea policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:50:41.252Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=e57aecde-ea9e-2048-5919-ee7659d0afea policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=03a0e680-8e86-bd14-14d3-7d2e8289bc4f 2022-10-12T12:50:41.252Z [DEBUG] policy_eval.worker: received policy for evaluation: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:50:41.252Z [DEBUG] policy_eval.worker: fetching current count: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:50:41.252Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:50:41.252Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:50:41.252Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:49:41.252458744 +0000 UTC m=+29883.634960933 2022-10-12 12:50:41.252458744 +0000 UTC m=+29943.634960933}" 2022-10-12T12:50:41.543Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:50:41.543Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:50:41.543Z [DEBUG] policy_eval.worker: no checks need to be executed: id=4571cfcd-8ab7-9632-4a08-19b8967f2d97 policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:50:41.544Z [DEBUG] policy_eval.broker: ack eval: eval_id=e57aecde-ea9e-2048-5919-ee7659d0afea token=03a0e680-8e86-bd14-14d3-7d2e8289bc4f eval_id=e57aecde-ea9e-2048-5919-ee7659d0afea token=03a0e680-8e86-bd14-14d3-7d2e8289bc4f 2022-10-12T12:50:41.544Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:50:41.544Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:50:41.544Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:50:41.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=114fc614-3053-8279-079c-5465b1a320c7 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:50:41.653Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=114fc614-3053-8279-079c-5465b1a320c7 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:50:41.653Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=114fc614-3053-8279-079c-5465b1a320c7 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=ed472ca2-3ba4-0d00-15df-755510cf5ccd 2022-10-12T12:50:41.653Z [DEBUG] policy_eval.worker: received policy for evaluation: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:50:41.653Z [DEBUG] policy_eval.worker: fetching current count: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:50:41.653Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:50:41.653Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:50:41.653Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:49:41.653636356 +0000 UTC m=+29884.036138545 2022-10-12 12:50:41.653636356 +0000 UTC m=+29944.036138545}" 2022-10-12T12:50:41.971Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=99 2022-10-12T12:50:41.971Z [INFO] policy_eval.worker: scaling target: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target from=99 to=100 reason="capped count from 68895 to 100 to stay within limits" meta="map[nomad_autoscaler.count.capped:true nomad_autoscaler.count.original:68895 nomad_autoscaler.reason_history:[scaling up because metric is 68895] nomad_policy_id:787ac75b-962e-1a9e-d64c-06d485acb008]" 2022-10-12T12:50:41.982Z [INFO] policy_eval.worker: scaling action skipped: id=65ba71cc-d46e-dadd-8706-268ee1697c26 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target reason="skipping scaling group workers/workers due to active deployment" 2022-10-12T12:50:41.983Z [DEBUG] policy_eval.broker: ack eval: eval_id=114fc614-3053-8279-079c-5465b1a320c7 token=ed472ca2-3ba4-0d00-15df-755510cf5ccd eval_id=114fc614-3053-8279-079c-5465b1a320c7 token=ed472ca2-3ba4-0d00-15df-755510cf5ccd 2022-10-12T12:50:41.984Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:50:41.984Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:50:41.985Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:50:45.219Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=c9acf6e1-12ef-6236-269b-e662fc9b1a3a 2022-10-12T12:50:46.077Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=71d147c8-2031-02b5-feea-daee12267e19 2022-10-12T12:50:46.641Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=76f875dd-8abf-d363-cfc4-7ff548f864b7 2022-10-12T12:50:46.875Z [DEBUG] internal_plugin.aws-asg: node drain complete: node_id=4592c999-9240-e62d-edd3-5c3faa37129e 2022-10-12T12:50:46.876Z [ERROR] policy_eval.worker: failed to evaluate policy: eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee eval_token=5261ed5e-4f6d-2f87-ed53-d03d66b2b87c id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster error="failed to scale target: failed to perform scaling action: failed to perform pre-scale Nomad scale in tasks: context done while monitoring node drain: received error while draining node: Error monitoring node: Get "http://10.10.0.85:4646/v1/node/c9acf6e1-12ef-6236-269b-e662fc9b1a3a?index=12918&namespace=dev®ion=global&stale=": dial tcp 10.10.0.85:4646: i/o timeout, context done while monitoring node drain: received error while draining node: Error monitoring allocations: Get "http://10.10.0.85:4646/v1/node/71d147c8-2031-02b5-feea-daee12267e19/allocations?index=12908&namespace=dev®ion=global&stale=": dial tcp 10.10.0.85:4646: i/o timeout, context done while monitoring node drain: received error while draining node: Error monitoring allocations: Get "http://10.10.0.85:4646/v1/node/76f875dd-8abf-d363-cfc4-7ff548f864b7/allocations?index=13089&namespace=dev®ion=global&stale=": dial tcp 10.10.0.85:4646: i/o timeout, context done while monitoring node drain: received error while draining node: Error monitoring allocations: Get "http://10.10.0.85:4646/v1/node/4592c999-9240-e62d-edd3-5c3faa37129e/allocations?index=13089&namespace=dev®ion=global&stale=": dial tcp 10.10.0.85:4646: i/o timeout" 2022-10-12T12:50:46.876Z [DEBUG] policy_eval.broker: nack eval: eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee token=5261ed5e-4f6d-2f87-ed53-d03d66b2b87c 2022-10-12T12:50:46.876Z [WARN] policy_eval.worker: failed to NACK policy evaluation: eval_id=7777755e-7e27-09f8-80c0-dd017c3b4dee eval_token=5261ed5e-4f6d-2f87-ed53-d03d66b2b87c id=61f292fc-760c-e172-3095-e196982d4e84 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster error="evaluation ID not found" 2022-10-12T12:50:46.876Z [DEBUG] policy_eval.broker: dequeue eval: queue=cluster 2022-10-12T12:50:46.877Z [DEBUG] policy_eval.broker: waiting for eval: queue=cluster 2022-10-12T12:51:11.251Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=3f0ca4b9-c42c-3bea-5158-28a06a4f39ff policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:51:11.251Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=3f0ca4b9-c42c-3bea-5158-28a06a4f39ff policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal token="" 2022-10-12T12:51:11.251Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=3f0ca4b9-c42c-3bea-5158-28a06a4f39ff policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 token=a19a321c-42a5-343f-1593-80717b91c86b 2022-10-12T12:51:11.252Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:51:11.252Z [DEBUG] policy_eval.worker: fetching current count: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:51:11.252Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:51:11.252Z [DEBUG] policy_eval.worker.check_handler: querying source: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) source=prometheus 2022-10-12T12:51:11.252Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query=sum(traefik_entrypoint_open_connections)/scalar(nomad_nomad_job_summary_running{task_group="webapp"}) range="{2022-10-12 12:50:11.252808672 +0000 UTC m=+29913.635310861 2022-10-12 12:51:11.252808672 +0000 UTC m=+29973.635310861}" 2022-10-12T12:51:11.534Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target count=1 2022-10-12T12:51:11.534Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=webapp_scaling id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal source=prometheus strategy=target-value target=nomad-target 2022-10-12T12:51:11.534Z [DEBUG] policy_eval.worker: no checks need to be executed: id=5b94f3fa-8f77-8ecc-d119-30cbbe724e5f policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 queue=horizontal target=nomad-target 2022-10-12T12:51:11.534Z [DEBUG] policy_eval.broker: ack eval: eval_id=3f0ca4b9-c42c-3bea-5158-28a06a4f39ff token=a19a321c-42a5-343f-1593-80717b91c86b eval_id=3f0ca4b9-c42c-3bea-5158-28a06a4f39ff token=a19a321c-42a5-343f-1593-80717b91c86b 2022-10-12T12:51:11.534Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=f2cceef8-5250-db1d-900d-d72d25f61db1 2022-10-12T12:51:11.534Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:51:11.534Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:51:11.653Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=896e1dd8-3673-a59c-e962-dfc394b15e46 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:51:11.653Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=896e1dd8-3673-a59c-e962-dfc394b15e46 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal token="" 2022-10-12T12:51:11.653Z [DEBUG] policy_eval.broker: eval dequeued: queue=horizontal eval_id=896e1dd8-3673-a59c-e962-dfc394b15e46 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 token=ca0b485a-63a6-d00f-56fa-abfb6d73b906 2022-10-12T12:51:11.653Z [DEBUG] policy_eval.worker: received policy for evaluation: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:51:11.653Z [DEBUG] policy_eval.worker: fetching current count: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target 2022-10-12T12:51:11.653Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target 2022-10-12T12:51:11.653Z [DEBUG] policy_eval.worker.check_handler: querying source: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" source=prometheus 2022-10-12T12:51:11.653Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(aws_sqs_approximate_number_of_messages_not_visible_average) + sum(aws_sqs_approximate_number_of_messages_visible_average) + sum(aws_sqs_approximate_number_of_messages_delayed_average))/10)+2" range="{2022-10-12 12:50:11.653974724 +0000 UTC m=+29914.036476903 2022-10-12 12:51:11.653974724 +0000 UTC m=+29974.036476903}" 2022-10-12T12:51:11.940Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=worker_scaling id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal source=prometheus strategy=pass-through target=nomad-target count=99 2022-10-12T12:51:11.940Z [INFO] policy_eval.worker: scaling target: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target from=99 to=100 reason="capped count from 74214 to 100 to stay within limits" meta="map[nomad_autoscaler.count.capped:true nomad_autoscaler.count.original:74214 nomad_autoscaler.reason_history:[scaling up because metric is 74214] nomad_policy_id:787ac75b-962e-1a9e-d64c-06d485acb008]" 2022-10-12T12:51:11.943Z [INFO] policy_eval.worker: scaling action skipped: id=5e1be8a7-6617-704c-1a6e-be345734d265 policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 queue=horizontal target=nomad-target reason="skipping scaling group workers/workers due to active deployment" 2022-10-12T12:51:11.943Z [DEBUG] policy_eval.broker: ack eval: eval_id=896e1dd8-3673-a59c-e962-dfc394b15e46 token=ca0b485a-63a6-d00f-56fa-abfb6d73b906 eval_id=896e1dd8-3673-a59c-e962-dfc394b15e46 token=ca0b485a-63a6-d00f-56fa-abfb6d73b906 2022-10-12T12:51:11.943Z [DEBUG] policy_eval.broker: eval ack'd: policy_id=787ac75b-962e-1a9e-d64c-06d485acb008 2022-10-12T12:51:11.943Z [DEBUG] policy_eval.broker: dequeue eval: queue=horizontal 2022-10-12T12:51:11.943Z [DEBUG] policy_eval.broker: waiting for eval: queue=horizontal 2022-10-12T12:51:15.121Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=cd786500-75c8-3c80-b0a0-7bc3777be301 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster token="" 2022-10-12T12:51:15.122Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=cd786500-75c8-3c80-b0a0-7bc3777be301 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster token="" 2022-10-12T12:51:15.122Z [DEBUG] policy_eval.broker: eval dequeued: queue=cluster eval_id=cd786500-75c8-3c80-b0a0-7bc3777be301 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 token=7458fc6d-10b5-35ab-c163-9e7c6a31e5e6 2022-10-12T12:51:15.122Z [DEBUG] policy_eval.worker: received policy for evaluation: id=e230dd28-ea02-01f1-5f43-133e033b2ef7 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg 2022-10-12T12:51:15.123Z [DEBUG] policy_eval.worker: fetching current count: id=e230dd28-ea02-01f1-5f43-133e033b2ef7 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg 2022-10-12T12:51:15.928Z [DEBUG] policy_eval.worker.check_handler: received policy check for evaluation: check=cpu_allocated_percentage id=e230dd28-ea02-01f1-5f43-133e033b2ef7 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster source=prometheus strategy=pass-through target=aws-asg 2022-10-12T12:51:15.928Z [DEBUG] policy_eval.worker.check_handler: querying source: check=cpu_allocated_percentage id=e230dd28-ea02-01f1-5f43-133e033b2ef7 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster source=prometheus strategy=pass-through target=aws-asg query="ceil((sum(nomad_client_allocs_cpu_allocated{namespace="dev"}) + sum(nomad_nomad_blocked_evals_job_cpu{namespace="dev"} OR on() vector(0)))/6000)" source=prometheus 2022-10-12T12:51:15.929Z [DEBUG] internal_plugin.prometheus: querying Prometheus: query="ceil((sum(nomad_client_allocs_cpu_allocated{namespace="dev"}) + sum(nomad_nomad_blocked_evals_job_cpu{namespace="dev"} OR on() vector(0)))/6000)" range="{2022-10-12 12:50:15.929526881 +0000 UTC m=+29918.312029070 2022-10-12 12:51:15.929526881 +0000 UTC m=+29978.312029070}" 2022-10-12T12:51:15.996Z [DEBUG] policy_eval.broker: enqueue eval: eval_id=05a609ec-1219-bda3-6820-4dde1f6d13ba policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster token="" 2022-10-12T12:51:15.998Z [DEBUG] policy_eval.broker: policy already enqueued: eval_id=05a609ec-1219-bda3-6820-4dde1f6d13ba policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster token="" 2022-10-12T12:51:15.998Z [DEBUG] policy_eval.broker: eval enqueued: eval_id=05a609ec-1219-bda3-6820-4dde1f6d13ba policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster token="" 2022-10-12T12:51:15.998Z [DEBUG] policy_eval.broker: eval dequeued: queue=cluster eval_id=05a609ec-1219-bda3-6820-4dde1f6d13ba policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 token=20c1e760-6dc5-6017-3580-702525f84f48 2022-10-12T12:51:15.998Z [DEBUG] policy_eval.worker: received policy for evaluation: id=149e91f2-4534-e0c0-934d-c38cd0ddd838 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg 2022-10-12T12:51:15.999Z [DEBUG] policy_eval.worker: fetching current count: id=149e91f2-4534-e0c0-934d-c38cd0ddd838 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg 2022-10-12T12:51:16.211Z [DEBUG] policy_eval.worker.check_handler: calculating new count: check=cpu_allocated_percentage id=e230dd28-ea02-01f1-5f43-133e033b2ef7 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster source=prometheus strategy=pass-through target=aws-asg count=50 2022-10-12T12:51:16.212Z [DEBUG] policy_eval.worker.check_handler: nothing to do: check=cpu_allocated_percentage id=e230dd28-ea02-01f1-5f43-133e033b2ef7 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster source=prometheus strategy=pass-through target=aws-asg from=50 to=50 2022-10-12T12:51:16.212Z [DEBUG] policy_eval.worker: no checks need to be executed: id=e230dd28-ea02-01f1-5f43-133e033b2ef7 policy_id=c4d837ab-f45c-303a-ea9a-8309a1b02590 queue=cluster target=aws-asg 2022-10-12T12:51:16.212Z [DEBUG] policy_eval.broker: ack eval: eval_id=cd786500-75c8-3c80-b0a0-7bc3777be301 token=7458fc6d-10b5-35ab-c163-9e7c6a31e5e6 eval_id=cd786500-75c8-3c80-b0a0-7bc3777be301 token=7458fc6d-10b5-35ab-c163-9e7c6a31e5e6