Consul snapshot restore backup from a different VM (consul_version: 1.7.1)

I am migrating data from one ununtu server to another. I have installed consul on the new server - and I am looking to copy all the data from the old server to the new server.
I did the following-

  1. On the old VM: consul snapshot save test.snap
  2. SCP test.snap from old server to the new one
  3. One the new VM: consul snapshot restore test.snap
    When I do step 3:
    Here’s the error that I get-
    Error restoring snapshot: Put http://127.0.0.1:8500/v1/snapshot: EOF

Can you tell me if I’d doing something wrong?

Here are my syslogs:
Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent: Node info in sync

Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=244.47µs

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.173Z [DEBUG] agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent: Node info in sync

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Node info in sync

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=946.615µs

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=946.615µs

Sep 6 21:22:01 gsoc-hashicorp CRON[56486]: (root) CMD (/opt/system_load_json.py > /dev/console)

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:02.675Z [ERROR] agent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:54828 error=“Permission denied”

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:02.675Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:54828 latency=128.693µs

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: agent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:54828 error=“Permission denied”

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:54828 latency=128.693µs

Sep 6 21:22:03 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:03.360Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/renew/e6f31960-57b2-4026-1462-40edb582e683 from=127.0.0.1:54814 latency=137.589µs

Sep 6 21:22:03 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/session/renew/e6f31960-57b2-4026-1462-40edb582e683 from=127.0.0.1:54814 latency=137.589µs

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Node info in sync

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=303.151µs

Look in the logs of the Consul server you are restoring to, for a better error message.

Here are my syslogs -
Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent: Node info in sync

Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:21:56 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=244.47µs

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.173Z [DEBUG] agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent: Node info in sync

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Node info in sync

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:01.174Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=946.615µs

Sep 6 21:22:01 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=946.615µs

Sep 6 21:22:01 gsoc-hashicorp CRON[56486]: (root) CMD (/opt/system_load_json.py > /dev/console)

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:02.675Z [ERROR] agent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:54828 error=“Permission denied”

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:02.675Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:54828 latency=128.693µs

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: agent.http: Request error: method=PUT url=/v1/snapshot from=127.0.0.1:54828 error=“Permission denied”

Sep 6 21:22:02 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/snapshot from=127.0.0.1:54828 latency=128.693µs

Sep 6 21:22:03 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:03.360Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/session/renew/e6f31960-57b2-4026-1462-40edb582e683 from=127.0.0.1:54814 latency=137.589µs

Sep 6 21:22:03 gsoc-hashicorp consul[54513]: agent.http: Request finished: method=PUT url=/v1/session/renew/e6f31960-57b2-4026-1462-40edb582e683 from=127.0.0.1:54814 latency=137.589µs

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: agent: Check status updated: check=vault:127.0.0.1:8200:vault-sealed-check status=passing

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Node info in sync

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Service in sync: service=vault:127.0.0.1:8200

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent: Check in sync: check=vault:127.0.0.1:8200:vault-sealed-check

Sep 6 21:22:05 gsoc-hashicorp consul[54513]: 2022-09-06T21:22:05.780Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/agent/check/pass/vault:127.0.0.1:8200:vault-sealed-check?note=Vault+Unsealed from=127.0.0.1:54816 latency=303.151µs

I just attached my syslogs - unfortunately I’m not able to figure out much from it

Seems fairly relevant and worth investigating further