We have deployed vault with a consul backend to AWS ec2 instances. All works great and we have got a mechanism to take consul snapshots and ship them to S3.
I am currently trying to test a restore - which completes - but alas the data seems off. When I try and access secrets stored in kv2 that are from the restore I get the following:
Ember Data Request GET /v1/secret/data/test2?version=1 returned a 500 Payload (Empty Content-Type) [object Object]
1 error occurred: * could not find version data
Tried different methods of redploying consul and restoring, redeploying everything and restoring but always end up at the same point. Been driving me mad and we cant really look to move forward without some form of recovery method. Certinaly cannot demonstrate its usefulness as a system and try and justify that expensive enterprise option…
Hi @timalexander-inv!
Sad to hear that you had such a bad experience with Consul and Vault!
Would you mind to share more information regarding the commands you’ve used to take the Consul snapshot and to restore it? It would be also super helpful to learn more about your infrastructure e.g. how many Consul instances do you have, distribution of Vault instances etc.
Additionally, I recommend to check the Consul logs from the Consul instance where the snapshot was taken from. Maybe the instance is in a bad state or has problems to synchronize with the other Consul servers?
I also want to highlight that if you are an enterprise customer, I recommend to open a new ticket via our enterprise support portal. Reporting through our enterprise support portal ensures that we can comply with contractually obligated SLAs around customer support requests, and that a support engineer or technical account manager can track the issue from report to resolution.
Cheers,
Michel
Thanks for the reply. We did some further testing and thought we had zoned in on the issue being with the method of secret creation. This led us to tear down the cluster and redploy. Alas it seems our idea, that it was gui based kv2 secrets with a single version were the issue, did not pan out and we now cannot replicate. Been frustrating but we can only guess that in our rush to test things we have bodged a command. It is probably more likely to do with the state of the consul back end as you suggest. Either way have been unable to replicate and am slowly getting through the backlog of things before we use it in prod.
Thanks