As I investigate Waypoint, it would be great to know what is stored in it’s database, and how we’re expected to handle HA deployments on ephemeral infrastructure.
For reference, we typically deploy everything to Nomad and have infrastructure that regularly self-terminates, not to mention that our hosting provider (AWS) may destroy infrastructure at any point in time. Storing data on an EBS volume also still means we would have a Waypoint outage if an AZ disappeared, if the EBS volume dies, or if we are upgrading Waypoint.
This is a good question and based on this we plan on augmenting our docs to better explain this. The database stores:
- Operation history (builds, deploys, etc.)
- Metadata about projects/apps (their name, what apps are in what projects, etc.)
- Application configuration (
waypoint config set)
- Server configuration (
waypoint server config-set)
- Job queue (remote runners)
And importantly the database does NOT store:
- Application logs (these are held in a circular buffer in-memory)
- This is the only sizable thing today so the databases today even for very active Waypoint servers are quite small by filesize.
We don’t have an HA answer today. We architected the internal package structure in a way where we could add that later based on the style we used with other HashiCorp products but we didn’t ship with an HA solution up front. Today, the best answer is “get the server back up’” and we also made backing up the data easy with an API and CLI for snapshotting.
We designed the entrypoint and runners to be resilient to the server being down. The server going down fails everything into a “fail static” mode: existing running workloads will continue to work fine, new workloads will come up but may not be able to query their config, but new operations (build/deploy/etc.) can’t be started.