Waypoint and configuration

[trondhindenes]

in my opinion, one of the most important task of any deployment tool is to allow developers to deal with configuring their apps in a structured (and humane) way. It needs to be possible to express this via statements such as:

• always set FOO=BAR when deploying to any environment, unless something else is set
• set the value of BAZ to the name of the deployed environment
• for all non-production environment, set FAZ=ABC. For all production environment, set FAZ=DEF
• set MEK to the value of this (other system lookup): “/myapp/$ENVIRONMENT/MEK”

I was very surprised to see that Waypoint doesnt offer anything more than an arbitrary “config setter”.

I was thinking “wow we can use Waypoint today to solve some real problems!” until I noticed the lack of a robust config story.

It would be very interesting to get the maintainer’s thoughts on any future extension/improvements in this space.

[nic]

Hi @trondhindenes

Waypoint is only in the initial release, but we do have plans to add deeper integration for configuration management and secrets as the project develops.

We talk about this a little on the Roadmap section of the docs https://www.waypointproject.io/docs/roadmap but essential expect to see integration with Consul, Vault, and other config and secrets solutions.

The intention behind releasing Waypoint in the early stages is to allow the community to help shape the direction of the project. The plugin ecosystem allows for some extension points already but this is just the beginning.

Thank you for the feedback, your use cases are really useful as we look into future feature development. It is difficult to commit right now to what the next features will be, stability is always a primary concern but production workflows are super high on our priorities.

Nic

[trondhindenes]

nice, thanks! I quickly looked at the “extensibility” section but didn’t see anything there - I guess that’s what kindof bummed me. I totally realize it’s very early still, and Waypoint looks super-interesting. It would be awesome if a config api was prioritized (an api or extension point is just as important imho than ready-made integrations with consul, vault etc.)

I think config injection is one of those things that different orgs do very differently, so there needs to be flexibility. We don’t use Consul or Vault for example, so if Waypoint’s config implementation relied on those products it wouldn’t be useful to us.

But I’m rambling. Thanks for the roadmap link, I’ll check that out.

[mitchellh]

@trondhindenes

We don’t use Consul or Vault for example

What do you use?

We do plan on making this pluggable because we don’t expect folks to all use Consul/Vault. We actually expect most don’t… so yeah it will be a pluggable system. We haven’t officially decided what will be in Waypoint 0.2 yet but I’m personally very excited by the config potential of Waypoint as well.

Regardless, more and more of this stuff is coming in soon.

[trondhindenes]

for non-secrets we use a “cascading” set of config files - one for defaults and one for optional environment-specific overrides. Secrets in AWS SSM can be referenced by name in these config files, and get resolved at deploy-time.

Glad to learn that this area is something you consider important and have plans for, looking forward to see how Waypoint takes shape in the coming months and years.

[sachin-ngpws]

So in the nomad plugin, I could not find variables for more advanced nomad config’s like the artefacts, templating or adding a service definition. Is there a way to do it now, like through a nomad file or some other configs or is it going to come in one of the future releases?

Similarly in docker I’d like to specify the volumes and networks in my use cases.

[jbayer]

Sachin, you could take a look at the Kubernetes exec plugin example which uses the exec plugin. The example demonstrates an approach that you could use to templated a shell script that do whatever you want. E.g. something special in Nomad or setup Docker networks and volumes until the Docker plugin gets more full featured in the platform capabilities. In your case, would you expect to be using networks and volumes that already exist, or would you expect Waypoint to create them during waypoint deploy.

Do you have an example Job file that shows what you’d want to be able to do in Nomad?

[sachin-ngpws]

Waypoint and configuration

Sachin, you could take a look at the Kubernetes exec plugin example which uses the exec plugin. The example demonstrates an approach that you could use to templated a shell script that do whatever you want. E.g. something special in Nomad or setup Docker networks and volumes until the Docker plugin gets more full featured in the platform capabilities. In your case, would you expect to be using networks and volumes that already exist, or would you expect Waypoint to create them during waypoint de…

Kube exec example This was helpful thank you. In my case, I was looking at using existing networks and volumes.

Nomad File with elements like

service {
        port = "http"
        name = "<service>"
      }
      artifact {
         source = "link.to/some.tar.gz" 
      }
      template {
            data = <<EOH
{{ with secret "path" "common_name=<service>.service.consul"  }}
{{- .Data.certificate -}}
{{ end }}
      }