Deprecating projects (or how I learned to love apps)

Since the initial version of Waypoint, the product has supported the ability to configure multiple apps within a single waypoint.hcl file. Starting with Waypoint 0.9, this functionality is deprecated and will be removed in 0.10. The vast majority of users are not using this functionality and it served mostly as a source of confusion for users. For users who are using a monorepo pattern, we plan to add better workflows for you.

With a waypoint.hcl now focused on the configuration of a single application, the concept of a project within the Waypoint data model will be removed, moving applications to the top level. This is already how users talk about using Waypoint and we are confident that it will improve the overall understanding of Waypoint as well.

We’d love to hear your comments and concerns about this change below. We’re looking to incorporate the feedback into future planning work, such as ways to improve the monorepo pattern usage.

7 Likes

Hey @evanphx, we actually use the multi-app feature quite a bit with nomad-jobspec deploys. Is there a path forward for this?

For instance, we have one waypoint “app” config that manages the web app and another “app” config that launches a nomad “periodic” jobspec.

We also have another use-case that has many apps that get launched because the code is not multi-tenant. So, in this particular case, ‘n’ number of clients means ‘n’ number of apps.

I think the later case we could refactor into the nomad jobspec, but the first case needs two different jobspecs because “periodic” is defined at the job level.

Hi @dwanek!

Thanks for reaching out on this! Any chance you could share the waypoint.hcl files and maybe a short write up on your workflow?

We’ll absolutely figure out a way forward for ya.

Sure, I’ve done some name changing in the files, but the content is pretty much the same.

The first one I’m attaching is the one I mentioned with ‘n’ number of client apps. Upon waypoint up, the first app definition builds the image and pushes it to ECR. Subsequent app definitions use that same image via docker-pull.

waypoint-example-with-n-apps.hcl.txt (4.3 KB)

The second has a web app and two Nomad periodic job apps that get deployed. It also makes heavy use of dynamic stanzas that allow some differing configuration for each application. The build is similar to the one above where the first app builds the image and the other two use docker-pull to fetch it.

waypoint-example-with-periodic-jobs.hcl.txt (9.0 KB)

Thanks for any input. :purple_heart:

2 Likes

Similar to @dwanek, I was investigating migrating our Nomad deployment tooling to waypoint. Waypoint touts itself as having it’s projects map 1:1 with a VCS repository. In that case, it’s not clear to me how I would deploy multiple processes.

For a hypothetical python codebase with two processes:

  • web: python web.py
  • worker: python worker.py

How would we specify that in waypoint now, given the deprecation of projects? The two processes are functionally the same app, just running two different processes. To me, it makes sense to at least have a project-level abstraction that ties both together.

6 Likes

We have been using Waypoint for our monorepo project, with a top-level ./waypoint.hcl that’s been working well for us. Having just tried to separate this into a waypoint.hcl per app (./app1/waypoint.hcl, ./app2/waypoint.hcl, etc.), a few comments:

  1. Seems we need to change directory to deploy multiple apps. Would be nice to have a replacement for “waypoint up -app app1” like “waypoint up -dir ./app1” that we can run from the VCS root directory. I thought “-p” might do this but seems it only relates to the server target. Perhaps I am missing something as fairly new to Waypoint.

  2. Similarly (but less importantly) something like “waypoint init -dir ./app1” would be nice instead of having to descend into each directory. Not clear how to use existing “-from-project” and “-into” when the application directory already exists.

  3. We did have a top-level config stanza for workspace-specific env vars shared between apps. It would be nice to be able to include these common definitions somehow in the app-level waypoint.hcl files.

  4. We also had been using a top-level variable stanza to declare input variables shared between apps. Thought having a shared var-file might help us out here, but the variable declarations also need to be shared.

3 Likes

Hello everyone!

Thanks again for providing feedback on the decision to remove multi-app workflows from Waypoint. We really appreciate you all taking the time to share your setups with us.

First off, we want to apologize for the hard failure that accompanied the the deprecation warning. Our intentions were to warn users with the message, but unfortunately it shipped as a config validation error instead. We will release a patch soon to update the deprecation warning and remove the error.

The team has taken your feedback into consideration and decided that we don’t want to remove multi-application orchestration from Waypoint just yet. We understand the power and value in orchestrating these applications easily without forcing users to deal with config sprawl or needing to write extra wrappers around Waypoint to orchestrate deploying multiple applications. We are exploring the possibility of disallowing the configuration of multiple applications within a single waypoint.hcl file but want to continue to enable multi-applications in Waypoint. We are very interested to hear how you all envision configuring multiple applications with Waypoint and develop our product in a way that can support your workflows.

Thank you!

3 Likes

Hey all -

We recently released a patch version of Waypoint (v0.9.1) today that removes the hard depreciation failure and sets it to a warning instead as intended. See the release post here: Waypoint v0.9.1 released. Thank you! <3

2 Likes

If this is moving forward are there any plans to allow a “Terraform-like” ability to author these across multiple files?

The huge files are going to get unwieldy and fast - and if we are focusing to an application level, then having the ability to split out the variables, configs and other items that currently make up the bulk of the waypoint.hcl file would keep things clean and readable.

I would like to second the comments in this thread about being able to run multiple variants of the same app/codebase.

Especially in a Ruby on Rails scenario, it’s quite common to have a few app types in Procfile. Maybe a web definition and one or multiple ones for several queues/Sidekiq/background job workers.

So either would be nice to have multiple app definitions or files within one directory and one Project/App in Waypoint as these would have the exact same set of Input Variables turned into ENV vars.

Could also be that in the app stanza we could somehow specify multiple “executors” with per executor also how many replications and resource constraints should be applied.

The last would allow a Heroku-style deployment where one waypoint up would fully deploy the whole app infrastructure in a rolling way with a new code version and/or new ENV var settings.

Hey all!

We wanted to follow up with this post and say that we have decided to remove the multi-app deprecation warning from Waypoint in a future release. We intend to continue to support defining multiple applications within a single waypoint.hcl for now. We greatly appreciate your feedback on this. If there’s a missing feature that you’d like to see to support your use-case with Waypoint, please feel free to open a feature request on our GitHub. We would love to hear from you!

Thank you!
The Waypoint team <3

2 Likes

Given waypoints lambda support I would be really curious how many people actually have a multiple repos per lambda. I currently have 20+ lambdas in a repo. Splitting this out doesn’t make sense to me (or likely my Github administrator) as every other single tool I use, golang,terraform etc. Can work with this configuration.