Vault in front of AWS Load Balancer

I reading through here and have hard time to understand how to proper setup Load Balancer on front of Vault in AWS. As per documentation I have to setup client_addr to point to AWS Load Balance DNS?

Thing is that I want to create a Lambda function that will use Vault API to complete some task, DNS Interface is unavailable in this scenario, so I decided to create private hosted name and internal load balancer for this purpose. But I find hard to understand complications and proper configuration. As per here I could not identify any non-standard configs, but they still not recommend to use it.

Thank you.

Hi again!

Have you seen the Vault Reference Architecture? I also did a webinar once on best practices using Terraform and Vault together here that is rather lengthy, but does show some examples of how to quickly use Terraform to spin up a best practices AWS cluster, which might also help.

-Becca

Hello Becca,
Thank you for your reply. I did phrase my question wrongly, apologies. I have hard not to setup cluster or Load Balancer, I haver hard times to configure and understand what are potential drawbacks and errors might appear.

Here are couple sources I went through:


https://groups.google.com/forum/#!topic/vault-tool/lcIxW7j38fY

  • So, I have Lambda application, that does not run either Vault or Consul client (and it cant), so Consul DNS Interface is not an option.
  • Lambda talks to AWS ALB that is attached to Vault Cluster. Since July 2018, ALB supports redirects, but I could not find what rules are proper to setup, or default will work.
  • I want to use forwarding instead of redirect, but documentation says that if forwarding will fail, it will fallback to redirect I still find confusing under what circumstances it will fallback.
  • As per configuration here, to enable forwarding all I need is to setup one listener for vault, and rest be propagated and forwarding is enabled per default.

I understand that AWS ALB is something not Vault related, but nowadays in serverless apps is something good to know if anyone encountered and how configured it, also what most worries is under what circumstances Vault can stop forwarding and start redirecting.

Thank you.

Gotcha! That is a good question. I haven’t played with ALB’s too much lately, so I will step back for other community members to respond.

Thank you @tyrannosaurus-becks, you might have idea who from vault team can give answer and highlight him? Thank you

Hi! I would point you to someone, but the question is more of an AWS implementation-specific one, rather than one regarding core Vault functionality. A great place to try asking might be in the Vault forum. It receives a lot of traffic from folks using Vault in many different ways, and may have people who’ve gone down a similar path.

@tyrannosaurus-becks I am interested to know under what circumstances internal forwarding can fail and use client-redirect as a fallback. What are most common cases?

Thank you.

AWS NLBs or Classic Load Balancers are preferred over AWS ALBs since they support a pass-through mode where TLS is terminated at Vault rather than the Load Balancer. See this Knowledge Base article for more information. https://support.hashicorp.com/hc/en-us/articles/4413810165395-Best-Practices-AWS-NLB-configuration-for-Vault

1 Like

What @jbayer said.
In case you’re going to setup a DR I’d recommend setting up two targets, 443->{any node} and 8200 & 8201->{active node) so that you can connect to the active node.
You can use /sys/health - HTTP API | Vault by HashiCorp to monitor which node is the active node.

1 Like

I was seeing these errors which was causing pods to restart on failed health check calls

  • forward request error: error=“error during forwarding RPC request”
  • error during forwarded RPC request: error=“rpc error: code = Canceled desc = context canceled”

I have an AWS ALB in front of vault. It was pointed to the vault service in the ingress. After I updated the ingress to point at the service that points to the active pod the errors went away.

Hopefully this will save someone some time