SSH OTP auth with remote hosts (scale set) behind a load balancer

Hello all

our vault works fine for OTP auth with SSH secrets engine (in accordance to SSH Secrets Engine: One-Time SSH Password | Vault - HashiCorp Learn). I have recently deployed a VM scale set on Azure and would like to use them to load balance SSH connections while having vault OTP enabled. The set will be used as a platform entry bastion.

The issue/question is: how to request an OTP password which can be used with any ‘landing’ remote SSH host behind the load balancer?

OTP works fine when requested for the individual VMs in the set, but this beats the purpose of having a single destination IP at the load balancer and it’s not possible to predict which SSH server will handle the login.

Ideally, I’d like to request OTP using vault cli/curl/Invoke-WebRequest by having as $REMOTE_HOST_IP either the load balancer IP or the scale set subnet.

Note: OTP role is configured (for testing purposes) with cidr_list=0.0.0.0/0.

Cheers

Thanks aram, but I’m not sure you had the opportunity to carefully read the topic and understand the issue.

Will the command below

vault write ssh/creds/otp_key_role ip=x.x.x.x

accept a CIDR as ip (‘x.x.x.x’)?

Even though the SSH endpoints are behing a load balancer, their helpers will communicate with vault using their own IP addresses, so in this case, requesting an OTP using the LB IP ends up failing due to the fact that no actual server has this IP as its own.

An authenticated client requests credentials from the Vault server and, if authorized, is issued an OTP. When the client establishes an SSH connection to the desired remote host, the OTP used during SSH authentication is received by the Vault helper, which then validates the OTP with the Vault server.

If you’re talking about the Vault side, it’s not something you need to worry about, whichever node you end talking to from the client, the “active” node is the one that will respond to the request, since it’s a “write” request.

If you’re talking on the client side, then you’re breaking SSH by trying to route a SSH call through some sort of load balancer, that’s the definition of a man-in-the-middle attack.

Lastly the CIDR on the role, is for blocking purposes, so that you can say this role only applied to this small segment of my network. As you have pointed out you set the cidr-block on the role to 0.0.0.0, which means “ANY”. Source IP is now moot.

Thanks, aram. I’m talking client side only. Here’s a brief worflow of how I understand SSH OTP works:

  1. user requests an OTP password by either using curl/vault write to AN SPECIFIC IP (ip=x.x.x.x or $REMOTE_HOST_IP)
  2. user connects via ssh client to the LB IP address
  3. LB forwards the SSH connection to one of the scale set members ($REMOTE_HOST_IP)
  4. The chosen scale set member ($REMOTE_HOST_IP) contacts vault to check the received credential (OTP)
  5. IF the IP of the scale set member matches the $REMOTE_HOST_IP specified in the first curl/vault write request, IT WORKS. IF the IP does not match (i.e. we land on a different server), authentication fails with ‘wrong password’. Here it’s the vault server that is denying/granting the SSH helper access or not based on the source IP address of the server.

‘man in the middle’ does not apply because the SSH servers are connecting to vault using their own IP addresses, hence my question if vault write […] ip=x.x.x.x would allow a CIDR range.

I’m looking for a solid response whether the CIDR range is supported or even if vault would allow an OTP password originally issued to a particular IP address to be reused by other servers (provided they are in the same CIDR).

If not, it would be safe to assume that vault does not support SSH OTP login when the target destination servers are behind a load balancing solution or working as members of a scale set.