SSH OTP does not work

I am self-hosting Vault on a Ubuntu 22.04 on AWS.
The server listens on 0.0.0.0:8200.
Cloudflare Tunnel is installed, with https://vault.mydomain.com proxied to port 8200.
Visiting https://vault.mydomain.com Vault works normally.

Following this guide:

I have got vault-ssh-helper v0.2.1 downloaded, placed and permission set on the other server, on Ubuntu 20.04 on AWS.

file /etc/vault-ssh-helper.d/config.hcl:

vault_addr = "https://vault.mydomain.com"
ssh_mount_point = "ssh"
tls_skip_verify = true
allowed_roles = "*"
$ vault-ssh-helper -verify-only -config /etc/vault-ssh-helper.d/config.hcl
2024/02/03 00:03:17 [INFO] using SSH mount point: ssh
2024/02/03 00:03:17 [INFO] using namespace: 
2024/02/03 00:03:17 [INFO] vault-ssh-helper verification successful!

file /etc/pam.d/sshd:

# PAM configuration for the Secure Shell service

# Standard Un*x authentication.
@include common-auth

# Disallow non-root logins when /etc/nologin exists.
account    required     pam_nologin.so

# Uncomment and edit /etc/security/access.conf if you need to set complex
# access limits that are hard to express in sshd_config.
# account  required     pam_access.so

# Standard Un*x authorization.
@include common-account

# SELinux needs to be the first session rule.  This ensures that any
# lingering context has been cleared.  Without this it is possible that a
# module could execute code in the wrong domain.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so close

# Set the loginuid process attribute.
session    required     pam_loginuid.so

# Create a new session keyring.
session    optional     pam_keyinit.so force revoke

# Standard Un*x session setup and teardown.
auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay
@include common-session

# Print the message of the day upon successful login.
# This includes a dynamically generated part from /run/motd.dynamic
# and a static (admin-editable) part from /etc/motd.
session    optional     pam_motd.so  motd=/run/motd.dynamic
session    optional     pam_motd.so noupdate

# Print the status of the user's mailbox upon successful login.
session    optional     pam_mail.so standard noenv # [1]

# Set up user limits from /etc/security/limits.conf.
session    required     pam_limits.so

# Read environment variables from /etc/environment and
# /etc/security/pam_env.conf.
session    required     pam_env.so # [1]
# In Debian 4.0 (etch), locale-related environment variables were moved to
# /etc/default/locale, so read that as well.
session    required     pam_env.so user_readenv=1 envfile=/etc/default/locale

# SELinux needs to intervene at login time to ensure that the process starts
# in the proper default security context.  Only sessions which are intended
# to run in the user's context should be run after this.
session [success=ok ignore=ignore module_unknown=ignore default=bad]        pam_selinux.so open

# Standard Un*x password updating.
@include common-password

file /etc/ssh/sshd_config

#	$OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

Include /etc/ssh/sshd_config.d/*.conf

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes
TrustedUserCAKeys /etc/ssh/ca.pub

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile	.ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem	sftp	/usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

On my own PC, Vault is installed.

$ vault login -method="userpass" username=myusername
Password (will be hidden): 
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  hvs.xxxxx
token_accessor         xxxxx
token_duration         768h
token_renewable        true
token_policies         ["default" "dev-team" "test"]
identity_policies      []
policies               ["default" "dev-team" "test"]
token_meta_username    myusername

$ vault write ssh/creds/rleung ip=[IP of Ubuntu Server 20.04]
Key                Value
---                -----
lease_id           ssh/creds/myusername/xxxxx
lease_duration     768h
lease_renewable    false
ip                 [IP of Ubuntu Server 20.04]
key                xxxx-xxxx-xxxx-xxxx-xxxx
key_type           otp
port               22
username           ubuntu

$ ssh ubuntu@[IP of Ubuntu Server 20.04]
(ubuntu@[IP of Ubuntu Server 20.04]) Password: 
(ubuntu@[IP of Ubuntu Server 20.04]) Password: 
(ubuntu@[IP of Ubuntu Server 20.04]) Password: 
ubuntu@[IP of Ubuntu Server 20.04]: Permission denied (publickey,keyboard-interactive).

Using the “key” (xxxx-xxxx-xxxx-xxxx-xxxx) does not work.

On Vault server (Ubuntu 22.04), audit log is enabled.
There is log when “vault login”, “vault write”, but not when “ssh”.
As far as I understand, the destination server (SSH to) should query Vault server when someone SSH into it.
What might I missed?

Thanks.

Hi @thematrixdev ,

Are you trying to SSH to the Vault server itself with the OTP or a 2nd Ubuntu instance?

Hello @jonathanfrappier
Vault server is on Ubuntu 22.04
Destination server is on Ubuntu 20.04
I am connecting from my PC to the destination server.

Thanks for clarifying (and apologies if I missed that). Can your destination Ubuntu instance connect to your Vault server directly? I went through this tutorial a few months ago and everything was working (Vault running in HCP, a Linux instance in AWS (forget if it was Ubuntu or AWS Linux), and my personal laptop making the request)

A quick test could be using cURL to query the seal-status endpoint:

curl https://vault.mydomain.com:8200/v1/sys/seal-status
$ curl https://vault.mydomain.com/v1/sys/seal-status
{"type":"shamir","initialized":true,"sealed":false,"t":3,"n":5,"progress":0,"nonce":"","version":"1.15.5","build_date":"2024-01-26T14:53:40Z","migration":false,"cluster_name":"vault-cluster-50503854","cluster_id":"1068717f-a7b0-f18d-f5f1-8ceb1bac74a1","recovery_seal":false,"storage_type":"file"}

I have just enabled SSH access log on the destination server, and found something useful.

On home PC:

$ vault ssh -mode=otp -role=otp_key_role ubuntu@DESTINATION_SERVER_IP
failed to run ssh command: exit status 5

On Vault server:

{"time":"2024-02-02T16:16:22.073951585Z","type":"request","auth":{"client_token":"hmac-sha256:b236b6042bc0ab99998fc7a53894fe65c8f11044939c501130a996c67b7b8900","accessor":"hmac-sha256:231db5e95f353b893fa0c910f6fed43d02328583b926fd15a8d8d2743f9489d0","display_name":"root","policies":["root"],"token_policies":["root"],"policy_results":{"allowed":true,"granting_policies":[{"name":"root","namespace_id":"root","type":"acl"}]},"token_type":"service","token_issue_time":"2024-02-01T22:25:53+08:00"},"request":{"id":"4f36fbf5-df09-b6c6-3711-33450484db30","client_id":"0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=","operation":"update","mount_point":"ssh/","mount_type":"ssh","mount_accessor":"ssh_a91ab4ec","mount_running_version":"v1.15.5+builtin.vault","mount_class":"secret","client_token":"hmac-sha256:b236b6042bc0ab99998fc7a53894fe65c8f11044939c501130a996c67b7b8900","client_token_accessor":"hmac-sha256:231db5e95f353b893fa0c910f6fed43d02328583b926fd15a8d8d2743f9489d0","namespace":{"id":"root"},"path":"ssh/creds/otp_key_role","data":{"ip":"hmac-sha256:268179fe75de2aa643dab891e8520c7107e5fc8d29c2faa77084dbd8efed18c7","username":"hmac-sha256:b51b24d0643c1fc4cd283723f3b366ba010c195fd84af7d29311a9fd8297aebf"},"remote_address":"172.18.0.23","remote_port":53212}}
{"time":"2024-02-02T16:16:22.075266907Z","type":"response","auth":{"client_token":"hmac-sha256:b236b6042bc0ab99998fc7a53894fe65c8f11044939c501130a996c67b7b8900","accessor":"hmac-sha256:231db5e95f353b893fa0c910f6fed43d02328583b926fd15a8d8d2743f9489d0","display_name":"root","policies":["root"],"token_policies":["root"],"policy_results":{"allowed":true,"granting_policies":[{"name":"root","namespace_id":"root","type":"acl"}]},"token_type":"service","token_issue_time":"2024-02-01T22:25:53+08:00"},"request":{"id":"4f36fbf5-df09-b6c6-3711-33450484db30","client_id":"0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=","operation":"update","mount_point":"ssh/","mount_type":"ssh","mount_accessor":"ssh_a91ab4ec","mount_running_version":"v1.15.5+builtin.vault","mount_class":"secret","client_token":"hmac-sha256:b236b6042bc0ab99998fc7a53894fe65c8f11044939c501130a996c67b7b8900","client_token_accessor":"hmac-sha256:231db5e95f353b893fa0c910f6fed43d02328583b926fd15a8d8d2743f9489d0","namespace":{"id":"root"},"path":"ssh/creds/otp_key_role","data":{"ip":"hmac-sha256:268179fe75de2aa643dab891e8520c7107e5fc8d29c2faa77084dbd8efed18c7","username":"hmac-sha256:b51b24d0643c1fc4cd283723f3b366ba010c195fd84af7d29311a9fd8297aebf"},"remote_address":"172.18.0.23","remote_port":53212},"response":{"mount_point":"ssh/","mount_type":"ssh","mount_accessor":"ssh_a91ab4ec","mount_running_plugin_version":"v1.15.5+builtin.vault","mount_class":"secret","secret":{"lease_id":"ssh/creds/otp_key_role/ZKtxnhvd85j0Icy46AU1plWk"},"data":{"ip":"hmac-sha256:268179fe75de2aa643dab891e8520c7107e5fc8d29c2faa77084dbd8efed18c7","key":"hmac-sha256:f13eb26b59b23c360f613b133a6e207474fd598701615396e5e21e253dd7e295","key_type":"hmac-sha256:188fe326f146ec824971e478fc52d2cbc13f7779ab2d5b95effd935a657ed0a1","port":22,"username":"hmac-sha256:b51b24d0643c1fc4cd283723f3b366ba010c195fd84af7d29311a9fd8297aebf"}}}

On the destination server:

$ sudo tail -f /var/log/auth.log

Feb  3 00:13:54 hk-uat sshd[3514480]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=HOME_IP user=ubuntu
Feb  3 00:13:56 hk-uat sshd[3514478]: error: PAM: Authentication failure for ubuntu from HOME_IP
Feb  3 00:13:56 hk-uat sshd[3514478]: Connection closed by authenticating user ubuntu HOME_IP port 51684 [preauth]

It seems that the destination did not contact Vault server, but failed immediately.

Which of the 3 machines involved is 172.18.0.23? And what is the CIDR list set up for in your role (SSH secrets engine: One-time SSH password | Vault | HashiCorp Developer)?

On the Ubuntu server 22.04 which Vault is running on, Cloudflare Tunnel (cloudflared) is run on a Docker container.
Vault run on the host listens on 0.0.0.0:8200.
It is configured on Cloudflare that vault.mydomain.com proxies to host:8200
The 172 IP address is Docker network.

Thanks.

I have updated the network configuration for easier debug.
Now vault.mydomain.com is a A record pointing to the server.
On the server NGINX listens on 80 and 443, reverse proxies vault.mydomain.com to 0.0.0.0:8200.
Access log is enabled.

When “vault ssh” is run, only this line is shown on the log:

172.71.218.239 - - [03/Feb/2024:10:32:58 +0800] "PUT /v1/ssh/creds/otp_key_role HTTP/2.0" 200 325 "-" "Go-http-client/2.0"

172.71.218.239 is a Cloudflare IP.
Since it is proxied from Cloudflare.

So the destination server does not make any request to Vault server.

I have just updated a little bit.
Vault server and destination server are on AWS in same region same zone.
I have modified the destination server to connect to Vault server via internal IP.

/etc/vault-ssh-helper.d/config.hcl

vault_addr = "http://10.x.x.x:8200"
ssh_mount_point = "ssh"
tls_skip_verify = true
allowed_roles = "*"
$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl
2024/02/03 10:43:51 ==> WARNING: Dev mode is enabled!
2024/02/03 10:43:51 [INFO] using SSH mount point: ssh
2024/02/03 10:43:51 [INFO] using namespace: 
2024/02/03 10:43:51 [INFO] vault-ssh-helper verification successful!

/etc/pam.d/sshd

...
auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl -log-level=trace
auth optional pam_unix.so not_set_pass use_first_pass nodelay
...

SSHD restarted.

Same result. Only one “PUT” request on NGINX log.

SSHD log (debug level)

Feb 03 03:27:43 monitor sshd[40409]: debug1: Forked child 40420.
Feb 03 03:27:43 monitor sshd[40420]: debug1: Set /proc/self/oom_score_adj to 0
Feb 03 03:27:43 monitor sshd[40420]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Feb 03 03:27:43 monitor sshd[40420]: debug1: inetd sockets after dupping: 4, 4
Feb 03 03:27:43 monitor sshd[40420]: Connection from HOME_PC_IP port 39986 on 10.28.5.43 port 22 rdomain ""
Feb 03 03:27:43 monitor sshd[40420]: debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
Feb 03 03:27:43 monitor sshd[40420]: debug1: Remote protocol version 2.0, remote software version OpenSSH_9.3p1 Ubuntu-1ubuntu3.2
Feb 03 03:27:43 monitor sshd[40420]: debug1: compat_banner: match: OpenSSH_9.3p1 Ubuntu-1ubuntu3.2 pat OpenSSH* compat 0x04000000
Feb 03 03:27:43 monitor sshd[40420]: debug1: permanently_set_uid: 109/65534 [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: SSH2_MSG_KEXINIT received [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: kex: host key algorithm: ssh-ed25519 [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: SSH2_MSG_KEX_ECDH_INIT received [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: ssh_packet_send2_wrapped: resetting send seqnr 3 [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: rekey out after 134217728 blocks [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: Sending SSH2_MSG_EXT_INFO [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: ssh_packet_read_poll2: resetting read seqnr 3 [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: rekey in after 134217728 blocks [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: KEX done [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: userauth-request for user ubuntu service ssh-connection method none [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: attempt 0 failures 0 [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: PAM: initializing for "ubuntu"
Feb 03 03:27:43 monitor sshd[40420]: debug1: PAM: setting PAM_RHOST to "HOME_PC_IP"
Feb 03 03:27:43 monitor sshd[40420]: debug1: PAM: setting PAM_TTY to "ssh"
Feb 03 03:27:43 monitor sshd[40420]: debug1: userauth-request for user ubuntu service ssh-connection method keyboard-interactive [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: attempt 1 failures 0 [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: keyboard-interactive devs  [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: auth2_challenge: user=ubuntu devs= [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: kbdint_alloc: devices 'pam' [preauth]
Feb 03 03:27:43 monitor sshd[40420]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Feb 03 03:27:43 monitor sshd[40420]: Postponed keyboard-interactive for ubuntu from HOME_PC_IP port 39986 ssh2 [preauth]
Feb 03 03:27:43 monitor sshd[40422]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=HOME_PC_IP  user=ubuntu
Feb 03 03:27:46 monitor sshd[40420]: error: PAM: Authentication failure for ubuntu from HOME_PC_IP
Feb 03 03:27:46 monitor sshd[40420]: Failed keyboard-interactive/pam for ubuntu from HOME_PC_IP port 39986 ssh2
Feb 03 03:27:46 monitor sshd[40420]: debug1: userauth-request for user ubuntu service ssh-connection method keyboard-interactive [preauth]
Feb 03 03:27:46 monitor sshd[40420]: debug1: attempt 2 failures 1 [preauth]
Feb 03 03:27:46 monitor sshd[40420]: debug1: keyboard-interactive devs  [preauth]
Feb 03 03:27:46 monitor sshd[40420]: debug1: auth2_challenge: user=ubuntu devs= [preauth]
Feb 03 03:27:46 monitor sshd[40420]: debug1: kbdint_alloc: devices 'pam' [preauth]
Feb 03 03:27:46 monitor sshd[40420]: debug1: auth2_challenge_start: trying authentication method 'pam' [preauth]
Feb 03 03:27:46 monitor sshd[40420]: Postponed keyboard-interactive for ubuntu from HOME_PC_IP port 39986 ssh2 [preauth]
Feb 03 03:27:46 monitor sshd[40420]: Connection closed by authenticating user ubuntu HOME_PC_IP port 39986 [preauth]
Feb 03 03:27:46 monitor sshd[40420]: debug1: do_cleanup [preauth]
Feb 03 03:27:46 monitor sshd[40420]: debug1: monitor_read_log: child log fd closed
Feb 03 03:27:46 monitor sshd[40420]: debug1: do_cleanup
Feb 03 03:27:46 monitor sshd[40420]: debug1: PAM: cleanup
Feb 03 03:27:46 monitor sshd[40420]: debug1: Killing privsep child 40421
Feb 03 03:27:46 monitor sshd[40420]: debug1: audit_event: unhandled event 12

I have got it working finally.

  1. I have mistakenly remarked @include common-session instead of @include common-auth

  2. auth requisite and auth optional should be above @include common-auth so as to trigger connection to Vault server

  3. @include common-auth must be remarked, or SSH will still fail even successfully verified by Vault server.

(Actually I want the destination server accept both traditional SSH key and also Vault OTP)

  1. In /etc/vault-ssh-helper.d/config.hcl, allowed_cidr_list must be defined.
    Vault ssh cannot connect to remote sucessfully - #8 by liu

Thanks.

Awesome! Glad to hear its working.