Vault SSH OTP passwords

[admin4]

Hi y’all,
I’ve completed the SSH OTP configuration for a Debian server and it’s functioning correctly.

The otp_key_role was configured for a test user:

vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=testuser \
    cidr_list=0.0.0.0/0

so testuser can now login to Vault and perform ssh with:

export SSHPASS=$(vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP | grep "key " | sed 's/^key *//')
sshpass -e ssh -o PubkeyAuthentication=no testuser@$REMOTE_HOST_IP

or more simply with:
vault ssh -role otp_key_role -mode otp -strict-host-key-checking=no testuser@$REMOTE_HOST_IP

Now I’d like to extend this function to other users: do I need to create a new role for each user or is there some templating allowing to inject the “Vault identity” into the role and assign it to the “default_user” parameter?

I found templating for the policies but not for the roles, is it possible to do the above? Thanks!

[Joffrey]

Hi,

A role is … a role, not a user. You create a role ‘dbadmin’, ‘operator’, ‘sysadmin’, … It’s a group of users. it’s nonsense to create a role for each user.
Users must pass their username on the creds endpoint (doc: SSH API ).

[maxb]

@Joffrey

A role is … a role, not a user. You create a role ‘dbadmin’, ‘operator’, ‘sysadmin’, … It’s a group of users. it’s nonsense to create a role for each user.

A role in Vault is not the same as a role in database permissions either. It’s, if not nonsense, pretty suboptimal to create different Vault roles for groups, too.

@Joffrey

Users must pass their username on the creds endpoint (doc: SSH API ).

Perhaps, depending on some architectural limitations, but not necessarily true if the defaulting can be made to do what you want.

[maxb]

@admin4

I found templating for the policies but not for the roles, is it possible to do the above? Thanks!

You should look at the documentation for the SSH secrets engine role configuration - there are several options containing template in their names, for turning on treating parts of the configuration as templates.

[admin4]

Hi,
I’m new to Vault and I am realizing there are many things I’ve not yet well understood, specially about roles, groups, identities, etc.

For the above test with SSH OTP I just followed this tutorial:
https://developer.hashicorp.com/vault/tutorials/secrets-management/ssh-otp

This point in the tutorial hints to create a role for each username:

so I was wondering if there was a way to “templating” this role creation (instead of manually create a new role for each user in the organization).

Maybe my question is better explained if I start from the beginning

• My organization uses an IDP to handle its users (Keycloak), so all the users are created inside the IDP and are authenticated thru an SSO entry point
• After installing Vault I enabled the OIDC Auth method, created the relative client in the IDP and then made some tests to get access to Vault thru SSO either via web as well as via CLI. All the tests where ok so now I can get access to Vault thru the SSO
• At this point I tried the SSH OTP function, so I created a Debian server and followed the tutorial above. Everything went smooth, so I can get SSH access to the server via OTP released by Vault. But this is valid for the “testuser” only, i.e. the username defined in the ssh/roles/otp_key_role created in Vault during the tutorial.

What I am trying to do is:

• Create two new users alice and bob in the IDP
• Assign some role or metadata to bob (and to bob only!) inside the IDP to give him access to the SSH OTP function on Vault
• Create the users bob and alice in the Debian server

At this point only bob should be able to access the Debian server via SSH OTP: Vault refuses to release the SSH OTP to alice, as alice (user authenticated in the IDP) is missing the role (or the metadata) allowing her to access the SSH OTP function on Vault.

According to the tutorial, in order to achive this granularity on Vault I need to create a specific role for each username I want to abilitate to the SSH OTP function on Vault. Something like this:

ssh/roles/otp_key_role_bob
ssh/roles/otp_key_role_alice

To do this for many users is time consuming so I was wondering if there was a templating function where I can inject the username (which I can send in the token from the IDP) inside the ssh/roles/otp_key_role.

But maybe my approach is wrong and there is an easier way to get this goal. Help highly appreciated, thanks!

[maxb]

@admin4

This point in the tutorial hints to create a role for each username:

Many tutorials, even HashiCorp’s own, recommend what is easy to set up and play with, not what is suitable for production use.

As mentioned above:

@maxb

You should look at the documentation for the SSH secrets engine role configuration - there are several options containing template in their names, for turning on treating parts of the configuration as templates.

In your case turning on default_user_template and allowed_users_template, and setting default_user and allowed_users to the same template, seems potentially suitable.

It sucks that you have to duplicate this configuration, but as per the docs, Vault has crazily permissive default behaviour for some settings in the SSH secrets engine.

This does, of course, depend on you being able to get your identity provider to supply suitable metadata, and the Vault OIDC auth method set up to capture it.

Be sure to test what happens if the user is missing the relevant metadata. Unless I confirmed the behaviour myself, I would worry that would then allow the user without metadata to log in as any user.

[admin4]

@maxb

I would worry that would then allow the user without metadata to log in as any user.

After making more tests, this is what’s actually happening! I can access to the SSH server with username “testuser” after authenticating as user alice on the IDP, even if alice does not have any metadata or role (related to Vault) assigned on the IDP.

I need to study this and make more tests in order to understand what’s happening. Thanks for your help!

[maxb]

There are several settings in the Vault SSH secrets engine which treat unconfigured/empty as “allow all”. This is a bad stance for a security tool. Some have been fixed over the years, but others have not.

Were I in a position of freshly deploying Vault today, I’d probably want to work with HashiCorp to fix them, potentially running a patched version until fixes could be merged.

[admin4]

@maxb

@admin4

This point in the tutorial hints to create a role for each username:

Many tutorials, even HashiCorp’s own, recommend what is easy to set up and play with, not what is suitable for production use.

I understand, but that point (to create a role for each username) is exactly what Hashicorp recommends for production use! I’m a bit disoriented on this, but I’ll keep doing tests until I figure out if I can trust this solution or if I should go back to the old SSH Keys (possibly signed by Vault )

[admin4]

@maxb

In your case turning on default_user_template and allowed_users_template, and setting default_user and allowed_users to the same template, seems potentially suitable.

After making some tests I found that unfortunately these options (default_user_template and allowed_users_template) are not allowed for the OTP key type (only for the CA type), so the only possible solution is to create an ssh role for each ssh_username to be used by the users.

In order to avoid unauthorized users to login with a username different than the one assigned, I use a template to inject the ssh_username in the policy, so that the user can create the OTP in the ssh/creds/ path only for the assigned ssh_username.

This is not an ideal solution, as it allows only a single ssh_username to be assigned to the user and forces the creation of as many ssh roles as the ssh_usernames to be used on the servers.

It would be much simpler if it were possible to use the template in the role definition, with two main advantages:

• the possibility to use a single role for all the ssh_usernames
• the possibility to assign more than one ssh_username (by injecting the list in the “allowed_users” field)
  With such configuration the authorization to use a specific ssh_username would be handle by the role and not by the policy.

[admin4]

Thanks for your suggestions (I’m not quoting as you deleted the post but I appreciate them indeed).
We are doing some tests for a possible project involving a campus with many depts and hundreds of users. A user can have on-campus access to servers in different departments. Off campus access is possible with VPN or through dedicated bastion servers. Users are provisioned by LDAP-AD. They have a campus password but no MFA. Users can login after uploading their public ssh key to LDAP.

We are thinking of a solution based on Keycloak as IdP, federated with the existent LDAP-AD system, plus HC Vault as SSH OTP engine. We can already achive that, but in order to authorize the ssh username we need to create a ssh role for each username with annexed policy to authorize the /creds path. It would be great to be able to use the template in the role definition, as with the CA signing roles.

[tsiamer]

Here we go.

This simple ansible playbook, made quick you can adjust as needed, can probably get you started, you should not focus only on users, consider as well cidr_list, in regards to roles try to think of them as tiering 0, 1 , 2 “to reduce the blast radius in an event of a compromise”, and make your plan accordingly, so any users in role 0 should not and must not access role 1 & 2 and vice versa as an example, not using otp role myself, using signed cert, for windows smartcard with Yubikey or yubkey - duo, with little imagination you can map users, allowed_users and cidr_list variables with jinja templating or powershell somehow.
One last thing it will be much manageable if you assign the policies by ldap groups instead of users. Hope this help.

- name: "Vault_OTP"
  hosts:  localhost
  become: true

  vars_files:
    - vault_otp.yml


  tasks:


    - name: Enable Vault OTP ssh
      ansible.builtin.shell: vault secrets enable ssh 
      environment:  
        VAULT_ADDR:  "{{VAULT_ADDR}}"
        VAULT_CACERT: "{{VAULT_CACERT}}"
        VAULT_CLIENT_CERT: "{{VAULT_CLIENT_CERT}}"
        VAULT_CLIENT_KEY: "{{VAULT_CLIENT_KEY}}"
        VAULT_TOKEN: "{{VAULT_TOKEN}}"
      register: vault_ssh_enabled
      changed_when: false
      ignore_errors: true

    - name: Vault OTP ssh users
      ansible.builtin.shell: vault write ssh/roles/"{{item.role}}" key_type=otp  default_user="{{item.default_user}}" cidr_list="{{item.cidr_list}}" allowed_users="{{item.allowed_users}}"                                 
      environment:  
        VAULT_ADDR:  "{{VAULT_ADDR}}"
        VAULT_CACERT: "{{VAULT_CACERT}}"
        VAULT_CLIENT_CERT: "{{VAULT_CLIENT_CERT}}"
        VAULT_CLIENT_KEY: "{{VAULT_CLIENT_KEY}}"
        VAULT_TOKEN: "{{VAULT_TOKEN}}"
      changed_when: false
      delegate_to: localhost
      with_items: "{{ssh_otp_users}}"
      when: vault_ssh_enabled.rc == 0

Variables

VAULT_ADDR:  "https://active.vault.service-consul:8200"
VAULT_CACERT: "/etc/ansible/consul-agent-ca.pem"
VAULT_CLIENT_CERT: "/etc/ansible/cli--consul-0.pem"
VAULT_CLIENT_KEY: "/etc/ansible/cli-consul-0-key.pem"
VAULT_TOKEN: "xxxxxx"




ssh_otp_users:
      - { role: 'siamert',   default_user: 'siamert', cidr_list: '192.168.1.20/24', allowed_users: 'siamert, bob'}
      - { role: 'siamert1',  default_user: 'siamert', cidr_list: '192.168.1.21/24, 192.168.1.23/24', allowed_users: 'siamert1'}
      - { role: 'siamert2',  default_user: 'siamert', cidr_list: '192.168.1.22/24', allowed_users: 'siamert2'}
      - { role: 'siamert3',  default_user: 'siamert', cidr_list: '192.168.1.23/24', allowed_users: 'siamert3'}
      - { role: 'siamert4',  default_user: 'siamert', cidr_list: '192.168.1.24/24', allowed_users: 'siamert4'}
      - { role: 'siamert5',  default_user: 'siamert', cidr_list: '192.168.1.25/24', allowed_users: 'siamert5'}
      - { role: 'siamert6',  default_user: 'siamert', cidr_list: '192.168.1.26/24', allowed_users: 'siamert6'}
      - { role: 'siamert7',  default_user: 'siamert', cidr_list: '192.168.1.27/24', allowed_users: 'siamert7'}
      - { role: 'siamert8',  default_user: 'siamert', cidr_list: '192.168.1.28/24', allowed_users: 'siamert8'}
      - { role: 'siamert9',  default_user: 'siamert', cidr_list: '192.168.1.29/24', allowed_users: 'siamert9'}
      - { role: 'siamert10', default_user: 'siamert', cidr_list: '192.168.1.30/24', allowed_users: 'siamert10'}
      - { role: 'siamert11', default_user: 'siamert', cidr_list: '192.168.1.31/24', allowed_users: 'siamert11'}
      - { role: 'siamert12', default_user: 'siamert', cidr_list: '192.168.1.32/24', allowed_users: 'siamert12'}
      - { role: 'siamert13', default_user: 'siamert', cidr_list: '192.168.1.33/24', allowed_users: 'siamert13'}
      - { role: 'siamert14', default_user: 'siamert', cidr_list: '192.168.1.34/24', allowed_users: 'siamert14'}
      - { role: 'siamert15', default_user: 'siamert', cidr_list: '192.168.1.35/24', allowed_users: 'siamert15'}

[admin4]

Thanks again for your reply. I do not use Ansible, maybe this is the right occasion to try it. I get your point and in my test I am doing exactly as you suggest: one role for each ssh username I need to allow to use SSH OTP. This is how I do it.

On IDP

• User authenticates and gets HCVault/SSHusers group and ssh_role attribute

On Vault

• User is assigned to the external group SSHUsers through alias HCVault/SSHusers
• User gets the sshuser policy from the SSHUsers Vault group
• sshuser policy is a TEMPLATED policy: the user gets access to the specific role by injection of the role name in the creds path:

# To use the configured SSH secrets engine role
path "ssh/creds/{{identity.entity.aliases.auth_oidc_xxxxxxx.metadata.ssh_role}}" {
  capabilities = ["create", "read", "update"]
}

• user is instructed to first login on the IDP and then use the assigned username as a role to access the server:

REMOTE_HOST_IP=< ssh server IP >
vault ssh -role username -mode otp -strict-host-key-checking=no username@$REMOTE_HOST_IP

It would be much simpler if I could use a TEMPLATE in the role definition. I could define, for example, a role for each department (with the appropriate CIDR/27) and inject “default_user” and “allowed_users” by using metadata taken from the OIDC token.

Unfortunately the use of a template in the role structure is possible for the key of type “ca” but it’s not allowed for key of type “otp”.

I was wondering if it could be possible to extend the template to the otp type key (or maybe this is a problem too hard to solve).

[tsiamer]

one role for each ssh username I need to allow to use SSH OTP. This is how I do it. No not like that.

You need to plan properly, my advise is once you understand who need to access what, put the users in the group if they are not yet .

Group the servers like so as an example:

DB servers
WEB servers
Infra and so on

What you do, let us say you have db servers managed by db team you can create a role db-something role add only users who need access to it to the allowed list, and the cidr_list where the servers resides, it will be not be practical, manageable to add one user per each role as you will have redundant roles, if there are many users that needs the same access and now you have setup the role with based on the username, it wont make sense, my example on the playbook was quickly done just think the role as group of servers, tiering…

Assign the policy to the AD- IDP groups not individual users.

Using groups has lots of advantage for example, let us say if someone leaves the company you just have to remove them from AD - IDP group.

You can use the same logic as the above playbook, your only item left to complete is to create policies and assign them to the groups.

[admin4]

I am not sure I understand your schema. Lets make a simple example (reality is a bit more complex, but not much more):

We have some Dept01 WEB servers on 10.1.1.0/24

• dept01web01
• dept01web02
• dept01web03

and some Dept01 DB servers on 10.1.2.0/24

• dept01db01
• dept01db02

Let’s say we do not need, for the moment, to specifically authorize access to the single server inside each family, so we can use just one CIDR for the whole server family.

Then we have our users:

• alice
• bob
• mario
• ted

Users need ssh access to those servers according to these rules:

• all the 4 users can access the WEB Servers
• only mario and ted can access the DB servers

Users authentication is handled by the IDP and we wanna use SSH OTP created in Vault.

If I understand it correctly, your schema is to define two roles, one for each family of servers and to add the usernames to the allowed_users field in each role. Like this:

• ssh_dept01-web: CIDR=10.1.1.0/24 default_user=nologin allowed_users=alice,bob,mario,ted
• ssh_dept01_db: CIDR=10.1.2.0/24 default_user=nologin allowed_users=mario,ted

Is this correct or am I misinterpreting it?

[tsiamer]

Exactly and create a policy depending on your naming convention, then map that policy for the group in AD not users.

Policy_name:

ssh_pol_dept01-web

Example:

Group_name: — grp_us_web_dep01
This group contains the 4 users:

alice,bob,mario,ted

vault write auth/ldap/groups/grp_us_web_dep01 policies=ssh_pol_dept01-web

[admin4]

Ok, that’s exactly what I tried in my first tests weeks ago.

There’s only a problem: each user will get their OTP key using the single role assigned to the server family and that role allows ALL the usernames!

For example, let’s say that bob wants to login to dept01web03. He authenticate on the IDP and gets the authorization to use the ssh_dept01_web role from the group he was assigned to. He can now get an OTP key for the dept01web03 server. To get the OTP he can write directly to the creds path and get the token to extract the key:

REMOTE_HOST_IP=10.1.1.37
vault write ssh/creds/ssh_dept01_web ip=$REMOTE_HOST_IP username=bob
Key                Value
---                -----
lease_id           ssh/creds/ssh_dept01_web/WIpvf6PAvQ8uOzJcI50jtGmL
lease_duration     8h
lease_renewable    false
ip                 10.1.1.37
key                f622f35e-1c08-f8c4-1bd4-656660da6539
key_type           otp
port               22
username           bob

and then he uses the key to login. If sshpass is available he can inject the key directly into the login command:

REMOTE_HOST_IP=10.1.1.37
export SSHPASS=$(vault write ssh/creds/ssh_dept01_web ip=$REMOTE_HOST_IP username=bob | grep "key " | sed 's/^key *//')
sshpass -e ssh -o PubkeyAuthentication=no bob@$REMOTE_HOST_IP

Or he could use the vault ssh command to login directly:

REMOTE_HOST_IP=10.1.1.37
vault ssh -role ssh_dept01_web -mode otp -strict-host-key-checking=no bob@$REMOTE_HOST_IP

The problem
How do you prevent bob to login with alice username and access all alice’s private files? Once he’s assigned the authorization to use the ssh_dept01_web role, there’s nothing to prevent him to login as alice:

REMOTE_HOST_IP=10.1.1.37
vault ssh -role ssh_dept01_web -mode otp -strict-host-key-checking=no alice@$REMOTE_HOST_IP

With this configuration bob can obtain a token for the alice username, as it is allowed to do that by the ssh_dept01_web role.

That’s why I’d like to be able to inject the username(s) into the role with a template, it’s the only way to ensure isolation between usernames when they authenticate via OIDC. Otherwise you need a role for each username. I haven’t found another way until now.

[tsiamer]

Yes valid point, by the look of it bob can impersonate Alice, have not played with ssh otp, if you can create roles per user and assign multiple subnets to each user role as per the findings, if you really want to stick with ssh otp.

[admin4]

I like the SSH OTP way because all the authentication and authorization layers are inside the IDP, and I just need to install vault_helper on the servers to get the job done (with audit on the IDP).

Unfortunately, unless I find a way to inject the username(s) inside the ssh roles, handling hundreds or even thousands of users assigning a single role to each of them it is not a viable solution imho.

I will check the signed CA as a possible alternative.

Thanks again for your help and your hints!

[tsiamer]

No problem thanks , hope that helps.