We are thinking about rolling out SSH host certificates on linux servers. The question is how do you roll those out if you have to authenticate the host against vault in the first place?
In the cloud world there would be some kind of Workload Identity which could be used to authenticate a host against vault and this could be mapped to a role, so those hosts could sign their SSH keys themselves…but with VMs in vSphere or even bare metal servers, how are you solving the problem of the first secret?
Or should we turn this around and use a config management (Ansible, Chef, Puppet,…) to roll out the certs from a central location? Or should we use the same config management to roll out AppRole creds and use this then to authenticate and sign certs. I would not prefer this solutions because then there is this single powerful place in the network which controls authenticity for every system. I would rather prefer to build something like certbot, but using the Vault agent and every server regularly connects to Vault to sign it’s key.
Can anybody recommend something or share some experience?
If you’re using signed SSH keys, then the SSH mount’s CA public key is an unauthenticated API endpoint and can be downloaded freely by anything that’s able to communicate with your Vault cluster.
You will need to develop a method of pulling down the correct CA cert(s) and updating the sshd.conf file to leverage this. I’m personally not aware of any existing “helpers” for this activity. (Note that HashiCorp’s Vault SSH Helper utility is for OTP setups.)
However, using config management software to push the config is an equally valid approach, IMO. Use whichever approach makes the most sense for your use cases.
Thanks for getting back to me. I think I explained a bit poorly what I wanted to achieve. What I want is every linux host having an SSH host certificate. I want to solve the trust-on-first-use issue with SSH and so every host key should be signed by the SSH CA. This means I am not talking about SSH user certificates and the host trusting the CA, but the host having itself a cert.
So if I have 500 servers, I can create a cronjob for example that creates a host key, authenticates against vault and gets this key signed. BUT how should the hosts authenticate against Vault if they don’t have creds already? I could use an AppRole in the script, but rolling out a script with plaintext credentials seems not a good solution. Is it maybe better to have a central script that pushed out the certs to the hosts? …but having this single place which can create certs for all the hosts seems not really great as well.
So what’s the most secure solution in terms of least privilege, automation and maybe even short lived certs (including automatic renewal) for SSH host certs?
Basically you’re looking to solve the authentication issue in step 6 within the host verification setup instructions, correct?
Or is it steps 5 and 6 (I’m not sure where the host’s public key is coming from here)?
Regardless you have some options:
Use a config management tool like you mentioned previously
Use a common credential (AppRole role_id+secret_id) in your script, which has the obvious drawbacks you mentioned
You could leverage an AppRole role with the bind_secret_id parameter set to false and set the token_bound_cidrs parameter to the list of your authorized host subnets (this may be best paired with the token_no_default_policy to better restrict the capabilities of the token). This method assumes you have a well regulated set of IPs/subnets per target role.
Use the Trusted Orchestrator or similar approach - this would be a bit of a hybrid model where you could include an AppRole role_id in your image then use a tool such as Ansible to drop a secret_id on the target so that only the target has the full credential. Then have the target run the script to auth and pull down the SSH cert. If all you’re aiming to accomplish is the SSH config then this might be a bit overkill but if you’re looking to enable broader Vault usage then this might be an attractive option.
I can’t tell you what’s best for your environment, of course. However, hopefully some of these options help you determine an appropriate solution for your use case.
I’d recommend building a matrix of sorts and poking holes in each of the options and the one that has the least/smallest holes in it would probably be your best bet. You can always add more layers (monitoring/alerting, for instance) to better secure the solution.
Our *nix hosts are Kerberized, so when they’re staged, they all get keytab files.
We’re trying to authenticate using the host’s Kerberos credentials to get the vault token and then use that to get the host key signed. This, we’ve managed to accomplish.
The problem now is how to limit the valid_principals that it can request? Ideally, when then host authenticates as hostname$@REALM (or host/hostname.realm@REALM), it should only be allowed to request a certificate for the principal hostname.realm. I don’t see a good way to do that.