No policies when logging in with Vault OIDC in Nomad

Nomad
1

After (a lot) of struggle I have followed the Using Vault as an OIDC provider for Single Sign-On | Nomad | HashiCorp Developer, and I can login using the “Log in with vault” button, that will redirect to vault and do all the magic. During the process, many questions come up as to what is exactly meant and what exactly the commands mean.

I can login, but the resulting user has no policies associated with him and can’t do anything. Is there anything I can do to debug?

My Nomad acl config is almost exactly as in the manual:

nomad acl role info 2c3b87b5-5712-b1fd-e931-0fa2232a0b4d ; echo
ID           = 2c3b87b5-5712-b1fd-e931-0fa2232a0b4d
Name         = avtdevops
Description  = acl role for avtdevops
Policies     = avtdevops
Create Index = 2955034
Modify Index = 2955865

nomad acl binding-rule info 8e1c94cc-5e20-3b1a-f580-d0d609dd87ae ; echo
ID           = 8e1c94cc-5e20-3b1a-f580-d0d609dd87ae
Description  = This is a rule for avtdevopssupport group
Auth Method  = vault
Selector     = "not avtdevops in list.roles or avtdevops in list.roles or avtdevopssupport in list.roles"
Bind Type    = role
Bind Name    = avtdevops
Create Time  = 2023-03-06 15:30:32.431627167 +0000 UTC
Modify Time  = 2023-03-07 10:31:48.049538823 +0000 UTC
Create Index = 2955689
Modify Index = 2971928

nomad acl auth-method info vault ; echo
Name         = vault
Type         = OIDC
Locality     = global
MaxTokenTTL  = 24h0m0s
Default      = true
Create Index = 2955160
Modify Index = 2956281

Auth Method Config

OIDC Discovery URL     =  ......
OIDC Client ID         = MVjNmtSdWZqapQcQoEUoipnygFgBjzkS
OIDC Client Secret     = ....
OIDC Scopes            = groups
Bound audiences        = MVjNmtSdWZqapQcQoEUoipnygFgBjzkS
Allowed redirects URIs = http://localhost:4649/oidc/callback,http://localhost:4646/ui/settings/tokens,.....
Discovery CA pem       = <none>
Signing algorithms     = <none>
Claim mappings         = <none>
List claim mappings    = {groups: roles}

As I guess I understand it as the following: the OIDC returns the “List claim mappings” which are then binding-rule mapped to a rule which binds the mapping to multiple policies.

In Vault my user is a member of avtdevops and avt groups. But anyway I setup the selector not avtdevops in list.roles or avtdevops in list.roles that it should be just always true for any value of list.roles. Is there a documentation what can be put in the selector and what does list.roles represent? How could I debug it?

What is http://localhost:4649/oidc/callback ? There is nothing on port 4649, should something be listening there?

The command on the end of manual nomad login -type=OIDC -method=OIDC-vault does not work - there is no OIDC-vault auth-method, I think that is documentation error. If I type nomad login -type=OIDC -method=vault the command just block infinitely! Strace only shows endless calls to sched_yield().

After inspecting with chrome with developer tools, I can see that the role of the token is correct, but there are no policies.

$ NOMAD_TOKEN=... nomad operator api /v1/acl/token/self | jq
{
  "ExpirationTTL": "24h0m0s",
  "AccessorID": "6b469026-45c0-d748-7ae7-dee121b57ade",
  "SecretID": "...",
  "Name": "OIDC-vault",
  "Type": "client",
  "Policies": [],
  "Roles": [
    {
      "ID": "2c3b87b5-5712-b1fd-e931-0fa2232a0b4d",
      "Name": "avtdevops"
    }
  ],
  "Global": true,
  "Hash": "...",
  "CreateTime": "2023-03-07T11:29:00.510819178Z",
  "ExpirationTime": "2023-03-08T11:29:00.510819178Z",
  "CreateIndex": 2972521,
  "ModifyIndex": 2972521
}
2

Hi @Kamilcuk,

Thanks for the feedback. SSO is a complex topic and we certainly want to improve the UX and docs here, so you have less hassle in the future.

How could I debug it?

I have opened #16360 to track adding a debug toggle for the SSO login flow which would expose data for debugging. This needs a little discussion to ensure the implementation is correct, but this is something we certainly want to add and have discussed previously.

What is http://localhost:4649/oidc/callback ? There is nothing on port 4649, should something be listening there?

This is the temporary server used for OIDC callback when running the nomad login command. This can be altered when running the command using the -oidc-callback-addr flag.

The command on the end of manual nomad login -type=OIDC -method=OIDC-vault does not work

Thanks, I’ll get this fixed up and released!

After inspecting with chrome with developer tools, I can see that the role of the token is correct, but there are no policies.

That is to be expected as the binding rule ties to a role, not an ACL policy. When creating the token, we do not unpack the policies linked to the role. This happens on each API request and allows operators to modify the policies linked via a role, without needing the regenerate tokens. The token will respect the update on the next API request.

Thanks,
jrasell and the Nomad team

1 Like
3

Hi! Thank you for your answer!

This is the temporary server used for OIDC callback when running the nomad login command

I can’t unpack this in my head. I have to do some reading on OIDC documentation to understand this.

The token will respect the update on the next API request.

That’s amazing! Ok, now I get it, thank you for clarification. The token indeed works and allows for scheduling a new job from terminal. The policy has all the permissions:

$ nomad acl policy info avtdevops
Name        = avtdevops
Description = For avtdevops people
CreateIndex = 2955005
ModifyIndex = 2973889

Rules


namespace "*" {
  policy = "write"
}

node {
  policy = "write"
}

agent {
  policy = "write"
}

operator {
  policy = "write"
}

plugin {
  policy = "list"
}

host_volume "*" {
  policy = "write"
}

If I update the policy, and change for example namespace "*" { policy = "write" to deny, the change is reflected in the UI after F5 and I no longer see any jobs. Like you described.

However, I now believe there is just an issue with UI buttons. The /ui/jobs “Run Job” button on right top corner is greyed out with a message of “no permission”. The “Drain” and “Eligible” buttons in /ui/clients/uuid are greyed out with “no permissions” message. I tested the token in terminal, and it has the permission to drain a node and start a job. The buttons to start/stop/purge existing jobs from UI are not greyed out and work.

I created When using token with role with proper permissions, UI buttons are disabled · Issue #16361 · hashicorp/nomad · GitHub .

1 Like