Creating an ACL token with "-secret" option breaks the consul security?

[neo3matrix]

Hi All,
Although I have been working with consul for a while now, I am new to consul ACL concepts and deployments.
Recently, I encountered a strange behavior while using the “consul acl token create” command with the “-secret=” option via CLI. After executing this command, I noticed that the external commands (like curl) as well as UI no longer prompt for authentication, and anyone browsing to the consul URL http://consul-ip:8500 OR doing any HTTP request via curl can directly access key/values, nodes, and services.
Additionally, using the “curl http://consul-ip:8500/v1/kv/(path-to-key-value)” command allows users to easily fetch any key-value without specifying any token.

This behavior seems counterintuitive and defeats the purpose of creating and enabling ACLs in the first place. However, I could be wrong and would appreciate any corrections.

I have tested this issue on two versions of Consul, v1.10.1 and the latest version as of today, and the behavior is the same on both.

To reproduce this issue, I followed these steps:

1. Deployed a docker-based Consul cluster on 3 nodes without enabling ACL.
2. Enabled ACL support in Consul using the following configuration:

acl = {
enabled = true
default_policy = “allow”
enable_token_persistence = true
tokens {
default = “my-custom-token”
}
}

3. Restarted all server containers.

4. In both the CLI and UI, Consul now prompts for an ACL token to log in.

5. Ran consul acl bootstrap and copied the bootstrap secretID.

6. Logged in to the UI or CLI using the secretID copied in step 5.

7. Created a new ACL token using the consul acl token create command with the “-secret” flag, as follows:

consul acl token create \
-description “cluster-wide access” \
-policy-id 00000000-0000-0000-0000-000000000001 \
-secret=(my-custom-token) \
-token=(bootstrap SecretID)

8. After executing step 7, the new token is successfully created with a secretID assigned with the provided secret, as expected.
9. However, after executing step 7, if I go back to the UI or CLI (even after logging out of the Consul UI), I no longer need to log in with any token. Any curl command can easily be successful without passing any token. I can fetch key/value either via UI or curl without any tokens.

This behavior suggests that the “-secret” flag breaks all authentication/ACL logic.

In contrast,

• if I don’t pass the “-secret” flag in the consul acl token create command in step#7, Consul creates a token with a new secretID assigned by Consul. The UI and outside CLI commands like curl still require an ACL token (either secretIDs), which is the expected behavior.
• Similarly, if I try to create a token from the UI, it also creates a Consul-generated secretID for a new token, and the system still expects me to log in with either secretIDs, or the curl command fails if I don’t provide any ACL tokens. Only step 7 seems to be causing this issue.

I have searched for any issues related to the “-secret” flag on Consul GitHub and community but couldn’t find any. Have any of you faced this issue? Can someone please help me?

[blake]

There are two things contributing to the behavior you’re seeing. First is the configuration of this parameter in your agent config.

acl {
  tokens {
    default = "my-custom-token"
  }
}

Per the docs on Consul’s HTTP authentication.

When authentication is enabled, a Consul token should be provided to API requests using the X-Consul-Token header or with the Bearer scheme in the Authorization header.…
If no token is provided for an HTTP request then Consul will use the default ACL token if it has been configured.

The reason the cluster is accessible without authentication is due to the policy you are assigning to that default token.

@neo3matrix

consul acl token create
-description “cluster-wide access”
-policy-id 00000000-0000-0000-0000-000000000001
-secret=(my-custom-token)
-token=(bootstrap SecretID)

You’re assigning this token the global management policy which has permission to read/write anything in Consul.

If your intent in configuring acl.tokens.default was to provide a token to be used solely by the agent, you must remove that config and instead specify the agent token under the acl.tokens.agent config option.

I would also recommend using a token with more limited privileges, such as one created by consul acl token create -node-identity=<node_name>:<datacenter>. See https://developer.hashicorp.com/consul/docs/security/acl#node-identities for more info on node identities.

[neo3matrix]

Thank you so much @Blake for quick reply.

I have some follow up comments below. I am still not able to completely understand the behavior of default token in your explanation due to following experience(s).

1. Me setting default token as part of config (acl.tokens.default)
   It doesn’t work for me at all!
   What I observe in sequence in my setup is:

• Consul starts with this config where default token is set.
• I can’t login to UI with default token OR run a successful curl command with default token in its X-Consul-Token header. consul cluster doesn’t recognize this token at all.
• Then, I have to run consul acl bootstrap - which generates a bootstrap token.
• When I run, consul acl token list -token=<bootstrap-token>, I don’t see default token listed in the token list (my-custom-token is not there under any name).
  Overall, it’s kind of worthless to specify default token in the config.

The reason the cluster is accessible without authentication is due to the policy you are assigning to that default token.

So, I think, the above statement might not be true in my case considering above scenario - as per my thought.
The cluster is accessible without authentication when I create a new token (not default token) with “-secret” option. (Q: Does this new token automatically becomes default token? )
Please correct me if I am wrong.

2. You’re assigning this token the global management policy which has permission to read/write anything in Consul.

Yes, I need my new token to have a cluster-wide access just like bootstrap token AND (just like bootstrap token), consul cluster should ask for authentcation for UI login OR curl command where I can use this token. Without this token, UI login or curl should fail (except for bootstrap token , of course) .

Even the bootstrap token also works for me - but I need a fixed token string which I can use across multiple restarts or disaster recoveries. Bootstrap is generated on the fly. I need to feed my own own fixed UUID.
In fact, I tried acls.tokens.master in my config - it eliminates the need of consul acl bootstrap but again leaves the cluster open for everyone without authentication).

3. I tried using acl.tokens.agent too but again no luck with that. Same experience like acl.tokens.default (my point#1 above).

I know for sure, I am missing something here. What is that?
Could you please help me?

Forgot to mention: The whole point of doing this is: In our security scans, we are getting a High severity vulnerability for Hashicorp Consul Web UI and API access - as it can be accessed unauthenticated (the way I have set it up).

NOTE: My final goal is simple. Create a fixed string token with cluster-wide access (for my application to create/modify key-value & services) and it should not break the default authentication requirement of consul ACL system.

[neo3matrix]

Sorry for my little understanding of ACLs. Getting confused with ACLs.

Another question along with my previous reply:
I tried following config too:
Instead of passing acl.tokens.default, if I just use:

acl = {
  enabled = true
  default_policy = "allow"
  enable_token_persistence = true
}

I don’t get any authorization from consul. Everything works unauthorized.
Even if I run consul acl bootstrap, still no authorization is required.

What requires here? As per the documentation, acl.enable need to be set to true.

Also, When I pass following in config file:

tokens {
    default = "my-custom-token"
  }

the system starts asking for authorization.

What’s the correct way to make system ask for authorization without using default token?

[jkirschner-hashicorp]

If the default policy is “allow”, every operation will be authorized (unless the request provides an ACL token with a matching rule that would override the default “allow” policy disposition).

If you want all requests to require authorization:

• Change the default_policy to “deny”
• Don’t provide a default token. (Or, provide a default token with only the privileges you want available to unauthenticated callers.)

[jkirschner-hashicorp]

That said, if you set the default_policy to “deny”, you’ll want to do the following as well:

• Specify acl.tokens.agent with a token that has a node identity for that agent. If the agent is named “client-1” in datacenter “dc1”, you could do this with consul acl token create -node-identity 'client-1:dc1' -description 'client-1 agent token'. (More details here)
• If you want to use Consul DNS to lookup services, the acl.tokens.default token will need suitable permissions for DNS lookups: Secure Consul with Access Control Lists (ACLs) | Consul | HashiCorp Developer

[neo3matrix]

Thank you @jkirschner-hashicorp for your valuable comments. I think it might have solved my issue. (I will let you know in couple of days).

Earlier I was skeptical about using "deny" policy but after your explanation I gave it a try and it exactly behaves the way I need it to - asking authentication for every step.

I have setup ACLs based on some of your suggestions. It’s little different than what you have suggested. Could you please look at my below ACL setup and let me know if I am doing it right? Are there any issues with this kind of setup where I might not be able to perform certain operations?
(Apart from the fact that my custom token will have cluster-wide access).

My setup:
3 hosts used for consul cluster.(H1, H2, H3)
I am using docker containers for my setup.

On every host, I have 1 container of consul server, 1 container of consul Client and 1 container of vault server running. (Vault using consul as storage backend).
So, 3 consul hosts, 3 consul clients & 3 Vault servers - this how my cluster is.

ACLs:
I used following common config between all consul servers and clients:

acl = {
  enabled = true
  default_policy = "deny"
  enable_token_persistence = true
  tokens = {
    agent = "my-hardcoded-UUID-token"
  }
}

For every vault server, the consul part of config file has been added with line:

token = “my-hardcoded-UUID-token”

Steps to setup ACLs:

1. On one of the consul server, ran the command: consul acl bootstrap
2. Created a new token with cluster-wide access

consul acl token create \
-description “my-global token” \
-policy-id 00000000-0000-0000-0000-000000000001 \
-secret my-hardcoded-UUID-token \
-token=(bootstrap-token)

3. On every consul server & consul client container, ran command:

consul acl set-agent-token agent “my-hardcoded-UUID-token”

That’s it!

I did not use -node-identity thinking that instead of creating a token for every consul agent (server or client), let’s just have one common agent token with cluster-wide access between all of them - for simplicity purpose.

Please let me know if this makes sense?

So, far my kv retrievals and health checks work fine. Vault access also works fine.
I will test service registry feature in couple of days.