FinServ Proof of Concept - need to enable ACLs on k8s consul image

Consul
1

Hi,

I’m trying to set up a “reachback” proof of concept; e.g., a k8s consul environment being able to connect microservices in the cloud to an on-prem, traditional database.

I’m working my way through the setup, but need to enable ACLs on the k8s images - and probably make a number of other config changes - in order to proceed through the steps. The filesystem is overlay2 and read-only - is there an elegant/supported way to do this?

2

Hi @joshua-burns,

Thanks for giving Consul a spin! This tutorial can help you with setting up ACLs as well as other security features when running Consul in K8s. Take a look, and let me know if this gets you where you need to go.

Cheers!

3

Thanks, will have a look

4

I attempted to do a helm upgrade using that .yaml statements described there, and got the same error:

Error: UPGRADE FAILED: timed out waiting for the condition

and “consul info” shows ACL support is still disabled… any ideas?

5

Can you show us the output of kubectl get pod?

7 
NAME                                                          READY   STATUS      RESTARTS   AGE

consul-connect-injector-webhook-deployment-785cc55456-gsfhg 1/1 Running 0 4d3h
consul-mesh-gateway-964bdfc75-sq8p4 2/2 Running 0 4d3h
consul-r595r 1/1 Running 0 4d3h
consul-server-0 1/1 Running 0 4d3h
consul-test 0/1 Completed 0 4d3h
[miles][08:28 PM][~]$ helm list --all
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
consul default 2 2020-08-31 16:24:08.571608414 -0400 EDT failed consul-0.24.1 1.8.2

8

Hi, I think you may be running into this issue (https://github.com/hashicorp/consul-helm/issues/579) where you cannot turn on ACLs and run a helm upgrade.

If this is a demo cluster, I’d recommend running helm delete and kubectl delete on the pvc's and then re-installing.

If you cannot delete the cluster, you need to restart your servers to enable ACLs by running

 kubectl rollout restart statefulset/consul-server

Then the servers will have ACLs enabled and should be able to run helm upgrade and have it succeed.

9

Hi, thanks for the tips - unfortunately, I tried both re-installing from scratch and running the rollout command, and “consul info” on consul-server-0 still shows that ACLs are disabled.

Any thoughts?

10

Hmm, this works for me:

# config.yaml
global:
  name: consul
  acls:
    manageSystemACLs: true

helm install consul hashicorp/consul -f config.yaml --wait
ACL=$(kubectl get secrets/consul-bootstrap-acl-token --template={{.data.token}} | base64 -D)
kubectl exec statefulset/consul-server -- consul info -token $ACL | grep -A 2 -B 2 acl

outputs:

	version = 1.8.2
consul:
	acl = enabled
	bootstrap = true
	known_datacenters = 1
11

helm-install-debug-v2.txt (2.3 MB)

Hi,

I tried upgrading, re-installing and re-starting the container and ACLs are still disabled - I turned on debugging (attached) during the last install; any insights on what the issue might be?

Note: Helm is pretty particular about versioning; 3.3.0 failed repeatedly; 3.3.1 now “works” (as in helm no longer reports that the install failed).

12

Hi,

I’ve been looking through your logs. No update so far, but I wanted to let you know we’re still looking into this.

13

Hi, do you have acls nested under global?

global:
  acls:
    manageSystemACLs: true

I believe you may have it as a top-level key instead in which case it won’t be taking effect:

global:
  ...
acls:
  # will not work, must be under global
14

Thanks - I’ve re-installed, nesting the “acls” section, but am now getting “Permission denied” (code 403) when I attempt to run “consul info” to verify the status of ACLs. Debug info attached.

helm-install-debug-v3.txt (1.4 MB)

15

Hi, that’s expected because you now need to use an ACL token depending on the data you’re accessing.

See https://www.consul.io/docs/k8s/installation/overview#acls-enabled for how you can get the bootstrap token.

You can then pass this in to Consul commands with the -token flag, e.g. consul info -token e7924dd1-dc3f-f644-da54-81a73ba0a178