Consul-Kubernetes

Consul
1

Hi,

I need your expert opinion. Should I pass the “acl token” as an environment variable inside the pods?

I have a consul agent server (1.7.1) running outside the k8s cluster (on-perm). ACL is enabled on the consul cluster.

Installed the consul client (1.7.1) using helm.

global:
enabled: false
datacenter: mdc
gossipEncryption:
secretName: “consul-gossip-encryption-key”
secretKey: “key”
client:
enabled: true
grpc: true
exposeGossipPorts: true
join:
- “consul01”
- “consul02”
- “consul03”
extraConfig: |
{
“log_level”: “DEBUG”
}
connectInject:
enabled: true
default: false
syncCatalog:
enabled: true
default: false
toConsul: true
toK8S: false
k8sTag: “k8s-test”
aclSyncToken:
secretName: “consul-acl-token-agent”
secretKey: “key”

Installed a sample application available online.

apiVersion: v1
kind: ServiceAccount
metadata:
name: static-server

apiVersion: v1
kind: Pod
metadata:
name: static-server
annotations:
“consul.hashicorp.com/connect-inject”: “true”
spec:
containers:
# This name will be the service name in Consul.
- name: static-server
image: hashicorp/http-echo:latest
args:
- -text=“hello world”
- -listen=:8080
ports:
- containerPort: 8080
name: http

If ACLs are enabled, the serviceAccountName must match the Consul service name.

serviceAccountName: static-server

apiVersion: v1
kind: ServiceAccount
metadata:
name: static-client

apiVersion: v1
kind: Pod
metadata:
name: static-client
annotations:
“consul.hashicorp.com/connect-inject”: “true”
“consul.hashicorp.com/connect-service-upstreams”: “static-server:1234”
spec:
containers:
# This name will be the service name in Consul.
- name: static-client
image: tutum/curl:latest
# Just spin & wait forever, we’ll use kubectl exec to demo
command: [ “/bin/sh”, “-c”, “–” ]
args: [ “while true; do sleep 30; done;” ]

If ACLs are enabled, the serviceAccountName must match the Consul service name.

serviceAccountName: static-client

But the services are not able to communicate.

kubectl exec static-client – curl -s http://127.0.0.1:1234/
Defaulting container name to static-client.
Use ‘kubectl describe pod/static-client -n default’ to see all of the containers in this pod.
command terminated with exit code 7

I have the following two policy for each of the services

service “static-client” {
policy = “write”
}
service “static-client-sidecar-proxy” {
policy = “write”
}
service_prefix “” {
policy = “read”
}
node_prefix “” {
policy = “read”
}

service “static-server” {
policy = “write”
}
service “static-server-sidecar-proxy” {
policy = “write”
}
service_prefix “” {
policy = “read”
}
node_prefix “” {
policy = “read”
}

Thanks,
AA

2

Hey @antonyaugustus,

Thanks for creating this topic and welcome to the discussion forum!

You don’t need to manually pass tokens for Connect. The connect injector uses the Consul’s kubernetes auth method to retrieve individual service tokens before the container starts up. What you need to do is the following:

  1. Create a kubernetes service account to be used for the auth method. You could take a look at how we’re doing it in the Helm chart:
    ServiceAccount
    ClusterRole
    ClusterRoleBinding
  2. Create an ACL auth method to be used by the injector. For example, using the Consul CLI, here is the command you could use (you can get the CA cert and token contents from the service account secret created for the service account):
     consul acl auth-method create -type "kubernetes" \
          -name "my-k8s" \
          -description "This is an example kube method" \
          -kubernetes-host "https://<kubernetes service cluster IP>:443" \
          -kubernetes-ca-file /path/to/kube.ca.crt \
          -kubernetes-service-account-jwt "<jwt token contents of the service account from step 1>"
  3. Set the auth method name connectInject.overrideAuthMethodName as a value in the Helm chart and re-install. This should instruct the connect injector to now use this auth method to get the service token for each service that uses sidecar injection.

Hope this helps!

4

Thank you @ishustava. I was able to move forward. I can see successfully login and token getting assigned.

I can see the services on the consul UI. Even with full write policy on service, I am unable to get the services integrate with each other. Any pointers will help.

service “” {
policy = “write”
}

Thanks
AA

5

Hey @antonyaugustus, could you give a bit more detail? What do you mean by not being able to integrate services? Any error messages, steps you followed would be helpful in understanding the problem you’re seeing.

Thanks!

6

@ishustava

I have Kubernetes AuthMethod Type with the JWT for Kubernetes Service Account - hashicorp-consul-connect-injector-authmethod-svc-account. This SA is in consul-system namespace and so are the consul agent daemonsets and connect-injector pods. While the test demo pods are running on “antony-test” namespace.

Policy Definition is pretty open for all service and service_prefix.
service “” {
policy = “write”
intentions = “write”
}
service_prefix “” {
policy = “write”
intentions = “write”
}
service_prefix “static-server” {
policy = “write”
}
service_prefix “static-client” {
policy = “write”
}

Bound the policy/role to AuthMethod with different selectors. I tried different combination but I assume there are better ways.

./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘antony-test-ns-role’ -selector=‘serviceaccount.namespace==“antony-test”’
./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘antony-test-ns-role’ -selector=‘serviceaccount.namespace==“consul-system”’
./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘antony-test-ns-role’ -selector=‘serviceaccount.name==“static-server”’
./consul acl binding-rule create -method=auth-method-consul-test -bind-type=role -bind-name=‘antony-test-ns-role’ -selector=‘serviceaccount.name==“static-client”’

Below are some of the logs from different containers in static-client pod

kubectl logs -n antony-test static-client consul-connect-inject-init
Registered service: static-client-sidecar-proxy
Registered service: static-client

Here is the result from trying to access the service from inside the client pod

root@static-client:/# curl -v http://127.0.0.1:1234

  • Rebuilt URL to: http://127.0.0.1:1234/
  • Hostname was NOT found in DNS cache
  • Trying 127.0.0.1…
  • Connected to 127.0.0.1 (127.0.0.1) port 1234 (#0)

GET / HTTP/1.1
User-Agent: curl/7.35.0
Host: 127.0.0.1:1234
Accept: /

  • Empty reply from server
  • Connection #0 to host 127.0.0.1 left intact
    curl: (52) Empty reply from server

It looks like the static server is listening

kubectl logs -n antony-test static-server static-server
2020/03/26 18:19:51 Server is listening on :8080

Thank you again for you assistance.

AA

7

@ishustava

I believe I got it working. My acl.default_policy=deny which was causing the service to deny by default. Please correct me if I’m wrong.

" The default intention behavior is defined by the default ACL policy. If the default ACL policy is “allow all”, then all Connect connections are allowed by default. If the default ACL policy is “deny all”, then all Connect connections are denied by default."

Thanks,
AA

8

Hey @antonyaugustus,

I’m glad you got it working and apologies for the delay.

Yes, default deny setting for ACLs will also mean default deny when it comes to intentions. This isn’t super clear in Consul when you look in the UI, for example, and I definitely think it’s confusing.

What you need to do is create an intention allowing traffic from static-client to static-server:

consul intention create static-client static-server -allow

You could also accomplish that by changing the default ACL policy, although I would not recommend that from a security standpoint.