Vault-agent-init error authenticating - certificate signed by unknown authority

Vault
1

I have my vault cluster setup within Kubernetes via the vault-helm chart.

vault v1.4.2
EKS v1.15

I followed the instructions listed here to configure TLS - https://www.vaultproject.io/docs/platform/k8s/helm/examples/standalone-tls

server configs:

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true

      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

      storage "dynamodb" {
        ha_enabled = "true"
        read_capacity  = "10"
        write_capacity = "15"
        max_parallel = "128"
        table = "<table_name>"
      }

      seal "awskms" {
        region = "us-east-1"
        kms_key_id = <kms_arn>
      }

      service_registration "kubernetes" {}

I am receiving authentication from agent-init when I deploy my test service.

$ kubectl logs -f centos-deployment-77c6f9d7f4-6tpgm -c vault-agent-init

2020-06-05T19:39:17.779Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=1.354056995
2020-06-05T19:39:19.133Z [INFO]  auth.handler: authenticating
2020-06-05T19:39:19.140Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=1.551085796
2020-06-05T19:39:20.691Z [INFO]  auth.handler: authenticating
2020-06-05T19:39:20.698Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=1.879938257
2020-06-05T19:39:22.578Z [INFO]  auth.handler: authenticating
2020-06-05T19:39:22.585Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=2.511108609

This is my injector configuration in my values file for helm, which I had left default. According to the docs, it says if left default, it would generate its own internal self-signed cert.

injector:
  certs:
    secretName: null
    caBundle: ""
    certName: tls.crt
    keyName: tls.key

So I attempted to generate my own self-signed cert to pass to the agent-injector based on this article - https://medium.com/@cogarius/a-vault-for-all-your-secrets-full-tls-on-kubernetes-with-kv-v2-c0ecd42853e1

However, the vault-agent-injector pod fails to start up

$ kubectl logs -f vault-agent-injector-58744958bd-wz9lj -n vault
{"@level":"info","@message":"Starting handler..","@module":"handler","@timestamp":"2020-06-05T00:41:38.886922Z"}
Listening on ":8080"...
Updated certificate bundle received. Updating certs...
2020/06/05 00:42:18 http: TLS handshake error from 172.21.61.2:34578: remote error: tls: bad certificate
2020/06/05 00:42:35 http: TLS handshake error from 172.21.61.2:34712: remote error: tls: bad certificate
2020/06/05 00:42:53 http: TLS handshake error from 172.21.61.2:34886: remote error: tls: bad certificate
2020/06/05 00:43:55 http: TLS handshake error from 172.21.61.2:35462: remote error: tls: bad certificate
2020/06/05 00:43:57 http: TLS handshake error from 172.21.61.2:35468: remote error: tls: bad certificate
2020/06/05 00:43:59 http: TLS handshake error from 172.21.61.2:35482: remote error: tls: bad certificate

I’ve reverted this change, and back to the original injector helm defaults. with certificate signed by unknown authority error

What am I doing incorrectly?

Thanks

2

I also followed these instructions to enable kubernetes auth method, but still run into the certificate signed by unknown authority issue

3

Figured this one out,

Added the following annotation to my service deployment yaml configuration

vault.hashicorp.com/ca-cert: "/run/secrets/kubernetes.io/serviceaccount/ca.crt"
3 Likes
4

Hey @popopanda, I believe I am now facing the same issue as you.

Can you tell me how you figure out to keep TLS for servers, but keep agent injector with his “self-generated certificate” ?

I tried that annotation as well on the containers
“vault.hashicorp.com/ca-cert” = “/run/secrets/kubernetes.io/serviceaccount/ca.crt”

But it doesn’t help ?

thanks !

5

Hey @JnMik,

For my server TLS configuration, followed the instruction at vault example standalone TLS.

defined the following for my vault-server config in the helm values file:

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  config: |
        listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

I left the injector section as default, so it would use the self-generated cert.

I believe for my ca cert, I have to use /run/secrets/kubernetes.io/serviceaccount/ca.crt in my app deployment because I enabled kubernetes auth by

# Ran inside Vault pod
$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://kubernetes.default:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

This was my test example “app” pod I deployed

kind: ServiceAccount
apiVersion: v1
metadata:
  name: centos
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: centos-deployment
  labels:
    app: centos
spec:
  replicas: 3
  selector:
    matchLabels:
      app: centos
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "m2_centos_role"
        vault.hashicorp.com/ca-cert: "/run/secrets/kubernetes.io/serviceaccount/ca.crt"
        vault.hashicorp.com/agent-inject-secret-foo.txt: "kv/devops/centos/aws_creds"
        vault.hashicorp.com/agent-inject-template-foo.txt: |
          {{- with secret "kv/devops/centos/aws_creds" -}}
          secretKeyfoo: {{ .Data.data.foo }}
          secretkeybaz: {{ .Data.data.baz }}
          {{- end -}}
      labels:
        app: centos
    spec:
      serviceAccountName: centos
      containers:
      - name: centos
        image: centos
        command: ["/bin/sh"]
        args: ["-c", "sleep 1000000"]

---
6

I also want to note that in Vault documentation in their injector examples page,

They used the following annotations for the app deployment

        vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
        vault.hashicorp.com/client-cert: "/vault/tls/client.crt"
        vault.hashicorp.com/client-key: "/vault/tls/client.key"
        vault.hashicorp.com/tls-secret: "vault-tls-client"

These values didnt work for me, as /vault/tls directory inside the pod does not exist. Though in that example, it looks like it’s expecting a vault-tls-client secret in my namespace, which I did not want to manage the same secret across multiple namespaces.

7

hey thanks for the quick reply !
I have the same exact use case as you, but I could’nt make it work with the
vault.hashicorp.com/ca-cert annotation.

I had to copy my TLS secret in all namespaces where I have applications, and I finally used
vault.hashicorp.com/tls-secret = “vault-tls”.
vault.hashicorp.com/ca-cert = “/vault/tls/ca.crt”

I think it will be a viable solution has my certificate is self-signed and won’t expired or anything (so I won’t have to replicate a change in the secret to all other namespaces)

At least I have a working setup with TLS !

8

But that means any actor that has access to the key, which is in multiple namespaces, can decrypt all traffic to and from vault server?

1 Like
9

I’m having some similar issues as described here, but haven’t had any luck with any of the suggestions.

What appears to be happening is that I configure vault to use a letsencrypt certificate. So, I can only reference vault using a public DNS name (because LE will not let me add additional private URLs such as those associated with kubernetes services), which means that any service attempting to connect to Vault needs to connect to the public DNS name, including the agent-injector, since vault.vault.svc is not in the tls certificate I associated with the vault application in the values.yaml, the ones that get addressed in the config “tls_cert_file” and “tls_key_file”.

Now, the agent injector gets injected into a pod in some random namespace where I’ve a role configured in vault to accept the service-account and namespace I’m using. The agent sidecar then will login to the vault service, but it ALWAYS connects to the internal service vault.vault.svc, even though the presented certificate is specifically for the public DNS. Even if I configure the annotations to reference vault.hashicorp.com/tls-server-name: “Public DNS URL”, I still get an error “Put “https://vault.vault.svc:8200/blahblahblah”: x509: certificate is valid for Public DNS URL”.

Am I missing something in this back and forth?

Also, if I want another kubernetes cluster to be able to login to this Vault instance, do I need to create another auth/kubernetes/config for this different agent? Otherwise, how does it distinguish from one clusters allowed namespace/service account, if they overlap? In other words, how do I enable multiple kubernetes clusters to talk to a single vault instance.

Thanks all!

10

My case may not be appliable to you, but I think a “dumb” cert secret can be used here. As the injector only interested in the ca.crt in that secret.

11

This works for me! Thanks!

13

vault.hashicorp.com/tls-skip-verify: "true"

14

For me the /vault/tls directory appeared in my application pod right after i added these annotations into my application deployment manifest:

vault.hashicorp.com/tls-secret: “rootca”
vault.hashicorp.com/ca-cert: “/vault/tls/tls.crt”

And I guess yes, it does expect tls-secret in all namespaces.

And my tls.crt (Root Certificate) appeared inside this directory as well.

I wasn’t able to launch my application, the init container is getting “permission denied” from vault, but that’s a different story.