Vault-agent-init error authenticating - certificate signed by unknown authority

I have my vault cluster setup within Kubernetes via the vault-helm chart.

vault v1.4.2
EKS v1.15

I followed the instructions listed here to configure TLS - https://www.vaultproject.io/docs/platform/k8s/helm/examples/standalone-tls

server configs:

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true

      listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

      storage "dynamodb" {
        ha_enabled = "true"
        read_capacity  = "10"
        write_capacity = "15"
        max_parallel = "128"
        table = "<table_name>"
      }

      seal "awskms" {
        region = "us-east-1"
        kms_key_id = <kms_arn>
      }

      service_registration "kubernetes" {}

I am receiving authentication from agent-init when I deploy my test service.

$ kubectl logs -f centos-deployment-77c6f9d7f4-6tpgm -c vault-agent-init

2020-06-05T19:39:17.779Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=1.354056995
2020-06-05T19:39:19.133Z [INFO]  auth.handler: authenticating
2020-06-05T19:39:19.140Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=1.551085796
2020-06-05T19:39:20.691Z [INFO]  auth.handler: authenticating
2020-06-05T19:39:20.698Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=1.879938257
2020-06-05T19:39:22.578Z [INFO]  auth.handler: authenticating
2020-06-05T19:39:22.585Z [ERROR] auth.handler: error authenticating: error="Put https://vault.vault.svc:8200/v1/auth/kubernetes/login: x509: certificate signed by unknown authority" backoff=2.511108609

This is my injector configuration in my values file for helm, which I had left default. According to the docs, it says if left default, it would generate its own internal self-signed cert.

injector:
  certs:
    secretName: null
    caBundle: ""
    certName: tls.crt
    keyName: tls.key

So I attempted to generate my own self-signed cert to pass to the agent-injector based on this article - https://medium.com/@cogarius/a-vault-for-all-your-secrets-full-tls-on-kubernetes-with-kv-v2-c0ecd42853e1

However, the vault-agent-injector pod fails to start up

$ kubectl logs -f vault-agent-injector-58744958bd-wz9lj -n vault
{"@level":"info","@message":"Starting handler..","@module":"handler","@timestamp":"2020-06-05T00:41:38.886922Z"}
Listening on ":8080"...
Updated certificate bundle received. Updating certs...
2020/06/05 00:42:18 http: TLS handshake error from 172.21.61.2:34578: remote error: tls: bad certificate
2020/06/05 00:42:35 http: TLS handshake error from 172.21.61.2:34712: remote error: tls: bad certificate
2020/06/05 00:42:53 http: TLS handshake error from 172.21.61.2:34886: remote error: tls: bad certificate
2020/06/05 00:43:55 http: TLS handshake error from 172.21.61.2:35462: remote error: tls: bad certificate
2020/06/05 00:43:57 http: TLS handshake error from 172.21.61.2:35468: remote error: tls: bad certificate
2020/06/05 00:43:59 http: TLS handshake error from 172.21.61.2:35482: remote error: tls: bad certificate

I’ve reverted this change, and back to the original injector helm defaults. with certificate signed by unknown authority error

What am I doing incorrectly?

Thanks

I also followed these instructions to enable kubernetes auth method, but still run into the certificate signed by unknown authority issue

Figured this one out,

Added the following annotation to my service deployment yaml configuration

vault.hashicorp.com/ca-cert: "/run/secrets/kubernetes.io/serviceaccount/ca.crt"
1 Like

Hey @popopanda, I believe I am now facing the same issue as you.

Can you tell me how you figure out to keep TLS for servers, but keep agent injector with his “self-generated certificate” ?

I tried that annotation as well on the containers
vault.hashicorp.com/ca-cert” = “/run/secrets/kubernetes.io/serviceaccount/ca.crt”

But it doesn’t help ?

thanks !

Hey @JnMik,

For my server TLS configuration, followed the instruction at vault example standalone TLS.

defined the following for my vault-server config in the helm values file:

  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca

  config: |
        listener "tcp" {
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
      }

I left the injector section as default, so it would use the self-generated cert.

I believe for my ca cert, I have to use /run/secrets/kubernetes.io/serviceaccount/ca.crt in my app deployment because I enabled kubernetes auth by

# Ran inside Vault pod
$ vault write auth/kubernetes/config \
        token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
        kubernetes_host="https://kubernetes.default:443" \
        kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

This was my test example “app” pod I deployed

kind: ServiceAccount
apiVersion: v1
metadata:
  name: centos
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: centos-deployment
  labels:
    app: centos
spec:
  replicas: 3
  selector:
    matchLabels:
      app: centos
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "m2_centos_role"
        vault.hashicorp.com/ca-cert: "/run/secrets/kubernetes.io/serviceaccount/ca.crt"
        vault.hashicorp.com/agent-inject-secret-foo.txt: "kv/devops/centos/aws_creds"
        vault.hashicorp.com/agent-inject-template-foo.txt: |
          {{- with secret "kv/devops/centos/aws_creds" -}}
          secretKeyfoo: {{ .Data.data.foo }}
          secretkeybaz: {{ .Data.data.baz }}
          {{- end -}}
      labels:
        app: centos
    spec:
      serviceAccountName: centos
      containers:
      - name: centos
        image: centos
        command: ["/bin/sh"]
        args: ["-c", "sleep 1000000"]

---

I also want to note that in Vault documentation in their injector examples page,

They used the following annotations for the app deployment

        vault.hashicorp.com/ca-cert: "/vault/tls/ca.crt"
        vault.hashicorp.com/client-cert: "/vault/tls/client.crt"
        vault.hashicorp.com/client-key: "/vault/tls/client.key"
        vault.hashicorp.com/tls-secret: "vault-tls-client"

These values didnt work for me, as /vault/tls directory inside the pod does not exist. Though in that example, it looks like it’s expecting a vault-tls-client secret in my namespace, which I did not want to manage the same secret across multiple namespaces.

hey thanks for the quick reply !
I have the same exact use case as you, but I could’nt make it work with the
vault.hashicorp.com/ca-cert annotation.

I had to copy my TLS secret in all namespaces where I have applications, and I finally used
vault.hashicorp.com/tls-secret = “vault-tls”.
vault.hashicorp.com/ca-cert = “/vault/tls/ca.crt”

I think it will be a viable solution has my certificate is self-signed and won’t expired or anything (so I won’t have to replicate a change in the secret to all other namespaces)

At least I have a working setup with TLS !

But that means any actor that has access to the key, which is in multiple namespaces, can decrypt all traffic to and from vault server?