Read data from vault with Vault Agent Sidecar Injector and approle

Hello, i want to get some passwords from external vault to pod in k8s 1.21. I want to authorized to vault by approle. In vault i do:

vault write auth/approle/role/kub-test secret_id_ttl=1440m token_num_uses=0 token_ttl=1440m token_max_ttl=0 secret_id_num_uses=0

vault write -force  auth/approle/role/kub-test/secret-id
Key                   Value
---                   -----
secret_id             9318XXX-xxx-xxxx-xxxx-xxxxxd1b8
secret_id_accessor    9d81xxxx-xxxx-xxxxx-xxxx-xxxxfa502

vault write auth/approle/role/kub-test token_policies="kub-test"

vault read auth/approle/role/kub-test/role-id
Key        Value
---        -----
role_id    003dxxx-xxx-xxx-xxx-xxxxxxxxb01

Then, in kubernetes i install helm upgrade --install vault --namespace vault-csi --values ../vault.values.yaml vault-k8s with values:

injector:
  # True if you want to enable vault agent injection.
  enabled: true

server:
  # True if you want to enable vault agent injection.
  enabled: false

ui:
  # True if you want to enable vault agent injection.
  enabled: false

csi:
  enabled: false

.

Next, i create my deployment:

apiVersion: v1
data:
  secret-id: #base64 with secret-id from vault write -force  auth/approle/role/kub-test/secret-id
  role-id: #base64 with role-id from vault read auth/approle/role/kub-test/role-id
kind: Secret
metadata:
  name: approle-kub-test
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-proxy-deployment
  labels:
    app: nginx-proxy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-proxy
  template:
    metadata:
      labels:
        app: nginx-proxy
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-vault-addr: 'https://vault.my.domain:8200'
        vault.hashicorp.com/auth-type: 'approle'
        vault.hashicorp.com/auth-path: 'auth/approle'
        vault.hashicorp.com/auth-config-role-id-file-path: '/vault/custom/role-id'
        vault.hashicorp.com/auth-config-secret-id-file-path: '/vault/custom/secret-id'
        vault.hashicorp.com/agent-extra-secret: 'approle-kub-test'
        vault.hashicorp.com/role: 'my-role'
        vault.hashicorp.com/log-level: 'debug'

When i read logs from vault-agent-init i have errors:

$ kubectl logs nginx-proxy-deployment-6dd77cbfd7-rv4vj vault-agent-init           
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.9.2
             Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf

2022-03-31T12:28:57.436Z [INFO]  sink.file: creating file sink
2022-03-31T12:28:57.436Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-03-31T12:28:57.437Z [INFO]  template.server: starting template server
2022-03-31T12:28:57.437Z [INFO]  template.server: no templates found
2022-03-31T12:28:57.437Z [INFO]  auth.handler: starting auth handler
2022-03-31T12:28:57.437Z [INFO]  auth.handler: authenticating
2022-03-31T12:28:57.437Z [INFO]  sink.server: starting sink server
2022-03-31T12:28:57.437Z [ERROR] auth.approle: error removing secret ID file after reading: error="remove /vault/custom/secret-id: read-only file system"

How i should configure integration with external vault with approle authenication?

Sorry this isn’t an answer to your question but you maybe making your life harder than it needs to be. The kub-auth method is the default option for this scenario and would integrate well.

As far as using approle, you would need a lot more debugging and logs to track that down.

It sounds like you want the remove_secret_id_file_after_reading = false option:

Thanks, now i have:

2022-04-01T07:32:18.035Z [INFO]  sink.file: creating file sink
2022-04-01T07:32:18.035Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-04-01T07:32:18.035Z [INFO]  sink.server: starting sink server
2022-04-01T07:32:18.035Z [INFO]  template.server: starting template server
2022-04-01T07:32:18.035Z [INFO]  template.server: no templates found
2022-04-01T07:32:18.035Z [INFO]  auth.handler: starting auth handler
2022-04-01T07:32:18.035Z [INFO]  auth.handler: authenticating
2022-04-01T07:33:18.037Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2022-04-01T07:33:19.037Z [INFO]  auth.handler: authenticating
2022-04-01T07:34:19.038Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.74s
2022-04-01T07:34:20.783Z [INFO]  auth.handler: authenticating

Hard to say from this what’s wrong - that’s just a generic timeout error. Perhaps the agent cannot reach the Vault server and either a network connect or read operation is timing out?

I found, vault dont read my annotation with vault server address, in vault-agent-init container in file /home/vault/config.json i have:

"vault": {
    "address": "http://vault.vault-csi.svc:8200",

I trying annotations:

vault.hashicorp.com/agent-vault-addr: 'https://vault.my.domain:8200'
vault.hashicorp.com/address: 'https://vault.my.domain:8200'
vault.hashicorp.com/vault-address: 'https://vault.my.domain:8200'

but dont works. What is annotation to give vavult server? I cannot found this on:
Agent Sidecar Injector Annotations | Vault by HashiCorp , only:

Vault annotations change how the Vault Agent containers communicate with Vault. For example, Vault’s address, TLS certificates to use, client parameters such as timeouts, etc.

Ok, i fund that is imposible to set vault address bu annotation, there isn’t code to this in vault-k8s/annotations.go at main · hashicorp/vault-k8s · GitHub

Working code:

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-configmap
data:
  config.hcl: |
    "auto_auth" = {
      "method" = {
        "type" = "approle"
        "mount_path" = "auth/approle"
        "config" = {
          "role" = "my-role"
          "remove_secret_id_file_after_reading" = "false"
          "role_id_file_path" = "/vault/custom/role-id"
          "secret_id_file_path" = "/vault/custom/secret-id"
          "token_path" = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        }
      }
      "sink" = {
        "type" = "file"
        "config" = {
          "path" = "/home/vault/.vault-token"
        }
      }
    }

    "exit_after_auth" = true
    "pid_file" = "/home/vault/.pid"

    "vault" = {
      "address" = "https://vault.my.domain:8200"
      "ca_cert" = "/vault/tls/ca.crt"
    }
  config-init.hcl: |
    "auto_auth" = {
      "method" = {
        "type" = "approle"
        "mount_path" = "auth/approle"
        "config" = {
          "role" = "my-role"
          "remove_secret_id_file_after_reading" = "false"
          "role_id_file_path" = "/vault/custom/role-id"
          "secret_id_file_path" = "/vault/custom/secret-id"
          "token_path" = "/var/run/secrets/kubernetes.io/serviceaccount/token"
        }
      }
      "sink" = {
        "type" = "file"
        "config" = {
          "path" = "/home/vault/.vault-token"
        }
      }
    }

    "exit_after_auth" = true
    "pid_file" = "/home/vault/.pid"

    "vault" = {
      "address" = "https://vault.my.domain:8200"
      "ca_cert" = "/vault/tls/ca.crt"
    }
---
apiVersion: v1
kind: Secret
metadata:
  name: approle-kub-test
type: Opaque
data:
  secret-id: #base64 with secret-id from vault write -force  auth/approle/role/kub-test/secret-id
  role-id: #base64 with role-id from vault read auth/approle/role/kub-test/role-id
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-tls
type: Opaque
data:
  ca.crt: #base64 with my CA what sign https://vault.my.domain:8200
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-proxy-deployment
  labels:
    app: nginx-proxy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-proxy
  template:
    metadata:
      labels:
        app: nginx-proxy
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-configmap: 'vault-configmap'
        vault.hashicorp.com/tls-secret: 'vault-tls'
        vault.hashicorp.com/agent-extra-secret: 'approle-kub-test'

Of course i need to write template to get secrets, but i havent errors in log:

nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.238Z [INFO]  template.server: no templates found
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.238Z [INFO]  auth.handler: authenticating
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.266Z [INFO]  auth.handler: authentication successful, sending token to sinks
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO]  auth.handler: starting renewal process
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO]  sink.file: token written: path=/home/vault/.vault-token
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO]  sink.server: sink server stopped
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO]  sinks finished, exiting
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO]  template.server: template server stopped
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO]  auth.handler: shutdown triggered, stopping lifetime watcher
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO]  auth.handler: auth handler stopped