WMP
March 31, 2022, 2:03pm
1
Hello, i want to get some passwords from external vault to pod in k8s 1.21. I want to authorized to vault by approle. In vault i do:
vault write auth/approle/role/kub-test secret_id_ttl=1440m token_num_uses=0 token_ttl=1440m token_max_ttl=0 secret_id_num_uses=0
vault write -force auth/approle/role/kub-test/secret-id
Key Value
--- -----
secret_id 9318XXX-xxx-xxxx-xxxx-xxxxxd1b8
secret_id_accessor 9d81xxxx-xxxx-xxxxx-xxxx-xxxxfa502
vault write auth/approle/role/kub-test token_policies="kub-test"
vault read auth/approle/role/kub-test/role-id
Key Value
--- -----
role_id 003dxxx-xxx-xxx-xxx-xxxxxxxxb01
Then, in kubernetes i install helm upgrade --install vault --namespace vault-csi --values ../vault.values.yaml vault-k8s with values:
injector:
# True if you want to enable vault agent injection.
enabled: true
server:
# True if you want to enable vault agent injection.
enabled: false
ui:
# True if you want to enable vault agent injection.
enabled: false
csi:
enabled: false
.
Next, i create my deployment:
apiVersion: v1
data:
secret-id: #base64 with secret-id from vault write -force auth/approle/role/kub-test/secret-id
role-id: #base64 with role-id from vault read auth/approle/role/kub-test/role-id
kind: Secret
metadata:
name: approle-kub-test
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-proxy-deployment
labels:
app: nginx-proxy
spec:
replicas: 3
selector:
matchLabels:
app: nginx-proxy
template:
metadata:
labels:
app: nginx-proxy
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-vault-addr: 'https://vault.my.domain:8200'
vault.hashicorp.com/auth-type: 'approle'
vault.hashicorp.com/auth-path: 'auth/approle'
vault.hashicorp.com/auth-config-role-id-file-path: '/vault/custom/role-id'
vault.hashicorp.com/auth-config-secret-id-file-path: '/vault/custom/secret-id'
vault.hashicorp.com/agent-extra-secret: 'approle-kub-test'
vault.hashicorp.com/role: 'my-role'
vault.hashicorp.com/log-level: 'debug'
When i read logs from vault-agent-init i have errors:
$ kubectl logs nginx-proxy-deployment-6dd77cbfd7-rv4vj vault-agent-init
==> Vault agent started! Log data will stream in below:
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.9.2
Version Sha: f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf
2022-03-31T12:28:57.436Z [INFO] sink.file: creating file sink
2022-03-31T12:28:57.436Z [INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-03-31T12:28:57.437Z [INFO] template.server: starting template server
2022-03-31T12:28:57.437Z [INFO] template.server: no templates found
2022-03-31T12:28:57.437Z [INFO] auth.handler: starting auth handler
2022-03-31T12:28:57.437Z [INFO] auth.handler: authenticating
2022-03-31T12:28:57.437Z [INFO] sink.server: starting sink server
2022-03-31T12:28:57.437Z [ERROR] auth.approle: error removing secret ID file after reading: error="remove /vault/custom/secret-id: read-only file system"
How i should configure integration with external vault with approle authenication?
aram
April 1, 2022, 7:05am
2
Sorry this isn’t an answer to your question but you maybe making your life harder than it needs to be. The kub-auth method is the default option for this scenario and would integrate well.
As far as using approle, you would need a lot more debugging and logs to track that down.
maxb
April 1, 2022, 7:24am
3
It sounds like you want the remove_secret_id_file_after_reading = false option:
WMP
April 1, 2022, 7:37am
4
Thanks, now i have:
2022-04-01T07:32:18.035Z [INFO] sink.file: creating file sink
2022-04-01T07:32:18.035Z [INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-04-01T07:32:18.035Z [INFO] sink.server: starting sink server
2022-04-01T07:32:18.035Z [INFO] template.server: starting template server
2022-04-01T07:32:18.035Z [INFO] template.server: no templates found
2022-04-01T07:32:18.035Z [INFO] auth.handler: starting auth handler
2022-04-01T07:32:18.035Z [INFO] auth.handler: authenticating
2022-04-01T07:33:18.037Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2022-04-01T07:33:19.037Z [INFO] auth.handler: authenticating
2022-04-01T07:34:19.038Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1.74s
2022-04-01T07:34:20.783Z [INFO] auth.handler: authenticating
maxb
April 1, 2022, 7:43am
5
Hard to say from this what’s wrong - that’s just a generic timeout error. Perhaps the agent cannot reach the Vault server and either a network connect or read operation is timing out?
WMP
April 1, 2022, 9:48am
6
I found, vault dont read my annotation with vault server address, in vault-agent-init container in file /home/vault/config.json i have:
"vault": {
"address": "http://vault.vault-csi.svc:8200",
I trying annotations:
vault.hashicorp.com/agent-vault-addr: 'https://vault.my.domain:8200'
vault.hashicorp.com/address: 'https://vault.my.domain:8200'
vault.hashicorp.com/vault-address: 'https://vault.my.domain:8200'
but dont works. What is annotation to give vavult server? I cannot found this on:Agent Sidecar Injector Annotations | Vault | HashiCorp Developer , only:
Vault annotations change how the Vault Agent containers communicate with Vault. For example, Vault’s address, TLS certificates to use, client parameters such as timeouts, etc.
WMP
April 1, 2022, 10:25am
7
Ok, i fund that is imposible to set vault address bu annotation, there isn’t code to this in vault-k8s/annotations.go at main · hashicorp/vault-k8s · GitHub
Working code:
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-configmap
data:
config.hcl: |
"auto_auth" = {
"method" = {
"type" = "approle"
"mount_path" = "auth/approle"
"config" = {
"role" = "my-role"
"remove_secret_id_file_after_reading" = "false"
"role_id_file_path" = "/vault/custom/role-id"
"secret_id_file_path" = "/vault/custom/secret-id"
"token_path" = "/var/run/secrets/kubernetes.io/serviceaccount/token"
}
}
"sink" = {
"type" = "file"
"config" = {
"path" = "/home/vault/.vault-token"
}
}
}
"exit_after_auth" = true
"pid_file" = "/home/vault/.pid"
"vault" = {
"address" = "https://vault.my.domain:8200"
"ca_cert" = "/vault/tls/ca.crt"
}
config-init.hcl: |
"auto_auth" = {
"method" = {
"type" = "approle"
"mount_path" = "auth/approle"
"config" = {
"role" = "my-role"
"remove_secret_id_file_after_reading" = "false"
"role_id_file_path" = "/vault/custom/role-id"
"secret_id_file_path" = "/vault/custom/secret-id"
"token_path" = "/var/run/secrets/kubernetes.io/serviceaccount/token"
}
}
"sink" = {
"type" = "file"
"config" = {
"path" = "/home/vault/.vault-token"
}
}
}
"exit_after_auth" = true
"pid_file" = "/home/vault/.pid"
"vault" = {
"address" = "https://vault.my.domain:8200"
"ca_cert" = "/vault/tls/ca.crt"
}
---
apiVersion: v1
kind: Secret
metadata:
name: approle-kub-test
type: Opaque
data:
secret-id: #base64 with secret-id from vault write -force auth/approle/role/kub-test/secret-id
role-id: #base64 with role-id from vault read auth/approle/role/kub-test/role-id
---
apiVersion: v1
kind: Secret
metadata:
name: vault-tls
type: Opaque
data:
ca.crt: #base64 with my CA what sign https://vault.my.domain:8200
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-proxy-deployment
labels:
app: nginx-proxy
spec:
replicas: 3
selector:
matchLabels:
app: nginx-proxy
template:
metadata:
labels:
app: nginx-proxy
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-configmap: 'vault-configmap'
vault.hashicorp.com/tls-secret: 'vault-tls'
vault.hashicorp.com/agent-extra-secret: 'approle-kub-test'
Of course i need to write template to get secrets, but i havent errors in log:
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.238Z [INFO] template.server: no templates found
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.238Z [INFO] auth.handler: authenticating
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.266Z [INFO] auth.handler: authentication successful, sending token to sinks
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO] auth.handler: starting renewal process
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO] sink.file: token written: path=/home/vault/.vault-token
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO] sink.server: sink server stopped
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO] sinks finished, exiting
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO] template.server: template server stopped
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO] auth.handler: shutdown triggered, stopping lifetime watcher
nginx-proxy-deployment-6546b54554-gxzsl vault-agent 2022-04-01T10:18:57.267Z [INFO] auth.handler: auth handler stopped