Team,
I have created a KV2 store to put Certificate and private key. I am able to access it via vault cli and API. But if I add the annotation to inject the certificate and private key. I am getting error to boot the container with error as “vault.read(internal/data/application/key): no secret exists at internal/data/application/key” in the vault-injector-init container. I tried both KV v1 and v2. Kindly guide what to check to know if I am missing some config. I am following the sidecar injection procedure as mentioned in Injecting Secrets into Kubernetes Pods via Vault Agent Containers | Vault | HashiCorp Developer
It is working for database username and password store in KV V2 but not able to initiate the pod as I save the cert and key in pem format.
manojitdas@Manojit Das ~ % vault kv get app-certs/certs
==== Secret Path ====
app-certs/data/certs
======= Metadata =======
Key Value
created_time 2023-04-04T08:47:19.358578032Z
custom_metadata
deletion_time n/a
destroyed false
version 1
======== Data ========
Key Value
certificates -----BEGIN CERTIFICATE-----
MIID8TCCAtmgAwIBAgIUQHkyWYZ2xWdGPEbeRdzIQB0+IYcwDQYJKoZIhvcNAQEL
BQAwgYcxCzAJBgNVBAYTAklOMRIwEAYDVQQIDAlLYXJuYXRha2ExEjAQBgNVBAcM
CUJhbmdhbG9yZTENMAsGA1UECgwEUkJJSDEOMAwGA1UECwwFTG9hbnMxDTALBgNV
BAMMBHJiaWgxIjAgBgkqhkiG9w0BCQEWE3NhdGlzaG1AcmJpaG1haWwuaW4wHhcN
MjMwNDAzMDQzMTQyWhcNMjYwNDAyMDQzMTQyWjCBhzELMAkGA1UEBhMCSU4xEjAQ
BgNVBAgMCUthcm5hdGFrYTESMBAGA1UEBwwJQmFuZ2Fsb3JlMQ0wCwYDVQQKDARS
QklIMQ4wDAYDVQQLDAVMb2FuczENMAsGA1UEAwwEcmJpaDEiMCAGCSqGSIb3DQEJ
ARYTc2F0aXNobUByYmlobWFpbC5pbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAMNbcZV/z8TaDCVg4rmgUb7prvfLD9SiGoNJGlYqew8tagrGyLaG1qhN
PiQGsjOz9qnZJ6WlnvioYYpPZ41ozZFBGEmQTxvim435gPIh1nBPM3F7bc9mNqhH
XtbOJwD/OoWaBT3RAEYXMqh7YtfZD2SHgXt2jp9V7zYqoOeYx6752YfLoBhW4DnR
COTxOeAkUn//NLKE4SB7AqBZ/vDV3Hdlbu+HU2zCQwF2ivl4UIez2+KQpiRTtROz
BgeCGJTZDI9GyDOBHjspPCs4n6EmKhwT/3/7jLl4I/KPtd/SSRg+B7UM2BUBiko7
gja37vl3Zn/v3Uthr/7cAmDLCFDzXFECAwEAAaNTMFEwHQYDVR0OBBYEFCBOfrQF
h0uww8CfsPqeUXMl/UHHMB8GA1UdIwQYMBaAFCBOfrQFh0uww8CfsPqeUXMl/UHH
MA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBACwImJIQCdRqil8Q
hmG0KN5O5UKiw3LZyKwLNLEgaeOEGti45HPqBVZB6SV9oNvQi0/UQMvbpTTolTKE
PQe2vPY5J67kIJ5DgJvXghGNKGkGUKp+F5DfVi3eYdNNkQ185gUl49ipIfnqnsCp
uq1hz/W5og9nr6JmMCWAh3Xcj/y0R2u1Gq1Dr4HzJUR2r34GoEiT5egXX++fhaPH
Q/vcEZa5VH8EYwmTNFG/NP+EpCFABk9pad8k/ytvLM4ACt7CghKCU1wJMaUXQURd
0IpEPzgY0j7YffI+BV88P1ODwNeLBSzsLiGgdbaTCcUTE6uYGQEc+kuI2TwB9AeR
P1chza4=
-----END CERTIFICATE-----
privatekeys -----BEGIN PRIVATE KEY-----
MIIEuwIBADANBgkqhkiG9w0BAQEFAASCBKUwggShAgEAAoIBAQDDW3GVf8/E2gwl
YOK5oFG+6a73yw/UohqDSRpWKnsPLWoKxsi2htaoTT4kBrIzs/ap2SelpZ74qGGK
T2eNaM2RQRhJkE8b4puN+YDyIdZwTzNxe23PZjaoR17WzicA/zqFmgU90QBGFzKo
e2LX2Q9kh4F7do6fVe82KqDnmMeu+dmHy6AYVuA50Qjk8TngJFJ//zSyhOEgewKg
Wf7w1dx3ZW7vh1NswkMBdor5eFCHs9vikKYkU7UTswYHghiU2QyPRsgzgR47KTwr
OJ+hJiocE/9/+4y5eCPyj7Xf0kkYPge1DNgVAYpKO4I2t+75d2Z/791LYa/+3AJg
ywhQ81xRAgMBAAECgf9wFKa1/v5VEkSlrVBm73P29FtOD1Kly9sqN2bEyNqRepy8
UhAVqIa/SJG+WZkj0LxT9iKpBryOLQzJLYVbvCU0kQQ+MG0ysnBMpDk2sfM1vDrn
gMkflKUHJutikiQOjmAlBpi7Po3JqMWHpEKxU/dg1d2T2BE7mTIFAbAPL7hgyjcw
j0pc+sKEOthZrPJdLc+hENOc9QFPzthmEgmQPmv+rPYMYosZe2086jGeFRdZ4l1o
/euApDaY9lp4cTOW/9q6+cbp7iS9B6jHn0BKI9e8QqLwSOJtV6I+4kkdBZNu7/Ek
6UoKRtEaDtpGHTiLoS82ChFsRdmV1NQBKugW6LkCgYEA2cr3sgp1TJp/g6SlOwBb
R5WYkN8DVXAdDNnA0saii1JUFRa9px8kpbEC9QdJq7fBlOUfMcXGJCHXOmUBoI1B
vQNlW4xR6QsqrjknZnoeOJZT9mmQ2sQ8vr3vznbPfvEPDoMrUpYp5aEr/v5oWoQw
PufTlLKojHM1QsDApKa3xwkCgYEA5aDlZO3RrOcEMvk8AkqKpNmfJV3L2hcXv9Gj
hiFuYs/tepKu7sgD+ABVM9jdA3wbu3lxwDLfHC813L72sQ5uOuI8e3Fxxt41M2ys
uvDAr7lsmIlfdM1m3RmT3dQapMHZkSC0+UU1k+DWEt96EvtP55l2pljyLnfx2Pdq
MKuktQkCgYAO8qFBdPpMAaaS0HAOtp2NMjVAbC38ZS95PFDBpqRtT38520+08qw/
OwDq9E8YHAqLP5Z+ggZxYGk3Nst/y3IhmOQR/nV4uy14u696lxMP6zFkDibE/Qb2
SG73Cqx/F+hipJoKlwfVHNS6MZfT7pKpHpwk7oYm7ZlQNY6DPnYKkQKBgBj1ItmU
a4yiUzeDFT2egnEmYgC6sBRVHDSyAVbrvUaqGMKQvMihHqVOy/PlpyfgzlbS/UUJ
gmxayTzj2IYn7l0iB7bW/qnBrwCW2DM9RYQT4DS4AZL9XCe9xu9+RSMRGqeGaYUK
O6eF8V59wZPckrO2X0n3MuNQEA4n3ffKQA2ZAoGBAKv3QitnFdKSX11B5iFW2wNP
Gicx/pguv0wKOC6b6JJrQYB19GZKs+C0wMn6enkTcy4CjgCZKqkOgSiK2Oa4zPV6
hoBZHbF73bslgdxr8ykYaKAxZ8E9Rv7rWQ/fNOk6Iv40YiPZG1VflXfNXJrJvSt9
AYVQPCL/sf5kdAY03YyI
-----END PRIVATE KEY-----
manojitdas@Manojit Das ~ %
manojitdas@Manojit Das ~ % vault policy read app-certs
path “app-certs/data/certs/*” {
capabilities = [“read”]
}
vault read auth/kubernetes/role/app-certs
Key Value
alias_name_source serviceaccount_uid
bound_service_account_names [app-certs]
bound_service_account_namespaces [default]
policies [app-certs]
token_bound_cidrs
token_explicit_max_ttl 0s
token_max_ttl 0s
token_no_default_policy false
token_num_uses 0
token_period 0s
token_policies [app-certs]
token_ttl 24h
token_type default
ttl 24h
I am able access the cert and key using access token of the policy but it is not working with the service account associated
The agent error shows the internal/data/application/key path but your example shows app-certs/data/certs
Sorry for the confusion. I just put the sample details. I have tested with various key path. It shows the error for each path.
Let me summarise what is working:
- I have a KV store for database password and a vault role and vault policy associated to a service account and it is working
- I created another KV store for API keys and a vault role and vault policy and created a token of the policy and it is working with vault API commands
what is not working:
for the second use case if I use the service account and the vault injector annotation the vault init injector pod is showing error to read the file path.
Then it would be helpful to know:
- Is it all running on the same cluster (Vault, service 1 and service 2)?
- Vault annotations for service 1 and 2
- Service account names and namespaces for 1 and 2
- the Kubernetes auth backend config
- Vault server logs from the time the error happened on the client
I store the value as below:
vault kv put internal/application certificates=@/Users/manojitdas/Downloads/rbihcert.pem privatekeys=@/Users/manojitdas/Downloads/rbihprivate.pem
I could fetch in the information with access token as
curl --header “X-Vault-Token: hvs.CAESID7hLUv-di2w3cLuOA32RE-1iu4G-dkoan_c29wvgO1LGh4KHGh2cy5VTG45YnYyY1hnaDhZSGU0SFVsNURramM” http://vault:8200/v1/internal/data/application
But the annotation of below is not working:
spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: ‘true’
vault.hashicorp.com/role: ‘app-sa’
vault.hashicorp.com/agent-inject-secret-application-config.txt: ‘internal/data/application’
The error in the pod showing as:
2023-04-04T12:14:06.867Z [INFO] (runner) creating watcher
2023-04-04T12:14:06.867Z [INFO] (runner) starting
2023-04-04T12:14:06.869Z [INFO] auth.handler: renewed auth token
2023-04-04T12:14:06.878Z [WARN] (view) vault.read(internal/data/application): no secret exists at internal/data/application (retry attempt 1 after “250ms”)
2023-04-04T12:14:07.131Z [WARN] (view) vault.read(internal/data/application): no secret exists at internal/data/application (retry attempt 2 after “500ms”)
2023-04-04T12:14:07.635Z [WARN] (view) vault.read(internal/data/application): no secret exists at internal/data/application (retry attempt 3 after “1s”)
2023-04-04T12:14:08.639Z [WARN] (view) vault.read(internal/data/application): no secret exists at internal/data/application (retry attempt 4 after “2s”)
2023-04-04T12:14:10.642Z [WARN] (view) vault.read(internal/data/application): no secret exists at internal/data/application (retry attempt 5 after “4s”)
2023-04-04T12:14:14.645Z [WARN] (view) vault.read(internal/data/application): no secret exists at internal/data/application (retry attempt 6 after “8s”)
2023-04-04T12:14:22.648Z [WARN] (view) vault.read(internal/data/application): no secret exists at internal/data/application (retry attempt 7 after “16s”)
Hope you can follow the steps.
The role you created is app-certs not app-sa
Not really. I don’t know if what you’re posting now refers to use-case 1 or use-case 2
I have multiple KV store and associated service account …for this discussion we can use “app-sa” service account/vault role and policy.
Use case 1 is for Database password in KV and use case 2 for api key in KV store
And you did this from the same Pod where the agent was injected?
from a different pod in the cluster. The pod is not coming up after the injection.
What I mean is, in the same namespace and using the same service account
yes same namespace… it is using access token key of the same vault policy to read the path.
Key Value
disable_iss_validation true
disable_local_ca_jwt false
issuer n/a
kubernetes_ca_cert n/a
kubernetes_host https://10.1.0.1:443
pem_keys
vault server logs
2023-04-04T01:08:39.732Z [ERROR] secrets.system.system_89da98bb: error occurred during enable credential: path=kubernetes/ error=“path is already in use at kubernetes/”
2023-04-04T01:22:02.564Z [ERROR] secrets.system.system_89da98bb: error occurred during enable mount: path=internal/ error=“path is already in use at internal/”
2023-04-04T01:23:32.472Z [ERROR] secrets.system.system_89da98bb: error occurred during enable credential: path=kubernetes/ error=“path is already in use at kubernetes/”
2023-04-04T06:58:24.891Z [INFO] core: successful mount: namespace=“” path=my-kv-store/ type=kv version=“”
2023-04-04T08:43:21.007Z [INFO] core: successful mount: namespace=“” path=app-certs/ type=kv version=“”
2023-04-04T08:43:21.016Z [INFO] secrets.kv.kv_957402f9: collecting keys to upgrade
2023-04-04T08:43:21.016Z [INFO] secrets.kv.kv_957402f9: done collecting keys: num_keys=1
2023-04-04T08:43:21.016Z [INFO] secrets.kv.kv_957402f9: upgrading keys finished
2023-04-04T09:25:58.286Z [ERROR] secrets.system.system_89da98bb: error occurred during enable credential: path=kubernetes/ error=“path is already in use at kubernetes/”
2023-04-04T11:43:03.705Z [INFO] expiration: revoked lease: lease_id=internal/data/application/h06e030610bc28ce4a63f0373a5766c92fdd3029d8b2b2beda87a63759a6a4f38
Which one is being used now?
I am using v2 now. v1 also does not worked.