Hi there,
We currently use vault-injector via helm chart in our EKS setup.
We have external vault setup on separate EKS cluster.
We have create new EKS clusters for both workloads and vault to migrate them from old K8s EKS cluster which is long EOL.
New EKS cluster has vault-injector and pods doing dynamic injection. They are able to connect to old EKS vault cluster and able to authenticate and use vault correctly.
We are in process of migrating vault to new EKS cluster. We cloned postgres database and use same EKS seal key as old cluster and created new test vault cluster using helm chart and it works as expected. We don’t see any issues.
Now, when I try to goto new EKS cluster and change vault injector helm external vault address to new vault test address, vault injector doesn’t complain.
But when I try to spin up a pod which uses vault sidecar for vault injector, it complains as following.
vault agent sidecar logs:
2024-03-28T15:15:28.678Z [INFO] agent.exec.server: starting exec server
2024-03-28T15:15:28.678Z [INFO] agent.exec.server: no env templates or exec config, exiting
2024-03-28T15:15:28.678Z [INFO] agent.auth.handler: starting auth handler
2024-03-28T15:15:28.678Z [INFO] agent.template.server: starting template server
2024-03-28T15:15:28.678Z [INFO] (runner) creating new runner (dry: false, once: false)
2024-03-28T15:15:28.678Z [INFO] agent.auth.handler: authenticating
2024-03-28T15:15:28.678Z [INFO] agent.sink.server: starting sink server
2024-03-28T15:15:28.678Z [INFO] (runner) creating watcher
2024-03-28T15:15:28.681Z [ERROR] agent.auth.handler: error authenticating: error="json: cannot unmarshal number into Go value of type api.Secret" backoff=1s
2024-03-28T15:15:29.681Z [INFO] agent.auth.handler: authenticating
2024-03-28T15:15:29.684Z [ERROR] agent.auth.handler: error authenticating: error="json: cannot unmarshal number into Go value of type api.Secret" backoff=1.97s
2024-03-28T15:15:31.658Z [INFO] agent.auth.handler: authenticating
2024-03-28T15:15:31.663Z [ERROR] agent.auth.handler: error authenticating: error="json: cannot unmarshal number into Go value of type api.Secret" backoff=3.78s
2024-03-28T15:15:35.446Z [INFO] agent.auth.handler: authenticating
2024-03-28T15:15:35.449Z [ERROR] agent.auth.handler: error authenticating: error="json: cannot unmarshal number into Go value of type api.Secret" backoff=7.36s
vault injector helm vaulues.yaml on new EKS cluster:
vault:
server:
dev:
enabled: true
injector:
enabled: true
logLevel: "info"
logFormat: "standard"
replicas: 3
webhook:
matchPolicy: Equivalent
global:
#externalVaultAddr: "https://vault.xxxxx.com"
externalVaultAddr: "https://argo-vault.xxxxxx.com"
New vault helm values.yaml on new EKS cluster:
vault:
injector:
enabled: false
server:
affinity: null
standalone:
enabled: false
ha:
enabled: true
config: |
ui = true
disable_mlock = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "postgresql" {
connection_url="postgres://xxxxxxx:xxxxxxx@xxxxxxx.xxxxxx.us-west-2.rds.amazonaws.com:5432/vault",
ha_enabled=true
}
service_registration "kubernetes" {}
seal "awskms" {
region = "us-west-2"
kms_key_id = "xxxxxxxxxx"
}
I tried to downgrade vault helm chart to v0.20.0 but that didn’t help. Currently both vault and vault-injector are using helm chart v0.28.0.
Do I need to make any additional changes apart from changing externalVaultAddr value in vault-injector?
I suppose jwt tokens for k8s authentication shall still be cloned and available from old vault since new vault-injector EKS cluster is already talking to it.