Summary
I have successfully installed a stand-alone instance of Vault on AWS EKS 1.17; however, when deploying a test deployment to validate the deployment can get a secret from Vault, the vault-agent-init never completes. Below are the details of the Vault config. It seems like some networking issue, possibly getting to the AWS EKS k8s API end-point or maybe some IAM role required, but no luck trying a variety of configurations. Any ideas or assistance would be greatly appreciated.
Environment
- AWS EKS 1.17 created using eksctl
- 2 public subnets for the K8s workers
- Vault
- installed using HELM in a
vaultnamespace - successfully unseal the vault using
awskms - primary followed these instructions to configure vault and the k8s auth method
- override values below
- installed using HELM in a
injector:
metrics:
enabled: true
logLevel: "debug"
server:
standalone:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
}
storage "file" {
path = "/vault/data"
}
seal "awskms" {
region = "us-east-1"
kms_key_id = "<removed>"
}
log_level = "debug"
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
extraVolumes:
- type: secret
name: vault-server-tls
extraSecretEnvironmentVars:
- envName: AWS_ACCESS_KEY_ID
secretName: eks-creds
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: eks-creds
secretKey: AWS_SECRET_ACCESS_KEY
dataStorage:
enabled: true
mountPath: "/vault/data"
size: 50Gi
storageClass: null
accessMode: ReadWriteOnce
auditStorage:
enabled: true
mountPath: "/vault/audit"
size: 10Gi
storageClass: null
accessMode: ReadWriteOnce
service:
enabled: true
ui:
enabled: true
serviceType: LoadBalancer
- Vault K8s auth
- kubernetes_ca_cert and kubernetes_host - obtained from local .kube/config
- role
vault write auth/kubernetes/role/app-user \
bound_service_account_names=vault-auth \
bound_service_account_namespaces=* \
policies=app-policy \
ttl=24h
- service account / cluster role binding
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-auth
namespace: vault
---
apiVersion: v1
kind: Secret
metadata:
name: vault-auth
namespace: vault
annotations:
kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: vault
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-auth
namespace: vault
Logs
- vault agent injector logs
2020-10-08T16:14:36.893Z [INFO] handler: Starting handler..
Registering telemetry path on "/metrics"
Listening on ":8080"...
Updated certificate bundle received. Updating certs...
2020-10-08T16:16:05.459Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2020-10-08T16:16:05.462Z [DEBUG] handler: checking if should inject agent..
2020-10-08T16:16:05.462Z [DEBUG] handler: checking namespaces..
2020-10-08T16:16:05.462Z [DEBUG] handler: setting default annotations..
2020-10-08T16:16:05.462Z [DEBUG] handler: creating new agent..
2020-10-08T16:16:05.462Z [DEBUG] handler: validating agent configuration..
2020-10-08T16:16:05.462Z [DEBUG] handler: creating patches for the pod..
Sample Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "app-user"
vault.hashicorp.com/agent-inject-secret-poc-secret: "secrets/dev/poc-secret"
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
- from kubectl describe po
Name: nginx-d5c74dcdc-cm9l6
Namespace: default
Priority: 0
Node: ip-192-168-39-123.ec2.internal/192.168.39.123
Start Time: Thu, 08 Oct 2020 11:16:05 -0500
Labels: app=nginx
pod-template-hash=d5c74dcdc
Annotations: kubernetes.io/psp: eks.privileged
vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-inject-secret-poc-secret: secrets/dev/poc-secret
vault.hashicorp.com/agent-inject-status: injected
vault.hashicorp.com/role: app-user
Status: Pending
IP: 192.168.51.37
Controlled By: ReplicaSet/nginx-d5c74dcdc
Init Containers:
vault-agent-init:
Container ID: docker://4fca294ebc2dd14f81997b29b4ef78cbf09c67c9738f63d97c0c64e78c027b71
Image: vault:1.5.2
Image ID: docker-pullable://vault@sha256:9aa46d9d9987562013bfadce166570e1705de619c9ae543be7c61953f3229923
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Running
Started: Thu, 08 Oct 2020 11:16:06 -0500
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
VAULT_LOG_LEVEL: info
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiYXBwLXVzZXIifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjp0cnVlLCJwaWRfZmlsZSI6Ii9ob21lL3ZhdWx0Ly5waWQiLCJ2YXVsdCI6eyJhZGRyZXNzIjoiaHR0cDovL3ZhdWx0LnZhdWx0LnN2Yzo4MjAwIn0sInRlbXBsYXRlIjpbeyJkZXN0aW5hdGlvbiI6Ii92YXVsdC9zZWNyZXRzL3BvYy1zZWNyZXQiLCJjb250ZW50cyI6Int7IHdpdGggc2VjcmV0IFwic2VjcmV0cy9kZXYvcG9jLXNlY3JldFwiIH19e3sgcmFuZ2UgJGssICR2IDo9IC5EYXRhIH19e3sgJGsgfX06IHt7ICR2IH19XG57eyBlbmQgfX17eyBlbmQgfX0iLCJsZWZ0X2RlbGltaXRlciI6Int7IiwicmlnaHRfZGVsaW1pdGVyIjoifX0ifV19
Mounts:
/home/vault from home-init (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4xkch (ro)
/vault/secrets from vault-secrets (rw)
Containers:
nginx:
Container ID:
Image: nginx:1.14.2
Image ID:
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4xkch (ro)
/vault/secrets from vault-secrets (rw)
vault-agent:
Container ID:
Image: vault:1.5.2
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
VAULT_LOG_LEVEL: info
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiYXBwLXVzZXIifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjpmYWxzZSwicGlkX2ZpbGUiOiIvaG9tZS92YXVsdC8ucGlkIiwidmF1bHQiOnsiYWRkcmVzcyI6Imh0dHA6Ly92YXVsdC52YXVsdC5zdmM6ODIwMCJ9LCJ0ZW1wbGF0ZSI6W3siZGVzdGluYXRpb24iOiIvdmF1bHQvc2VjcmV0cy9wb2Mtc2VjcmV0IiwiY29udGVudHMiOiJ7eyB3aXRoIHNlY3JldCBcInNlY3JldHMvZGV2L3BvYy1zZWNyZXRcIiB9fXt7IHJhbmdlICRrLCAkdiA6PSAuRGF0YSB9fXt7ICRrIH19OiB7eyAkdiB9fVxue3sgZW5kIH19e3sgZW5kIH19IiwibGVmdF9kZWxpbWl0ZXIiOiJ7eyIsInJpZ2h0X2RlbGltaXRlciI6In19In1dfQ==
Mounts:
/home/vault from home-sidecar (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4xkch (ro)
/vault/secrets from vault-secrets (rw)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-4xkch:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4xkch
Optional: false
home-init:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
home-sidecar:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
vault-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/nginx-d5c74dcdc-cm9l6 to ip-192-168-39-123.ec2.internal
Normal Pulled 22m kubelet, ip-192-168-39-123.ec2.internal Container image "vault:1.5.2" already present on machine
Normal Created 22m kubelet, ip-192-168-39-123.ec2.internal Created container vault-agent-init
Normal Started 22m kubelet, ip-192-168-39-123.ec2.internal Started container vault-agent-init