Hi,
I have deployed hashicorp vault in openshift cluster using the official helm chart provided by the hashicorp. I have enabled vault server and vault injector from the values yaml file. The vault server statefulset is running fine and the vault-injector deployment also up and running.
When I am trying to inject the secrets in the sample application deployed in the same openshift cluster. The vault is failing to inject the init container to the application deployment. I have used the below annotation to inject the secrets into the deployment.
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/agent-inject-secret-login: "secret/login"
vault.hashicorp.com/agent-inject-template-login: |
{{- with secret "secret/login" -}}
pattoken={{ .Data.data.pattoken }}
{{- end }}
vault.hashicorp.com/agent-inject-secret-my-first-secret: "secret/my-first-secret"
vault.hashicorp.com/agent-inject-template-my-first-secret: |
{{- with secret "secret/my-first-secret" -}}
username={{ .Data.data.username }}
password={{ .Data.data.password }}
{{- end }}
vault.hashicorp.com/role: "vault-role"
The blocking issue that I am getting is from the mutation webhook. Below is the logs from the openshift kube-api server.
W0802 13:03:26.436354 16 dispatcher.go:208] Context canceled when calling webhook vault.hashicorp.com E0802 13:03:26.436374 16 dispatcher.go:214] failed calling webhook "vault.hashicorp.com": failed to call webhook: Post "[https://vault-agent-injector-svc.vault-injector.svc:443/mutate?timeout=30s](https://vault-agent-injector-svc.vault-injector.svc/mutate?timeout=30s)": context canceled`.
Also when I check the events in the deployment of the sample app. I got the error `Error creating: Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s
I have also check the ca certificate of the vault injector. There is no issue with respect to certificate.
Appreciate your help.
Thanks
Hi @manso502,
Interesting error you got here. I cannot reproduce the error since I’m not in front of a computer right now, but do you mind sharing your yaml/value files used for the deployment on OpenShift?
The annotations you have configured look totally legit.
Best,
Andreas
Hi Andreas,
Sorry for the delay in response. This issue still persist. I am pasting here the injector config.
vault:
global:
# enabled is the master enabled switch. Setting this to true or false
# will enable or disable all the components within this chart by default.
enabled: true
namespace: ""
imagePullSecrets:
- name: docker-secret
# TLS for end-to-end encrypted transport
tlsDisable: false
# Setting this will disable deployment of a vault server.
externalVaultAddr: ""
# If deploying to OpenShift
openshift: true
injector:
# True if you want to enable vault agent injection.
# @default: global.enabled
enabled: true
replicas: 1
# Configures the port the injector should listen on
port: 8080
# If multiple replicas are specified, by default a leader will be determined
# so that only one injector attempts to create TLS certificates.
leaderElector:
enabled: true
# If true, will enable a node exporter metrics endpoint at /metrics.
metrics:
enabled: false
# Deprecated: Please use global.externalVaultAddr instead.
externalVaultAddr: ""
# image sets the repo and tag of the vault-k8s image to use for the injector.
image:
repository: "private-registry"
tag: "1.4.2"
pullPolicy: IfNotPresent
# agentImage sets the repo and tag of the Vault image to use for the Vault Agent
# containers. This should be set to the official Vault image. Vault 1.3.1+ is
# required.
agentImage:
repository: "private-registry"
tag: "1.17.2"
# The default values for the injected Vault Agent containers.
agentDefaults:
# For more information on configuring resources, see the K8s documentation:
# https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
cpuLimit: "500m"
cpuRequest: "250m"
memLimit: "128Mi"
memRequest: "64Mi"
# ephemeralLimit: "128Mi"
# ephemeralRequest: "64Mi"
# Default template type for secrets when no custom template is specified.
# Possible values include: "json" and "map".
template: "map"
# Default values within Agent's template_config stanza.
templateConfig:
exitOnRetryFailure: true
staticSecretRenderInterval: ""
# Used to define custom livenessProbe settings
livenessProbe:
# When a probe fails, Kubernetes will try failureThreshold times before giving up
failureThreshold: 2
# Number of seconds after the container has started before probe initiates
initialDelaySeconds: 5
# How often (in seconds) to perform the probe
periodSeconds: 2
# Minimum consecutive successes for the probe to be considered successful after having failed
successThreshold: 1
# Number of seconds after which the probe times out.
timeoutSeconds: 5
# Used to define custom readinessProbe settings
readinessProbe:
# When a probe fails, Kubernetes will try failureThreshold times before giving up
failureThreshold: 2
# Number of seconds after the container has started before probe initiates
initialDelaySeconds: 5
# How often (in seconds) to perform the probe
periodSeconds: 2
# Minimum consecutive successes for the probe to be considered successful after having failed
successThreshold: 1
# Number of seconds after which the probe times out.
timeoutSeconds: 5
# Used to define custom startupProbe settings
startupProbe:
# When a probe fails, Kubernetes will try failureThreshold times before giving up
failureThreshold: 12
# Number of seconds after the container has started before probe initiates
initialDelaySeconds: 5
# How often (in seconds) to perform the probe
periodSeconds: 5
# Minimum consecutive successes for the probe to be considered successful after having failed
successThreshold: 1
# Number of seconds after which the probe times out.
timeoutSeconds: 5
# Mount Path of the Vault Kubernetes Auth Method.
authPath: "auth/kubernetes"
# Configures the log verbosity of the injector.
# Supported log levels include: trace, debug, info, warn, error
logLevel: "trace"
# Configures the log format of the injector. Supported log formats: "standard", "json".
logFormat: "standard"
# Configures all Vault Agent sidecars to revoke their token when shutting down
revokeOnShutdown: false
webhook:
# Configures failurePolicy of the webhook. The "unspecified" default behaviour depends on the
# API Version of the WebHook.
# To block pod creation while the webhook is unavailable, set the policy to `Fail` below.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy
#
failurePolicy: Ignore
# matchPolicy specifies the approach to accepting changes based on the rules of
# the MutatingWebhookConfiguration.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchpolicy
# for more details.
#
matchPolicy: Exact
# timeoutSeconds is the amount of seconds before the webhook request will be ignored
# or fails.
# If it is ignored or fails depends on the failurePolicy
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#timeouts
# for more details.
#
timeoutSeconds: 10
# namespaceSelector is the selector for restricting the webhook to only
# specific namespaces.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
# for more details.
# Example:
namespaceSelector:
matchLabels:
sidecar-injector-sbx: enabled
#namespaceSelector: {}
# objectSelector is the selector for restricting the webhook to only
# specific labels.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector
# for more details.
# Example:
# objectSelector:
# matchLabels:
# vault-sidecar-injector: enabled
objectSelector: |
matchExpressions:
- key: app.kubernetes.io/name
operator: NotIn
values:
- {{ template "vault.name" . }}-agent-injector
# Extra annotations to attach to the webhook
annotations: {}
# Deprecated: please use 'webhook.failurePolicy' instead
# Configures failurePolicy of the webhook. The "unspecified" default behaviour depends on the
# API Version of the WebHook.
# To block pod creation while webhook is unavailable, set the policy to `Fail` below.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy
#
failurePolicy: Ignore
# Deprecated: please use 'webhook.namespaceSelector' instead
# namespaceSelector is the selector for restricting the webhook to only
# specific namespaces.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-namespaceselector
# for more details.
# Example:
# namespaceSelector:
# matchLabels:
# sidecar-injector-sbx: enabled
namespaceSelector: {}
# Deprecated: please use 'webhook.objectSelector' instead
# objectSelector is the selector for restricting the webhook to only
# specific labels.
# See https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-objectselector
# for more details.
# Example:
# objectSelector:
# matchLabels:
# vault-sidecar-injector: enabled
objectSelector: {}
# Deprecated: please use 'webhook.annotations' instead
# Extra annotations to attach to the webhook
webhookAnnotations: {}
certs:
# secretName is the name of the secret that has the TLS certificate and
# private key to serve the injector webhook. If this is null, then the
# injector will default to its automatic management mode that will assign
# a service account to the injector to generate its own certificates.
secretName: secretName
# caBundle is a base64-encoded PEM-encoded certificate bundle for the CA
# that signed the TLS certificate that the webhook serves. This must be set
# if secretName is non-null unless an external service like cert-manager is
# keeping the caBundle updated.
caBundle: "bas364EncodedCA"
# certName and keyName are the names of the files within the secret for
# the TLS cert and private key, respectively. These have reasonable
# defaults but can be customized if necessary.
certName: tls.crt
keyName: tls.key
1 Like
Hi @manso502,
Thanks a lot for providing your values, this is insightful.
I took these exact values and applied it against a fresh minikube cluster. I know, it’s not an OpenShift cluster, but at least I can confirm the error message you receive, even when I change the values to openshift: false (because I test on minikube):
$ kubectl create ns vault
$ helm install hashicorp/vault --generate-name --values values.yaml -n vault
$ kubectl logs -n kube-system kube-apiserver-minikube
...
W0820 20:58:50.460819 1 dispatcher.go:210] Failed calling webhook, failing open vault.hashicorp.com: failed calling webhook "vault.hashicorp.com": failed to call webhook: Post "https://vault-1724187528-agent-injector-svc.vault.svc:443/mutate?timeout=30s": dial tcp 10.102.212.58:443: connect: connection refused
That said, I find the indentation in your values file is somewhat off around the line where you start your injector configuration. The injector should have an indent of 0, same as the global key (see vault-helm/values.yaml at main · hashicorp/vault-helm · GitHub). So it would like similar to this (extract):
# If deploying to OpenShift
openshift: false
# << reset the indentation all the way to the left here
injector:
# True if you want to enable vault agent injection.
# @default: global.enabled
enabled: true
Try resetting the injector key to have no indent at all (same level as global key). When I do this, I don’t get the error message (timeout) in the logs anymore. I also removed the top-level vault key in my example, it was not needed.
Let me know if this helps.
Best,
Andreas
Hi @andreas.gruhler ,
I have checked the indentation, It is absolutely fine as you mentioned it is on the same level as global. I also have kubernetes lab environment in that environment I am able to make it work without any initial errors.
Thanks,
Shahrukh
1 Like