Vault Agent Injector Not Being Triggered

Vault
,
21

Thanks for all your ongoing help. I am also a bit unsure at this point.

I gave this a try today. I deleted my Kubernetes network policy and replaced it with a Calico network policy:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: base-allow
  namespace: vault
spec:
  ingress:
  - action: Allow
    source:
      namespaceSelector: "kubernetes.io/metadata.name in {'vault', 'istio-system', 'monitoring', 'kube-system', 'test'}"
  selector: "all()"
  types:
  - Ingress
  egress: []

I tried explicitly allowing the egress to see if that would make any difference, then recreated the test pod.

In my Calico pod logs, I can see the test pod gets spun up with no errors:

calico-node-xbs4g calico-node 2023-06-01 12:28:52.932 [INFO][75] felix/endpoint_mgr.go 481: Re-evaluated workload endpoint status adminUp=true failed=false known=true operUp=true status="up" workloadEndpointID=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test/test-pod-for-vault", EndpointId:"eth0"}
calico-node-xbs4g calico-node 2023-06-01 12:28:52.932 [INFO][75] felix/status_combiner.go 58: Storing endpoint status update ipVersion=0x4 status="up" workload=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test/test-pod-for-vault", EndpointId:"eth0"}
calico-node-xbs4g calico-node 2023-06-01 12:28:52.933 [INFO][75] felix/status_combiner.go 81: Endpoint up for at least one IP version id=proto.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"test/test-pod-for-vault", EndpointId:"eth0"} ipVersion=0x4 status="up"

I donā€™t see anything regarding the web hook here.

22

Hi @nat-ray

I couldnā€™t reproduce your issue. Iā€™ve used Kind version 1.23.2, disabled the default CNI and installed Calico.

I installed Vault in the vault namespace with

helm install vault hashicorp/vault -n vault --create-namespace --set injector.enabled=true

and attached to Pod vault-0 and initialized and unsealed Vault.

I then created a Network Policy with a default deny Ingress policy for all the Pods in the vault namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: vault
spec:
  podSelector: {}
  policyTypes:
  - Ingress

And tested it from a Pod in the kube-system namespace using the alpine image.

/ # curl -k -L -vvv https://vault-agent-injector-svc.vault --connect-
timeout 3
*   Trying 10.96.240.26:443...
* ipv4 connect timeout after 2996ms, move on!
* Failed to connect to vault-agent-injector-svc.vault port 443 after 3001 ms: Timeout was reached
* Closing connection 0
curl: (28) Failed to connect to vault-agent-injector-svc.vault port 443 after 3001 ms: Timeout was reached

The moment I appended

  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system

to the policy and applied it, the same curl command worked fine.

/ # curl -k -L -vvv https://vault-agent-injector-svc.vault --connec
t-timeout 3
*   Trying 10.96.240.26:443...
* Connected to vault-agent-injector-svc.vault (10.96.240.26) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):

... REDACTED FOR BREVITY ...

< HTTP/2 404 
< content-type: text/plain; charset=utf-8
< x-content-type-options: nosniff
< content-length: 19
< date: Thu, 01 Jun 2023 17:57:38 GMT
< 
404 page not found

And the most interesting part, whenever I tried to create a Pod with the following manifest

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: asdf
  annotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'asdf'
  name: asdf
  namespace: default
spec:
  containers:
  - image: nginx
    name: asdf
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

it worked fine and the agent was injected (independently of whether the Network Policy was blocking ingress traffic to the Vault namespace or not) :man_shrugging:

23

Hi @ macmiranda!

I found a kind of workaround. I created a new network policy like this:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-agent-injector-netpol
  namespace: vault
spec:
  ingress:
  - from:
    - ipBlock:
        cidr: <master_node_IP_CIDR>
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: test
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
  podSelector:
    matchExpressions:
    - key: app.kubernetes.io/name
      operator: In
      values:
      - vault-agent-injector
      - vault
  policyTypes:
  - Ingress

This allows the Kube API Server to talk to the web hook and also allows my pod to connect directly to the Vault via the service hostname, <service-name>.<namespace>.svc.cluster.local.

2 Likes