Vault injector with GKE private

Vault
,
1

Hello,

I work since the last days on Vault injecotr implementation on our GKE private cluster. (1)
As we already have a external production cluster, we don’t use a Vault server inside GKE. (2)

So the first steps are good.
We deploy the injector with success, create a kubernetes auth method config and role with success too. If we try the login api, its works too.

The problem is when we start to work add annotations on our application.
The init container is not created, and the pod with the application too.

If I take a look on the injector log, nothing. Just a log that the sert was opened on 8080 port.

After some research, I found on hashicorp github issue (3) that GKE private can have networking issue, as the admission webhook (running on master plane it seems) need to reach the injector on 8080 port, and by default GKE private open only 10250 and 443 from master to worker nodes.

So I opened the 8080 port between the master nodes to the worker nodes, and now the injector logs are:
2021-02-26T16:29:24.221Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-02-26T16:29:24.223Z [DEBUG] handler: checking if should inject agent…
2021-02-26T16:29:24.223Z [DEBUG] handler: checking namespaces…
2021-02-26T16:29:24.223Z [DEBUG] handler: setting default annotations…
2021-02-26T16:29:24.223Z [DEBUG] handler: creating new agent…
2021-02-26T16:29:24.223Z [DEBUG] handler: validating agent configuration…

So its better, but it seems that something else is missing…
The log is not really verbose. Only get the /mutate uri information. No host, no port…
Maybe the log can be more precise with the full url.

Does somebody alreay had this issue ? And how I can solve it.

Thanks a lot for your help !

(1) Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar | Vault - HashiCorp Learn
(2) Integrate a Kubernetes Cluster with an External Vault | Vault - HashiCorp Learn
(3) Timeout errors in MutatingWebhookConfiguration · Issue #46 · hashicorp/vault-k8s · GitHub

1 Like
2

Its me again !

I will let a note, maybe it can help someone in the futur !
So its work now !

Few things. We manage GKE with Terraform. So first things to do when working on GKE, open the port, and do it in the good way, with the GKE module.

You have a variable “firewall_inbound_ports” that need to be updated by adding the 8080 port in the list

firewall_inbound_ports = [
“8080”,
“8443”,
“9443”,
“15017”
]

After that, I get the timeout like I said earlier.
As I have nothing on the deployment event, or injector logs, I checked the GKE logs on stackdriver and found this error:

Error creating: admission webhook “vault.hashicorp.com” denied the request: error validating agent configuration: no service account name or path found

After some research, I found that the service accout of my application was not mounted.
After adding the “automountServiceAccountToken” to True in the k8s deployment, the pod with injector was trying to start ! Champagne !

Next error was our Vault cluster is behind our internal authority, so

x509: certificate signed by unknown authority

The solution is to create a secret with our CA Root certificate inside the namespace of our app, and add this 2 new annotations:

vault.hashicorp.com/ca-cert: “/vault/tls/nameOfYourFileInTheSecret”
vault.hashicorp.com/tls-secret: “nameOfTheSecret”

Done. Now every steps are good, and we are able get our database credentials formated in the file. Still have to find why the export as environment variable is not working.

So I happy to start my weekend with this success :smiley:
Bye !