I was using Vault in the following environment.
Workload Identity to access KMS and GCS from the vault server.
This was working well until yesterday (2020/05/11).
I updated GKE version to
1.15.11-gke.12 yesterday, and suddenly vault doesn’t start.
2020-05-12T13:35:04.633Z [WARN] storage migration check error: error="failed to read value for "core/migration": Get https://storage.googleapis.com/xxxx-storage/core/migration: compute: Received 403 ` Unable to generate token; IAM returned 403 Forbidden: Request had insufficient authentication scopes. This error could be caused by a missing IAM policy binding on the target IAM service account. You can create the necessary policy binding with: gcloud iam service-accounts add-iam-policy-binding \ --role=roles/iam.workloadIdentityUser \ --member="serviceAccount:xxxx.svc.id.goog[default/vault]" \ email@example.com For more information, refer to the Workload Identity documentation: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity `"
I tried reverting the version of GKE, creating a new GKE cluster and installing vault from helm again, but I can’t start it with the same error.
I tried adding full access to GCS to ServiceAccount, but it didn’t work.
I thought it was a Workload Identity issue, so I started POD with the same service account and tried to access GCS from gsutil, and this worked.
I am at a loss for the cause.
Do you have any possible problems?