Vault does not started. GCS Backend & GKE & Workload Identity

Vault
1

I was using Vault in the following environment.

vault: https://github.com/hashicorp/vault-helm/releases/tag/v0.5.0
GKE: 1.15.11-gke.5
Backend: GCS

I use Workload Identity to access KMS and GCS from the vault server.
This was working well until yesterday (2020/05/11).

I updated GKE version to 1.15.11-gke.12 yesterday, and suddenly vault doesn’t start.

Error Log:

2020-05-12T13:35:04.633Z [WARN]  storage migration check error: error="failed to read value for "core/migration": Get https://storage.googleapis.com/xxxx-storage/core/migration: compute: Received 403 `
Unable to generate token; IAM returned 403 Forbidden: Request had insufficient authentication scopes.

This error could be caused by a missing IAM policy binding on the target IAM service account.

You can create the necessary policy binding with:

  gcloud iam service-accounts add-iam-policy-binding \
    --role=roles/iam.workloadIdentityUser \
    --member="serviceAccount:xxxx.svc.id.goog[default/vault]" \
    xxxx-vault-server@xxxx.iam.gserviceaccount.com

For more information, refer to the Workload Identity documentation:

    https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity

`"

I tried reverting the version of GKE, creating a new GKE cluster and installing vault from helm again, but I can’t start it with the same error.
I tried adding full access to GCS to ServiceAccount, but it didn’t work.

I thought it was a Workload Identity issue, so I started POD with the same service account and tried to access GCS from gsutil, and this worked.

I am at a loss for the cause.
Do you have any possible problems?

2

I rebuilt the GKE cluster and reverted the version of master to 1.15.11-gke.5, and now I can start it normally.

3

In case it helps anyone who lands here after googling that error message, I saw the same thing on a different service using Workload Identity, on GKE 1.17.5-gke-0 (RAPID channel). I don’t know how long the issue has been present becuase I just started using Workload Identity. 1.15.9-gke.24 fixes it.

4

Hi,

I am using the same GKE version but its failing with below error after upgrading GKE from to 1.15.9-gke.24,

severity: “ERROR”
textPayload: "2020-05-29T06:40:06.523Z [WARN] storage migration check error: error=“failed to read value for “core/migration”: googleapi: got HTTP response code 403 with body: <?xml version='1.0' encoding='UTF-8'?>AccessDeniedAccess denied.

Primary: /namespaces/project.svc.id.goog with additional claims does not have storage.objects.get access to the Google Cloud Storage object.
”