Error authenticating on Google Cloud Cloud Run Job - Error 400: Invalid form of account ID default

Hello there,

I am trying to use Vault enterprise version 13.1 (Docker)

I want to run the Vault agent in Cloud Run as a Job (Cloud Run Service can be another option as well)

I have set up my service account (mySA.myProject@iam.gserviceaccount.com) with the roles

ServiceAccountKeyAdmin and serviceAccountTokenCreator

I have created a simple docker image where the vault image and alpine image are pulled then in the entrypoint it is just running the v
vault agent -config /vault-agent/agent.hcl command

The agent.hcl file looks like this

exit_after_auth = false
pid_file = "/pid/pidfile"

vault {
address = "{my vault address here}:8200"
retry {
num_retries = 3
}
}
auto_auth {
method "gcp" {
mount_path = "auth/gcp_poc"
config = {
type = "iam"
role = "gcp-iam-role"
}
}

}

listener "tcp" {
address = "127.0.0.1:8100"
tls_disable = true
}

api_proxy {
use_auto_auth_token = "force"
enforce_consistency = "always"
}

On the vault server side I have set up a GCP IAM vault role (as the GCE role cannot be used)

vault write auth/gcp_poc/role/gcp-iam-role \
type="iam" \
policies="dev" \
bound_service_accounts="mySA.myProject@iam.gserviceaccount.com"

When the Google Cloud Run Job starts, we do see the vault agent has started as we can see from the message in the image (vaultagentstart)

However, below that, there is an error here.

"2023-08-31T18:17:22.512Z [ERROR] agent.auth.handler: error getting path or data from method: error="unable to sign JWT for projects/-/serviceAccounts/default using given Vault credentials: googleapi: Error 400: Invalid form of account ID default. Should be [Gaia ID |Email |Unique ID |] of the account, badRequest" backoff=1s"

I am confused by this error because I am running this job using a specific service account and not the default service account. It also should have the capabilities for signingJWT.

I have also tried to use this IAM role on a GCE instance.

Previosuly this GCE instance VM was able to authenticate using this GCE role.

But when I changed the attached service account for this particular GCE instance with the same service account (mySA.myProject@iam.gserviceaccount.com) with the permissions serviceAccountKeyAdmin as well as serviceAccountTokenCreator.

I got the exact same error 400 above Error 400: Invalid form of account ID default. Should be [Gaia ID |Email |Unique ID |] of the account, badRequest"

I am further confused because I thought that attaching the service Account to the GCE instance would be all that is necessary for the authentication to work.

What is the proper way to use the GCP IAM role?

Any help or information regarding this would be greatly appreciated.
Thanks