Module.gke_cluster.google_container_cluster fails with "Gaia id not found"

Every day during the last 5 week I get the following error when module.gke_cluster.google_container_cluster tries to create the gke cluster.

Error: Error: googleapi: Error 404: Generic::not_found: Not found; Gaia id not found for email XXXXXX-compute@developer.gserviceaccount.com., notFound

│ with module.gke_cluster.google_container_cluster.primary,
│ on modules/gke_cluster/main.tf line 2, in resource “google_container_cluster” “primary”:
│ 2: resource “google_container_cluster” “primary” {

Notes: This XXXXXX-compute@developer.gserviceaccount.com service account was the default service account and was deleted by mistake about two years ago.
It was replaced with another service account as the default service account, but Terraform keeps using it for a certain action.

For the node pools I could enter a customized service account, but no such an option for the GKE cluster (master) itself.

It should be noted I started to see these errors few week ago after about two years.

Usually when retrying, it does NOT happen and the cluster is created.

Adding the debug log around this failure:

-----------------------------------------------------: timestamp=2022-09-29T04:09:11.182Z
2022-09-29T04:09:11.182Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2022-09-29T04:09:11.182Z
2022-09-29T04:09:11.182Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-09-29T04:09:11.182Z
2022-09-29T04:09:11.182Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [DEBUG] Got RUNNING while polling for operation operation-1664424550363-5e9c90a147334-d7cc47d8-8d6eeaeb’s status: timestamp=2022-09-29T04:09:11.182Z
2022-09-29T04:09:11.182Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [TRACE] Waiting 10s before next try: timestamp=2022-09-29T04:09:11.182Z
2022-09-29T04:09:11.261Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [DEBUG] Google API Response Details:
—[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Thu, 29 Sep 2022 04:09:11 GMT
Etag: cdDdzAEpzAXE-Jr6KZLA5NB4Eig=/8fCyXmj2EPPKh9lSKPGR43VUDnY=
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
“kind”: “compute#operation”,
“id”: “9129457215694480521”,
“name”: “operation-1664424550370-5e9c90a148ba3-c652d70f-c188f7ef”,
“operationType”: “insert”,
“targetLink”: “https://www.googleapis.com/compute/v1/projects/example-project/global/backendBuckets/example-ui-backend-bucket”,
“targetId”: “7179832390214788233”,
“status”: “RUNNING”,
“user”: “XXXX@cloudbuild.gserviceaccount.com”,
“progress”: 0,
“insertTime”: “2022-09-28T21:09:10.741-07:00”,
“startTime”: “2022-09-28T21:09:10.754-07:00”,
“selfLink”: “https://www.googleapis.com/compute/v1/projects/example-project/global/operations/operation-1664424550370-5e9c90a148ba3-c652d70f-c188f7ef
}
-----------------------------------------------------: timestamp=2022-09-29T04:09:11.261Z
2022-09-29T04:09:11.261Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2022-09-29T04:09:11.261Z
2022-09-29T04:09:11.261Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2022-09-29T04:09:11.261Z
2022-09-29T04:09:11.261Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [DEBUG] Got RUNNING while polling for operation operation-1664424550370-5e9c90a148ba3-c652d70f-c188f7ef’s status: timestamp=2022-09-29T04:09:11.261Z
2022-09-29T04:09:11.261Z [INFO] provider.terraform-provider-google_v4.36.0_x5: 2022/09/29 04:09:11 [TRACE] Waiting 10s before next try: timestamp=2022-09-29T04:09:11.261Z
2022-09-29T04:09:12.317Z [INFO] provider.terraform-provider-google-beta_v4.36.0_x5: 2022/09/29 04:09:12 [DEBUG] Google API Response Details:
—[ RESPONSE ]--------------------------------------
HTTP/2.0 404 Not Found
Cache-Control: private
Content-Type: application/json; charset=UTF-8
Date: Thu, 29 Sep 2022 04:09:12 GMT
Server: ESF
Vary: Origin
Vary: X-Origin
Vary: Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

{
“error”: {
“code”: 404,
“message”: “Generic::not_found: Not found; Gaia id not found for email XXXX-compute@developer.gserviceaccount.com.”,
“errors”: [
{
“message”: “Generic::not_found: Not found; Gaia id not found for email XXXX-compute@developer.gserviceaccount.com.”,
“domain”: “global”,
“reason”: “notFound”
}
],
“status”: “NOT_FOUND”
}
}