Error: rpc error: code = Unavailable desc = transport is closing - Azure provider

jananath · June 29, 2022, 2:08pm

Hi Team,

When I am trying to run the terraform apply, I am getting the following error:


Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing


Releasing state lock. This may take a few moments...
Environment creation failed.
`terraform apply` exit code 1

It was working fine before but now, it is giving me this error. There hasn’t been any modifications to any of the modules.

Terraform version: v0.14.5

And here’s my provider.tf looks like:

terraform {
  required_version = "~> 0.13"
  backend "azurerm" {
  }

  required_providers {
    azurerm = "= 2.44.0"
    azuread = "= 1.2.2"
    random  = "= 2.2"
    azuredevops = {
      source  = "microsoft/azuredevops"
      version = "=0.1.0"
    }
  }
}

Could someone help me with this? Thank you

apparentlymart · June 29, 2022, 8:18pm

Hi @jananath!

This message seems like what Terraform Core can return if a provider plugin crashes in an ungraceful way, which causes the connection between Terraform Core and the provider (the “transport” in this error message) to close with an error.

Unfortunately it seems like either the provider isn’t producing a message when it crashes or it’s producing it in a way that Terraform Core doesn’t understand how to relay into its own output, and so there’s no context about what crashed. You might be able to see more information if you set the environment variable TF_LOG=trace before re-running this operation, which will generate a lot of noisy logging but hopefully will end with something that helps explain what went wrong in the provider.

kugelblitzz · July 11, 2022, 5:47pm

Hi @apparentlymart,

I and Jananth are both facing the same issue, I’ve run the same with the TF_LOG=trace option,

2022/07/11 17:35:47 [TRACE] EvalWriteState: using RefreshState for module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint[0]
2022/07/11 17:35:47 [TRACE] EvalWriteState: recording 13 dependencies for module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint[0]
2022/07/11 17:35:47 [TRACE] EvalWriteState: writing current state object for module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint[0]
2022/07/11 17:35:47 [TRACE] Re-validating config for "module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint[0]"
2022/07/11 17:35:47 [TRACE] GRPCProvider: ValidateResourceTypeConfig
2022/07/11 17:35:47 [TRACE] GRPCProvider: PlanResourceChange
2022/07/11 17:35:47 [TRACE] EvalWriteState: writing current state object for module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint[0]
2022/07/11 17:35:47 [TRACE] EvalWriteDiff: recorded NoOp change for module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint[0]
2022/07/11 17:35:47 [TRACE] vertex "module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint[0]": visit complete
2022/07/11 17:35:47 [TRACE] vertex "module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint": dynamic subgraph completed successfully
2022/07/11 17:35:47 [TRACE] vertex "module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint": visit complete
2022/07/11 17:35:47 [TRACE] vertex "module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint (expand)": dynamic subgraph completed successfully
2022/07/11 17:35:47 [TRACE] vertex "module.secret-ad-key-vault.azurerm_private_endpoint.private_endpoint (expand)": visit complete
2022/07/11 17:35:47 [TRACE] dag/walk: visiting "module.secret-ad-key-vault (close)"
2022/07/11 17:35:47 [TRACE] vertex "module.secret-ad-key-vault (close)": starting visit (*terraform.nodeCloseModule)
2022/07/11 17:35:47 [TRACE] vertex "module.secret-ad-key-vault (close)": visit complete
2022/07/11 17:35:47 [TRACE] dag/walk: visiting "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)"
2022/07/11 17:35:47 [TRACE] vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)": starting visit (*terraform.graphNodeCloseProvider)
2022/07/11 17:35:47 [TRACE] GRPCProvider: Close
2022/07/11 17:35:47 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2022-07-11T17:35:47.905Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.44.0/linux_amd64/terraform-provider-azurerm_v2.44.0_x5 pid=15505
2022-07-11T17:35:47.906Z [DEBUG] plugin: plugin exited
2022/07/11 17:35:47 [TRACE] vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)": visit complete
2022/07/11 17:35:47 [TRACE] dag/walk: upstream of "root" errored, so skipping

Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Canceled desc = context canceled


2022/07/11 17:35:47 [DEBUG] Azure Backend Request:
HEAD /terraform/confidentialName.stage.terraform.tfstate HTTP/1.1
Host: myTFSTATE.blob.core.windows.net
User-Agent: Terraform/0.14.8
X-Ms-Date: Mon, 11 Jul 2022 17:35:47 GMT
X-Ms-Lease-Id: 2fd6afdb-685f-0378-5f8d-4efca881b7f3
X-Ms-Version: 2018-11-09



Error: rpc error: code = Canceled desc = context canceled



Error: rpc error: code = Unavailable desc = transport is closing



Error: rpc error: code = Unavailable desc = transport is closing


2022/07/11 17:35:47 [DEBUG] Azure Backend Response for https://myTFSTATE.blob.core.windows.net/terraform/confidentialName.stage.terraform.tfstate:
HTTP/1.1 200 OK
Content-Length: 627537
Accept-Ranges: bytes
Content-Md5: 0hBad9kwP5mhbkThorw6Pw==
Content-Type: application/json
Date: Mon, 11 Jul 2022 17:35:47 GMT
Etag: "0x8DA6363BE940955"
Last-Modified: Mon, 11 Jul 2022 17:35:27 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Access-Tier: Hot
X-Ms-Access-Tier-Inferred: true
X-Ms-Blob-Type: BlockBlob
X-Ms-Creation-Time: Wed, 24 Mar 2021 16:30:53 GMT
X-Ms-Lease-Duration: infinite
X-Ms-Lease-State: leased
X-Ms-Lease-Status: locked
X-Ms-Meta-Terraformlockid: ey**********9
X-Ms-Request-Id: 9c8e042a-201e-0098-744c-950a2d000000
X-Ms-Server-Encrypted: true
X-Ms-Version: 2018-11-09


2022/07/11 17:35:47 [DEBUG] Azure Backend Request:
HEAD /terraform/confidentialName.stage.terraform.tfstate HTTP/1.1
Host: myTFSTATE.blob.core.windows.net
User-Agent: Terraform/0.14.8
X-Ms-Date: Mon, 11 Jul 2022 17:35:47 GMT
X-Ms-Lease-Id: 2fd6afdb-685f-0378-5f8d-4efca881b7f3
X-Ms-Version: 2018-11-09


2022/07/11 17:35:47 [DEBUG] Azure Backend Response for https://myTFSTATE.blob.core.windows.net/terraform/confidentialName.stage.terraform.tfstate:
HTTP/1.1 200 OK
Content-Length: 627537
Accept-Ranges: bytes
Content-Md5: 0hBad9kwP5mhbkThorw6Pw==
Content-Type: application/json
Date: Mon, 11 Jul 2022 17:35:47 GMT
Etag: "0x8DA6363BE940955"
Last-Modified: Mon, 11 Jul 2022 17:35:27 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Access-Tier: Hot
X-Ms-Access-Tier-Inferred: true
X-Ms-Blob-Type: BlockBlob
X-Ms-Creation-Time: Wed, 24 Mar 2021 16:30:53 GMT
X-Ms-Lease-Duration: infinite
X-Ms-Lease-State: leased
X-Ms-Lease-Status: locked
X-Ms-Meta-Terraformlockid: ey***********9
X-Ms-Request-Id: 9c8e0430-201e-0098-794c-950a2d000000
X-Ms-Server-Encrypted: true
X-Ms-Version: 2018-11-09


2022/07/11 17:35:47 [DEBUG] Azure Backend Request:
PUT /terraform/confidentialName.stage.terraform.tfstate?comp=metadata HTTP/1.1
Host: myTFSTATE.blob.core.windows.net
User-Agent: Terraform/0.14.8
Content-Length: 0
X-Ms-Date: Mon, 11 Jul 2022 17:35:47 GMT
X-Ms-Lease-Id: 2fd6afdb-685f-0378-5f8d-4efca881b7f3
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip


2022/07/11 17:35:47 [DEBUG] Azure Backend Response for https://myTFSTATE.blob.core.windows.net/terraform/confidentialName.stage.terraform.tfstate?comp=metadata:
HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 11 Jul 2022 17:35:47 GMT
Etag: "0x8"
Last-Modified: Mon, 11 Jul 2022 17:35:47 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Request-Id: 9c8e0432-201e-0098-7b4c-950a2d000000
X-Ms-Request-Server-Encrypted: true
X-Ms-Version: 2018-11-09


2022/07/11 17:35:47 [DEBUG] Azure Backend Request:
PUT /terraform/confidentialName.stage.terraform.tfstate?comp=lease HTTP/1.1
Host: myTFSTATE.blob.core.windows.net
User-Agent: Terraform/0.14.8
Content-Length: 0
X-Ms-Date: Mon, 11 Jul 2022 17:35:47 GMT
X-Ms-Lease-Action: release
X-Ms-Lease-Id: 2fd6afdb-685f-0378-5f8d-4efca881b7f3
X-Ms-Version: 2018-11-09
Accept-Encoding: gzip


2022/07/11 17:35:47 [DEBUG] Azure Backend Response for https://myTFSTATE.blob.core.windows.net/terraform/confidentialName.stage.terraform.tfstate?comp=lease:
HTTP/1.1 200 OK
Content-Length: 0
Date: Mon, 11 Jul 2022 17:35:47 GMT
Etag: "0x8"
Last-Modified: Mon, 11 Jul 2022 17:35:47 GMT
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
X-Ms-Request-Id: 9c8e0439-201e-0098-024c-950a2d000000
X-Ms-Version: 2018-11-09


2022-07-11T17:35:47.925Z [DEBUG] plugin: plugin exited
Environment creation failed.
`terraform apply` exit code 1

But I’m unable to exactly pinpoint what exactly is going wrong observing the above trace, your help would be much appreciated.

Thanks
Jonathan

kugelblitzz · August 22, 2022, 2:27pm

This is resolved. Issue was the PAT i was using to authenticate with Azure Devops.

whilst i did have stakeholder permission to the pipelines it still needed Project Administrator permission, Once i asked my co-worker to add me as proj admin this worked without any errors.

This verbose log doesn’t really show anything regarding that soughts of error and just figured this out by trial and error. Is this truly even verbose ?