Hello,
I am not sure whether this is the correct place for the issue we are facing, but maybe it is.
We are hosting our Terraform Backend on Artifactory in a Terraform backend repository.
After the night (from 08-Apr-25 to 09-Apr-25) we are not able to perform any Terraform commands for existing workspaces anymore. Regardless what Terrform command we use, we always face these errors which hint corrupt state file:
$ tf state pull
Error: Error refreshing state: 2 problems:
- Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 1.
- Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.
Or
$ tf plan -out plan.tfplan
Acquiring state lock. This may take a few moments...
Error: Unsupported state file format
The state file could not be parsed as JSON: syntax error at byte offset 1.
Error: Unsupported state file format
The state file does not have a "version" attribute, which is required to identify the format version.
Enabling traces reveals this
.....
2025-04-14T15:18:07.419+0300 [INFO] CLI command args: []string{"state", "pull"}
2025-04-14T15:18:07.441+0300 [TRACE] Meta.Backend: BackendOpts.Config not set, so using settings loaded from main.tf:20,3-19
2025-04-14T15:18:07.441+0300 [TRACE] Meta.Backend: built configuration for "remote" backend with hash value 140817437
2025-04-14T15:18:07.442+0300 [TRACE] Meta.Backend: working directory was previously initialized for "remote" backend
2025-04-14T15:18:07.442+0300 [TRACE] Meta.Backend: using already-initialized, unchanged "remote" backend configuration
2025-04-14T15:18:07.442+0300 [DEBUG] Service discovery for censored.jfrog.io at https://censored.jfrog.io/.well-known/terraform.json
2025-04-14T15:18:07.442+0300 [TRACE] HTTP client GET request to https://censored.jfrog.io/.well-known/terraform.json
2025-04-14T15:18:08.227+0300 [TRACE] Meta.BackendForPlan: backend *remote.Remote supports operations
2025-04-14T15:18:08.227+0300 [DEBUG] Service discovery for censored.jfrog.io aliased as localterraform.com
2025-04-14T15:18:08.227+0300 [TRACE] Meta.Backend: instantiated backend of type *remote.Remote
2025-04-14T15:18:08.229+0300 [TRACE] providercache.fillMetaCache: scanning directory .terraform/providers
2025-04-14T15:18:08.229+0300 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/azure/azapi v1.13.1 for linux_amd64 at .terraform/providers/registry.terraform.io/azure/azapi/1.13.1/linux_amd64
2025-04-14T15:18:08.230+0300 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/azuread v2.48.0 for linux_amd64 at .terraform/providers/registry.terraform.io/hashicorp/azuread/2.48.0/linux_amd64
2025-04-14T15:18:08.230+0300 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/azurerm v3.99.0 for linux_amd64 at .terraform/providers/registry.terraform.io/hashicorp/azurerm/3.99.0/linux_amd64
2025-04-14T15:18:08.230+0300 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/random v3.6.1 for linux_amd64 at .terraform/providers/registry.terraform.io/hashicorp/random/3.6.1/linux_amd64
2025-04-14T15:18:08.230+0300 [TRACE] getproviders.SearchLocalDirectory: found registry.terraform.io/hashicorp/time v0.11.1 for linux_amd64 at .terraform/providers/registry.terraform.io/hashicorp/time/0.11.1/linux_amd64
2025-04-14T15:18:08.230+0300 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/azure/azapi/1.13.1/linux_amd64 as a candidate package for registry.terraform.io/azure/azapi 1.13.1
2025-04-14T15:18:08.230+0300 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/azuread/2.48.0/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/azuread 2.48.0
2025-04-14T15:18:08.230+0300 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/azurerm/3.99.0/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/azurerm 3.99.0
2025-04-14T15:18:08.230+0300 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/random/3.6.1/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/random 3.6.1
2025-04-14T15:18:08.230+0300 [TRACE] providercache.fillMetaCache: including .terraform/providers/registry.terraform.io/hashicorp/time/0.11.1/linux_amd64 as a candidate package for registry.terraform.io/hashicorp/time 0.11.1
2025-04-14T15:18:08.259+0300 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/providers
2025-04-14T15:18:08.466+0300 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/providers
2025-04-14T15:18:08.478+0300 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/providers
2025-04-14T15:18:08.491+0300 [TRACE] providercache.fillMetaCache: using cached result from previous scan of .terraform/providers
2025-04-14T15:18:08.764+0300 [DEBUG] checking for provisioner in "."
2025-04-14T15:18:08.766+0300 [DEBUG] checking for provisioner in "/usr/bin"
2025-04-14T15:18:08.766+0300 [TRACE] Meta.Backend: backend *remote.Remote supports operations
Logs/traces always stop there.
The state files can be seen on Artifactory side, they do not look corrupt.
It happens with all exisiting workspaces, regardless which provider version.
Terraform versions tried: 1.5.7, 1.8.3, 1.10.4, 1.11.3 - all show same behavior.
All users on different PCs/Laptops with Mac OS or Windows WSL2 face same issues.
The intersting part is that when I create a new workspace in the same backend repository, that works without problems, state files are created and updated, e.g access rights look ok. We only face issues with the existing workspaces.
It looks like it is Artifactory related, but I am wondering whether anyone who used Terraform has seen that kind of behavior and maybe knows what to do.
Thanks for reading,
Klaus