Terraform 0.12 to 0.13.7 upgrade plug in error

[bimalviswam]

We are upgrading from 0.12 to 0.13.7 . This is our code.

  #has to be removed after upgrade
  /usr/local/bin/terraform/0.13.7/terraform 0.13upgrade -yes
  /usr/local/bin/terraform/0.13.7/terraform state replace-provider -auto-approve registry.terraform.io/-/template  registry.terraform.io/hashicorp/template
  /usr/local/bin/terraform/0.13.7/terraform state replace-provider -auto-approve registry.terraform.io/-/aws  registry.terraform.io/hashicorp/aws

    echo "Init terraform for prod account env=$env"
    #remove -reconfigure after the upgrade release
    /usr/local/bin/terraform/0.13.7/terraform init -backend-config="bucket=comp-fpp-api-terraform-prod" -backend-config="dynamodb_table=comp-fpp-api-terraform" -reconfigure
  

 echo "List workspaces"
  /usr/local/bin/terraform/0.13.7/terraform workspace list

  echo "Select workspace $env"
  /usr/local/bin/terraform/0.13.7/terraform workspace select ${env} || terraform workspace new ${env}

  echo "Show workspace selected"
  /usr/local/bin/terraform/0.13.7/terraform workspace show

  echo "Request to secret manager"
  secretARN=<command to get secret>



  echo "Call terraform plan"
  echo "/usr/local/bin/terraform/0.13.7/terraform plan -var 'secret_arn=$secretArn' -var-file 'var/${env}/terraform.tfvars'" | bash -

We hit the below error related to plug in.

[1me[32mUpgrade complete!e[0m

Use your version control system to review the proposed changes, make any
necessary adjustments, and then commit.
e[0m
e[0me[0me[1me[33mBackend reinitialization required. Please run "terraform init".e[0m
e[33mReason: Initial configuration of the requested backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.
e[0m
e[31mError loading the state: Initialization required. Please see the error message above.

Please ensure that your Terraform state exists and that you've
configured it properly. You can use the "-state" flag to point
Terraform at another state file.e[0me[0m
e[0me[0me[1me[33mBackend reinitialization required. Please run "terraform init".e[0m
e[33mReason: Initial configuration of the requested backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.
e[0m
e[31mError loading the state: Initialization required. Please see the error message above.

Please ensure that your Terraform state exists and that you've
configured it properly. You can use the "-state" flag to point
Terraform at another state file.e[0me[0m
Init terraform for prod account env=uat
e[0me[1mInitializing modules...e[0m
- auto_scaling in modules/auto-scaling
- create-experience-result-lambda in modules/lambda
- create-kyc-result-lambda in modules/lambda
- create-questionnaire-result-lambda in modules/lambda
- create-suitability-score-lambda in modules/lambda
- datapoints_crawler in modules/glue
- datapoints_crawler.sample_crawler_role in modules/iam_role
- datapoints_event in modules/event_bridge
- datapoints_event_bridge_iam_role in modules/iam_role
- datapoints_lambda_iam_role in modules/iam_role
- datapoints_step_function in modules/step_function
- datapoints_step_function_iam_role in modules/iam_role
- dynamodb_clients_table in modules/dynamodb_table
- dynamodb_experience_results_table in modules/dynamodb_table
- dynamodb_investment_profiles_table in modules/dynamodb_table
- dynamodb_kyc_results_table in modules/dynamodb_table
- dynamodb_pjm_table in modules/dynamodb_table
- dynamodb_questionnaire_results_table in modules/dynamodb_table
- dynamodb_questionnaire_table in modules/dynamodb_table
- dynamodb_suitability_score_table in modules/dynamodb_table
- ecr_principal in modules/ecr
- ecr_secondary in modules/ecr
- ecs in modules/ecs_cluster
- image_im_cs in modules/s3_object
- image_im_cze in modules/s3_object
- image_im_de in modules/s3_object
- image_im_en in modules/s3_object
- image_im_fr in modules/s3_object
- image_im_zaa in modules/s3_object
- nlb in modules/nlb
- role in modules/iam_role
- s3_bucket in modules/s3
- s3_vpc_endpoint_interface in modules/vpc
- secret_manager in modules/secret_manager
- task_definition in modules/ecs_task

e[0me[1mInitializing the backend...e[0m
e[0me[32m
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.e[0m

e[0me[1mInitializing provider plugins...e[0m
- Finding hashicorp/aws versions matching "~> 3.0, ~> 3.0"...
- Finding latest version of hashicorp/template...
- Installing hashicorp/aws v3.76.1...
- Installed hashicorp/aws v3.76.1 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/template: version = "~> 2.2.0"

e[0me[1me[32mTerraform has been successfully initialized!e[0me[32me[0m
e[0me[32m
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.e[0m
List workspaces
* default
  dr
  prd
  prod
  uat
  uat-dr

Select workspace uat
e[0me[32mSwitched to workspace "uat".e[0m
Show workspace selected
uat
Request to secret manager
Call terraform plan
e[31m
e[1me[31mError: e[0me[0me[1mCould not load plugine[0m

e[0m
Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

2 problems:

- Failed to instantiate provider "registry.terraform.io/-/aws" to obtain
schema: unknown provider "registry.terraform.io/-/aws"
- Failed to instantiate provider "registry.terraform.io/-/template" to obtain
schema: unknown provider "registry.terraform.io/-/template"

This is our version file which was generated by terraform upgrade command

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
    template = {
      source = "hashicorp/template"
    }
  }
  required_version = ">= 0.13"
}

This is from our main.tf

# Configure the AWS Provider
provider "aws" {
  version = "~> 3.0"
  region  = var.region
}

provider "aws" {
  alias   = "secondary_region"
  version = "~> 3.0"
  region  = var.region_secondary
}

[maxb]

There are multiple problems here.

1) You are running the shell script without configuring it to stop when there is an error. As a result, you’re not spotting that things are failing as early as your second terraform command.

A lot of the other problems you are encountering are because you are doing things in the wrong order.

2) If you read the error messages coming from terraform state replace-provider they are telling you this needs to happen after terraform init. But actually you shouldn’t be running terraform state replace-provider at all as part of normal 0.12 to 0.13 upgrades. You can run it at the very end, if for some reason Terraform has incorrectly not updated all your registry.terraform.io/-/ provider references automatically, but other than that you shouldn’t need it at all.

3) It is not needed or appropriate to be using -reconfigure on the terraform init command here - the error messages you have quoted reveal this to be a fresh configuration, not a reconfigure scenario.

4) You are also running terraform init too early - it should be after workspace selection, so that it can take into account the contents of the state file. That is why terraform plan is saying you need to init again - because the details of what needed to be initialized changed when you changed workspace.

5) Once you have started using a terraform { required_providers {} } block, you should no longer be specifying version constraints in your provider blocks. Migrate version contraints to required_providers - https://developer.hashicorp.com/terraform/language/providers/requirements#version-constraints

[bimalviswam]

Thank you for the quick response. If we exit when there is an error, would that release the state lock gracefully?

I have changed the order based on the feedback .

 echo "> Environment is: $env"
  echo "> Region to apply changes: $region"

  ls /usr/local/bin/

  /usr/local/bin/terraform/0.13.7/terraform --version
  cd terraform

  echo "Request to secret manager"
  uimSecretArn=<command to get secret>

  #has to be removed after upgrade
  /usr/local/bin/terraform/0.13.7/terraform 0.13upgrade -yes

  echo "List workspaces"
  /usr/local/bin/terraform/0.13.7/terraform workspace list

  echo "Select workspace $env"
  /usr/local/bin/terraform/0.13.7/terraform workspace select ${env} || terraform workspace new ${env}

  echo "Show workspace selected"
  /usr/local/bin/terraform/0.13.7/terraform workspace show



  if [ $env == "dev" ] || [ $env == "qa" ] || [ $env == "stg" ] || [ $env == "dev-dr" ]|| [ $env == "stg-dr" ]; then
    echo "Init terraform for non-prod account env=$env"
    #remove -reconfigure after the upgrade release
    /usr/local/bin/terraform/0.13.7/terraform init
  else
    echo "Init terraform for prod account env=$env"
    #remove -reconfigure after the upgrade release
    /usr/local/bin/terraform/0.13.7/terraform init -backend-config="bucket=comp-fpp-api-terraform-prod" -backend-config="dynamodb_table=comp-fpp-api-terraform" 
  fi

  /usr/local/bin/terraform/0.13.7/terraform state replace-provider -auto-approve registry.terraform.io/-/template  registry.terraform.io/hashicorp/template
  /usr/local/bin/terraform/0.13.7/terraform state replace-provider -auto-approve registry.terraform.io/-/aws  registry.terraform.io/hashicorp/aws

  echo "Call terraform plan"
  echo "/usr/local/bin/terraform/0.13.7/terraform plan -var 'uim_secret_arn=$uimSecretArn' -var-file 'var/${env}/terraform.tfvars'" | bash -

Now terraform notifies to initialize backend with List Workspace command . Should I initialize twice? Or what should be the right order?

e[1me[32mUpgrade complete!e[0m

Use your version control system to review the proposed changes, make any
necessary adjustments, and then commit.
e[0m
List workspaces
e[0me[0me[1me[33mBackend reinitialization required. Please run "terraform init".e[0m
e[33mReason: Initial configuration of the requested backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.
e[0m
e[31m
e[1me[31mError: e[0me[0me[1mInitialization required. Please see the error message above.e[0m

e[0me[0me[0m

[maxb]

@bimalviswam

If we exit when there is an error, would that release the state lock gracefully?

If Terraform has already exited with an error, it will already have released the state lock.

@bimalviswam

Now terraform notifies to initialize backend with List Workspace command . Should I initialize twice? Or what should be the right order?

Ah, I see. I haven’t worked with workspaces much. OK, yes, it makes sense that you’d need to init twice, once to establish connectivity to the backend to be able to list workspaces, and again after selecting a workspace, to take account of potential requirements for extra providers, mentioned only in the state file and not in the configuration (although this should be a rare edge case).