Getting Error: Required plugins are not installed

I am getting this error in my build plan. sometimes it works and sometimes it doesnt work. basically i dont get this error always, i get it 60-70% of the times when i run my bamboo plan and other times it works.

Terraform version : tried 1.5.6 and 1.7.3

Error: Required plugins are not installed
13-Feb-2024 15:31:04	│
13-Feb-2024 15:31:04	│ The installed provider plugins are not consistent with the packages
13-Feb-2024 15:31:04	│ selected in the dependency lock file:
13-Feb-2024 15:31:04	│   - registry.terraform.io/hashicorp/null: there is no package for registry.terraform.io/hashicorp/null 3.2.1 cached in .terraform/providers
13-Feb-2024 15:31:04	│   - registry.terraform.io/hashicorp/random: there is no package for registry.terraform.io/hashicorp/random 2.3.1 cached in .terraform/providers
13-Feb-2024 15:31:04	│   - registry.terraform.io/hashicorp/template: there is no package for registry.terraform.io/hashicorp/template 2.2.0 cached in .terraform/providers
13-Feb-2024 15:31:04	│   - registry.terraform.io/hashicorp/vsphere: there is no package for registry.terraform.io/hashicorp/vsphere 1.15.0 cached in .terraform/providers
13-Feb-2024 15:31:04	│   - registry.terraform.io/hashicorp/asml********dns: there is no package for registry.terraform.io/hashicorp/asml********dns 1.0.0 cached in .terraform/providers
13-Feb-2024 15:31:04	│
13-Feb-2024 15:31:04	│ Terraform uses external plugins to integrate with a variety of different
13-Feb-2024 15:31:04	│ infrastructure services. You must install the required plugins before
13-Feb-2024 15:31:04	│ running Terraform operations.

My .terraform.lock.hcl looks like this.

provider "registry.terraform.io/hashicorp/asmlvcpdns" {
	  version     = "1.0.0"
	  constraints = "1.0.0"
	  hashes = [
	    "h1:0AmNLRuhkyQzRFdgJUUgqZo98fmsXWbL4KJmenrPK/A=",
	  ]
	}
	
	provider "registry.terraform.io/hashicorp/null" {
	  version     = "3.2.1"
	  constraints = "3.2.1"
	  hashes = [
	    "h1:FbGfc+muBsC17Ohy5g806iuI1hQc4SIexpYCrQHQd8w=",
	  ]
	}
	
	provider "registry.terraform.io/hashicorp/random" {
	  version     = "2.3.1"
	  constraints = "2.3.1"
	  hashes = [
	    "h1:bPBDLMpQzOjKhDlP9uH2UPIz9tSjcbCtLdiJ5ASmCx4=",
	  ]
	}
	
	provider "registry.terraform.io/hashicorp/template" {
	  version     = "2.2.0"
	  constraints = "2.2.0"
	  hashes = [
	    "h1:94qn780bi1qjrbC3uQtjJh3Wkfwd5+tTtJHOb7KTg9w=",
	  ]
	}
	
	provider "registry.terraform.io/hashicorp/vsphere" {
	  version     = "1.15.0"
	  constraints = "1.15.0"
	  hashes = [
	    "h1:LOJPpHUWYqbBSoJjW7UUrF5edYZj4NRvBoRHwl9BEOk=",
	  ]
	}

provider.tf looks like this

terraform {
  backend "remote" {
        hostname = "*******"
        organization = "wakanda-tf"
        workspaces {
            prefix = "sf-"
        }
        token = "*****"
    }
  
  required_providers {
    vsphere = {
      source = "hashicorp/vsphere"
      # version = "2.4.2"
      version = "1.15.0"
    }

    random = {
      source  = "hashicorp/random"
      version = "2.3.1"
    }

    template = {
      source = "hashicorp/template"
      version = "2.2.0"
    }

    null = {
      source = "hashicorp/null"
      version = "3.2.1"
    }

    asmlvcpdns = {
      version = "1.0.0"
    }
  }
}

provider "asmlvcpdns" {
  user           = "******"
  password       = "*******"
  vsphere_server = "*********"
  
  # if you have a self-signed cert
  allow_unverified_ssl = true
}

provider "vsphere" {
  user           = "*********"
  password       = "***********"
  vsphere_server = "*********"
  client_debug   = false

  # if you have a self-signed cert
  allow_unverified_ssl = true

  persist_session = true
}

running below command

export TF_PLUGIN_CACHE_DIR="/home/sa-wfabamboo/terraform.d/plugins"
  terraform init -input=false -plugin-dir /home/sa-wfabamboo/terraform.d/plugins

There is no internet access on the bamboo agent where i am running the build. Thats the reason i am specifying the plugin dir path with terraform init so that it doesnt need to download the provider plugins.

after running terraform init, sometimes it works and wometimes it doesnt and i get the error as mentioned above. I have tried different terraform versions but no luck. So this behavior is not consistent.

Also, before running the terraform init command, i am making sure that all the required providers setup is there in the directory that i have mentioned in the terraform init command.

cd /home/sa-wfabamboo
terraform --version
mv terraform-provider-asmlvcpdns terraform-provider-asmlvcpdns_v1.0.0_x4

echo $HOME
mkdir -p /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/vsphere/1.15.0/linux_amd64
cd /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/vsphere/1.15.0/linux_amd64
cp -rf /home/sa-wfabamboo/terraform-provider-vsphere_v1.15.0_x4 .
echo ####################################
pwd
chmod +x terraform-provider-vsphere_v1.15.0_x4

mkdir -p /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/random/2.3.1/linux_amd64
cd /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/random/2.3.1/linux_amd64
cp -rf /home/sa-wfabamboo/terraform-provider-random_v2.3.1_x4 .
chmod +x terraform-provider-random_v2.3.1_x4

mkdir -p /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64
cd /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64
cp -rf /home/sa-wfabamboo/terraform-provider-template_v2.2.0_x4 .
chmod +x terraform-provider-template_v2.2.0_x4

mkdir -p /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/null/3.2.1/linux_amd64
cd /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/null/3.2.1/linux_amd64
cp -rf /home/sa-wfabamboo/terraform-provider-null_v3.2.1_x5 .
chmod +x terraform-provider-null_v3.2.1_x5

mkdir -p /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/asmlvcpdns/1.0.0/linux_amd64
cd /home/sa-wfabamboo/terraform.d/plugins/registry.terraform.io/hashicorp/asmlvcpdns/1.0.0/linux_amd64
cp -rf /home/sa-wfabamboo/terraform-provider-asmlvcpdns_v1.0.0_x4 .
chmod +x terraform-provider-asmlvcpdns_v1.0.0_x4

# env
cd ${bamboo.build.working.directory}
ls -lRta

I am not sure why its giving the error that required plugins are not installed.

Any help would be appreciated.

Thanks