Hi,
I tried to use the provider docker by: kreuzwerker.
My local version is Terraform v0.14.6
the terraform cloud uses the latest one Terraform v0.14.7
i use this command to initialize the repo terraform init -backend=false
I have a file called main.tf it looks like this:
terraform {
required_version = ">= 0.12"
required_providers {
docker = {
source = "kreuzwerker/docker"
version = "2.11.0"
}
}
backend "remote" {
organization = "TFCloudOrg"
}
}
provider "docker" {
// alias = "docker_apps"
// If using the docker engine API port
//host = "tcp://${var.netLB_elasticIP}:2376"
// source = "kreuzwerker/docker"
host = "ssh://ubuntu@${var.netLB_elasticIP}:22"
}
provider "aws" {
access_key = var.access_key
secret_key = var.secret_key
region = var.region
}
locals {
// Assigned to all AWS Cloud resources
common_tags = {
"cc:environment" = var.name_environment
}
}
This is how I call the Docker module with my container definitions
module "bitbucket" {
// providers = {
// docker = docker.docker_apps
// }
source = "./modules/docker-apps"
depends_on = [aws_instance.ec2_instancet.id]
}
and this is the module’s structure
modules/docker-apps/
├── README.md
├── docker.tf
├── main.tf
└── outputs.tf
When I push changes to my VCS or run from terraform Cloud UI I get the following
Terraform v0.14.7
Configuring remote state backend...
Initializing Terraform configuration...
Setup failed: Failed terraform init (exit 1): <nil>
Initializing the backend...
Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Finding latest version of hashicorp/template...
- Finding latest version of hashicorp/docker...
- Finding kreuzwerker/docker versions matching "2.11.0"...
- Finding hashicorp/aws versions matching ">= 2.42.0, >= 2.49.0"...
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing kreuzwerker/docker v2.11.0...
- Installed kreuzwerker/docker v2.11.0 (self-signed, key ID 24E54F214569A8A5)
- Installing hashicorp/aws v3.28.0...
- Installed hashicorp/aws v3.28.0 (signed by HashiCorp)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider
hashicorp/docker: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/docker
If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.
Did you intend to use kreuzwerker/docker? If so, you must specify that source
address in each module which requires that provider. To see which modules are
currently depending on hashicorp/docker, run the following command:
terraform providers
I have opened a support ticket. I will update this thread.
Reference [StackOver flow Question How to select correct docker provider in terraform 0.14
Hi @aleon1220 ,
Can you post the result of the “terraform providers” command ?
It seems that either your code or your state file requires the old-style “hashicorp/docker” instead of “kreuzwerker/docker”.
You might need to use the terraform state replace-provider command to fix the address of the provider in the state file.
Regards,
Filip
thanks for answering this is what i get
terraform providers
Backend reinitialization required. Please run "terraform init".
Reason: Initial configuration of the requested backend "remote"
The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.
Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.
If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.
Error: Initialization required. Please see the error message above.
Did some extra steps
$ terraform state replace-provider hashicorp/docker kreuzwerker/docker
Backend reinitialization required. Please run "terraform init".
Reason: Initial configuration of the requested backend "remote"
The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.
Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.
If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.
Error loading the state: Initialization required. Please see the error message above.
Please ensure that your Terraform state exists and that you've
configured it properly. You can use the "-state" flag to point
Terraform at another state file.
$ terraform init -backend-config=backend-production.hcl
Initializing modules...
Initializing the backend...
Successfully configured the backend "remote"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Finding latest version of hashicorp/docker...
- Reusing previous version of kreuzwerker/docker from the dependency lock file
- Using previously-installed hashicorp/aws v3.28.0
- Using previously-installed hashicorp/template v2.2.0
- Using previously-installed kreuzwerker/docker v2.11.0
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider
hashicorp/docker: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/docker
If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.
Did you intend to use kreuzwerker/docker? If so, you must specify that source
address in each module which requires that provider. To see which modules are
currently depending on hashicorp/docker, run the following command:
terraform providers
$ terraform state replace-provider hashicorp/docker kreuzwerker/docker
No matching resources found.
Releasing state lock. This may take a few moments...
Ohh it seems is working now from terraform cloud. Is not clear though if the provider was installed or not but I am getting a different error now 
Terraform Cloud OutPut
Terraform v0.14.7
Configuring remote state backend...
Initializing Terraform configuration...
module.db.module.db_subnet_group.aws_db_subnet_group.this[0]: Refreshing state... [id=bitbucket-postgres-20210217230030669800000002]
module.db.module.db_option_group.aws_db_option_group.this[0]: Refreshing state... [id=bitbucket-postgres-20210217230030675500000003]
module.db.module.db_parameter_group.aws_db_parameter_group.this[0]: Refreshing state... [id=bitbucket-postgres-20210217230030668900000001]
module.postgresql_security_group.module.sg.aws_security_group.this_name_prefix[0]: Refreshing state... [id=sg-0c5486a89e98c587f]
aws_security_group.secg_bitbucket_conf: Refreshing state... [id=sg-0d5bb1ee5601849a8]
aws_iam_role.Bitbucket_Backup_iam_role: Refreshing state... [id=iam-rnd-prd-Bitbucket_Backup_iam_role-use1]
aws_iam_instance_profile.bitbucket_iam_profile: Refreshing state... [id=iam-rnd-prd-Bitbucket-Backup-use1]
module.postgresql_security_group.module.sg.aws_security_group_rule.ingress_with_self[0]: Refreshing state... [id=sgrule-73620040]
aws_iam_role_policy_attachment.AmazonEC2RoleforSSM-policy-attach: Refreshing state... [id=iam-rnd-prd-Bitbucket_Backup_iam_role-use1-20210217220844173900000005]
aws_iam_role_policy_attachment.AmazonSSMFullAccess-policy-attach: Refreshing state... [id=iam-rnd-prd-Bitbucket_Backup_iam_role-use1-20210217220844165100000004]
module.postgresql_security_group.module.sg.aws_security_group_rule.ingress_rules[0]: Refreshing state... [id=sgrule-3541315741]
module.postgresql_security_group.module.sg.aws_security_group_rule.egress_rules[0]: Refreshing state... [id=sgrule-3376683493]
aws_iam_role_policy_attachment.EC2InstanceConnect-policy-attach: Refreshing state... [id=iam-rnd-prd-Bitbucket_Backup_iam_role-use1-20210217220844155600000002]
aws_iam_role_policy_attachment.CloudWatchFullAccess-policy-attach: Refreshing state... [id=iam-rnd-prd-Bitbucket_Backup_iam_role-use1-20210217220844156000000003]
aws_instance.ec2_bitbucket: Refreshing state... [id=i-0c5a81f3d58de6034]
module.db.module.db_instance.aws_db_instance.this[0]: Refreshing state... [id=bitbucket-postgres]
aws_lb_target_group_attachment.tg_conf_ssh_inst: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:270128389919:targetgroup/tg-rnd-prod-bitbucket/35de290bb03ff096-20210218015529267700000001]
aws_lb_target_group_attachment.tg_conf_tls_inst: Refreshing state... [id=arn:aws:elasticloadbalancing:us-east-1:270128389919:targetgroup/tg-rnd-prod-bitbucket-tls/93326cdd2157d8ab-20210218015529357400000002]
aws_ebs_volume.Bitbucket_ebs: Refreshing state... [id=vol-02f585ee51ceff854]
aws_volume_attachment.ebs_att: Refreshing state... [id=vai-1835672283]
Warning: Value for undeclared variable
The root module does not declare a variable named "name_segment" but a value
was found in file "/terraform/rnd/bitbucket/prd/terraform.tfvars". To use this
value, add a "variable" block to the configuration.
Using a variables file to set an undeclared variable is deprecated and will
become an error in a future release. If you wish to provide certain "global"
settings to all configurations in your organization, use TF_VAR_...
environment variables to set these instead.
Error: Error pinging Docker server: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
on <empty> line 0:
(source code not available)
Fix in the module structure
- create a versions.tf file with the content such as this
// Used in terraform version 0.14.6
// check https://discuss.hashicorp.com/t/docker-provider-in-terraform-cloud-error-failed-to-query-available-provider-packages/21148
terraform {
required_providers {
docker = {
source = "kreuzwerker/docker"
}
}
required_version = ">= 0.13"
}
How did you manage to get Docker working on Terraform Cloud? As far as I understand, the Docker provider assumes there is Docker already running on the machine and Docker can’t be installed on Terraform Cloud.
Hi @ngocketit. Thanks for your comment.
There was no way to do this at the time.
The Docker provider needs some improvements. A great feature to have is that you are able to SSH to the instance from terraform cloud and give instructions to the Docker engine.
One way I was thinking to achieve this is by using 2 different workspaces
- workspace to provision a prepared EC2 instance with docker installed and configured
- Another workspace that just handles the docker containers that make up the application
If there is a docker-compose terraform provider that would be much wonderful.
If anyone has ideas on how to achieve this please let me know.
In the end I solved the issue by only creating the EC2 infrastructure and using user_data (cloud-init) to provision the EC2 instance. Manually i get the docker-compose file and I execute the docker-compose app that reads from a manually prepared .env file with the secrets.
This is still WIP as i want to automate it further.
@aleon1220 Thanks for your reply. I think currently working with Terraform Cloud is annoying since suddenly what works on local machine stops working when migrating to TC. Simple things such as packaging a NodeJS lambda function turn out to be difficult (which I asked here Build & package lambda function on Terraform Cloud). TC should make it easier to install extra packages, not just binaries, similar to what Github Actions is doing.