Depends on not working as expected after upgrade to 0.13.3

When I upgraded terraform to version 0.13.3 I had some weird issues with dependencys on only 1 module that I’ve noticed. It used to look like this

Info: Check mongo module
e[0me[1mInitializing modules...e[0m
- module.mongo_remove
  Getting source "./remove"
- module.mongo_docker
  Getting source "./../../common/docker"
- module.mongo_firewall
  Getting source "./../../firewall/mongo"
- module.mongo_install
  Getting source "./install"
- module.mongo_prerequisites
  Getting source "./../../common/checkers/prerequisites/check"
- module.mongo_validate_prerequisites
  Getting source "./../../common/checkers/prerequisites/validate"
- module.mongo_check
  Getting source "./../../common/checkers/docker-service/check"
- module.mongo_docker.docker_check
  Getting source "./../checkers/system-service/check"
- module.mongo_docker.docker_remove
  Getting source "./remove"
- module.mongo_docker.docker_install
  Getting source "./install"
- module.mongo_firewall.firewall_install
  Getting source "./install"

e[0me[1mInitializing provider plugins...e[0m

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.external: version = "~> 1.0"
* provider.null: version = "~> 1.0"

e[0me[1me[32mTerraform has been successfully initialized!e[0me[32me[0m
e[0me[32m
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.e[0m
e[0me[1mdata.external.check[2]: Refreshing state...e[0m
e[0me[1mdata.external.check[2]: Refreshing state...e[0m
e[0me[1mdata.external.check[1]: Refreshing state...e[0m
e[0me[1mdata.external.check[1]: Refreshing state...e[0m
e[0me[1mdata.external.check[0]: Refreshing state...e[0m
e[0me[1mdata.external.check[0]: Refreshing state...e[0m
e[0me[1mdata.external.check[1]: Refreshing state...e[0m
e[0me[1mdata.external.check[0]: Refreshing state...e[0m
e[0me[1mdata.external.check[2]: Refreshing state...e[0m
e[0me[1mmodule.mongo_docker.module.docker_remove.null_resource.pre_remove_event: Creating...e[0m
  triggers.%:             "" => "1"
  triggers.dependency_id: "" => ""e[0m
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.docker_installer_configurer: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.remove_event: Creating...e[0m
  triggers.%:             "" => "1"
  triggers.dependency_id: "" => ""e[0m
e[0me[1mmodule.mongo_install.null_resource.validate_prerequisites_event: Creating...e[0m
  triggers.%:             "" => "1"
  triggers.dependency_id: "" => ""e[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.firewall_installer_configurer: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.validate_prerequisites_event: Creation complete after 0s (ID: 9050386070001685934)e[0me[0m
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.sync_docker_remover: Creating...e[0m
  triggers.%:             "" => "1"
  triggers.dependency_id: "" => ""e[0m
e[0me[1mmodule.mongo_docker.module.docker_remove.null_resource.pre_remove_event: Creation complete after 0s (ID: 8879117556107838365)e[0me[0m
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.docker_installer_configurer: Creation complete after 0s (ID: 3488604205563503930)e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.remove_event: Creation complete after 0s (ID: 7847106381723204898)e[0me[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.firewall_installer_configurer: Creation complete after 0s (ID: 8506272082534208864)e[0me[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.docker_installed_event: Creating...e[0m
  triggers.%:             "" => "1"
  triggers.dependency_id: "" => "3488604205563503930"e[0m
e[0me[1mmodule.mongo_install.null_resource.docker_installed_event: Creating...e[0m
  triggers.%:             "" => "1"
  triggers.dependency_id: "" => "3488604205563503930"e[0m
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.sync_docker_remover: Creation complete after 0s (ID: 4375431121785790095)e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.firewall_installed_event: Creating...e[0m
  triggers.%:             "" => "1"
  triggers.dependency_id: "" => "8506272082534208864"e[0m
e[0me[1mmodule.mongo_install.null_resource.docker_installed_event: Creation complete after 0s (ID: 7924848570717042763)e[0me[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.docker_installed_event: Creation complete after 0s (ID: 7585172036620367338)e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.firewall_installed_event: Creation complete after 0s (ID: 1956364395947805049)e[0me[0m
e[0me[1me[32m
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.e[0m

Now it looks like

Info: Check mongo module
e[0me[1mInitializing modules...e[0m
- mongo_check in tf-modules/common/checkers/docker-service/check
- mongo_docker in tf-modules/common/docker
- mongo_docker.docker_check in tf-modules/common/checkers/system-service/check
- mongo_docker.docker_install in tf-modules/common/docker/install
- mongo_docker.docker_remove in tf-modules/common/docker/remove
- mongo_firewall in tf-modules/firewall/mongo
- mongo_firewall.firewall_install in tf-modules/firewall/mongo/install
- mongo_install in tf-modules/_base/mongo/install
- mongo_prerequisites in tf-modules/common/checkers/prerequisites/check
- mongo_remove in tf-modules/base/mongo/remove
- mongo_validate_prerequisites in tf-modules/common/checkers/prerequisites/validate

e[0me[1mInitializing the backend...e[0m

e[0me[1mInitializing provider plugins...e[0m
- Using previously-installed hashicorp/null v2.1.2
- Using previously-installed hashicorp/external v1.2.0

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/external: version = "~> 1.2.0"
* hashicorp/null: version = "~> 2.1.2"

e[0me[1me[32mTerraform has been successfully initialized!e[0me[32me[0m
e[0me[32m
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.e[0m
e[0me[1mmodule.mongo_check.data.external.check[1]: Refreshing state...e[0m
e[0me[1mmodule.mongo_prerequisites.data.external.check[2]: Refreshing state...e[0m
e[0me[1mmodule.mongo_check.data.external.check[0]: Refreshing state...e[0m
e[0me[1mmodule.mongo_check.data.external.check[2]: Refreshing state...e[0m
e[0me[1mmodule.mongo_prerequisites.data.external.check[0]: Refreshing state...e[0m
e[0me[1mmodule.mongo_prerequisites.data.external.check[1]: Refreshing state...e[0m
e[0me[1mmodule.mongo_docker.module.docker_check.data.external.check[2]: Refreshing state...e[0m
e[0me[1mmodule.mongo_docker.module.docker_check.data.external.check[0]: Refreshing state...e[0m
e[0me[1mmodule.mongo_docker.module.docker_check.data.external.check[1]: Refreshing state...e[0m
e[0me[1mmodule.mongo_install.null_resource.mongo_nodes[1]: Creating...e[0me[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.firewall_installer_configurer: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.mongo_nodes[2]: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.mongo_nodes[0]: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.mongo_nodes[0]: Provisioning with 'remote-exec'...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.validate_prerequisites_event: Creating...e[0me[0m
e[0me[1mmodule.mongo_docker.module.docker_remove.null_resource.pre_remove_event: Creating...e[0me[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.firewall_installer_configurer: Creation complete after 0s [id=954969314837947499]e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.mongo_nodes[1]: Provisioning with 'remote-exec'...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.remove_event: Creating...e[0me[0m
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.docker_installer_configurer: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.mongo_nodes[2]: Provisioning with 'remote-exec'...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.validate_prerequisites_event: Creation complete after 0s [id=6912375400821085256]e[0me[0m
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[2] (remote-exec): Connecting to remote host via SSH...
module.mongo_install.null_resource.mongo_nodes[2] (remote-exec):   Host: 10.32.25.138
module.mongo_install.null_resource.mongo_nodes[2] (remote-exec):  
module.mongo_install.null_resource.mongo_nodes[2] (remote-exec):   Password: false
module.mongo_install.null_resource.mongo_nodes[2] (remote-exec):   Private key: true
module.mongo_install.null_resource.mongo_nodes[2] (remote-exec):   Certificate: false
module.mongo_install.null_resource.mongo_nodes[2] (remote-exec):   SSH Agent: false
module.mongo_install.null_resource.mongo_nodes[2] (remote-exec):   Checking Host Key: false
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[1] (remote-exec): Connecting to remote host via SSH...
module.mongo_install.null_resource.mongo_nodes[1] (remote-exec):   Host: 10.32.25.140
module.mongo_install.null_resource.mongo_nodes[1] (remote-exec):   Password: false
module.mongo_install.null_resource.mongo_nodes[1] (remote-exec):   Private key: true
module.mongo_install.null_resource.mongo_nodes[1] (remote-exec):   Certificate: false
module.mongo_install.null_resource.mongo_nodes[1] (remote-exec):   SSH Agent: false
module.mongo_install.null_resource.mongo_nodes[1] (remote-exec):   Checking Host Key: false
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[0] (remote-exec): Connecting to remote host via SSH...
module.mongo_install.null_resource.mongo_nodes[0] (remote-exec):   Host: 10.32.25.139
module.mongo_install.null_resource.mongo_nodes[0] (remote-exec):   Password: false
module.mongo_install.null_resource.mongo_nodes[0] (remote-exec):   Private key: true
module.mongo_install.null_resource.mongo_nodes[0] (remote-exec):   Certificate: false
module.mongo_install.null_resource.mongo_nodes[0] (remote-exec):   SSH Agent: false
module.mongo_install.null_resource.mongo_nodes[0] (remote-exec):   Checking Host Key: false
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.sync_docker_remover: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.firewall_installed_event: Creating...e[0me[0m
e[0me[1mmodule.mongo_docker.module.docker_remove.null_resource.pre_remove_event: Creation complete after 0s [id=8617710274066431057]e[0me[0m
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.sync_docker_remover: Creation complete after 0s [id=4343277536061595297]e[0me[0m
e[0me[1mmodule.mongo_docker.module.docker_install.null_resource.docker_installer_configurer: Creation complete after 0s [id=1891531917289835356]e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.remove_event: Creation complete after 0s [id=1560545592791064908]e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.firewall_installed_event: Creation complete after 0s [id=2438770378749676536]e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.docker_installed_event: Creating...e[0me[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.docker_installed_event: Creating...e[0me[0m
e[0me[1mmodule.mongo_install.null_resource.docker_installed_event: Creation complete after 0s [id=4052929886878125079]e[0me[0m
e[0me[1mmodule.mongo_firewall.module.firewall_install.null_resource.docker_installed_event: Creation complete after 0s [id=5610732938907299751]e[0me[0m
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[0] (remote-exec): Connected!
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[1] (remote-exec): Connected!
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[2] (remote-exec): Connected!
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[1] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[0] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[2] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[0] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[2] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[1] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[0] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[2] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[1] (remote-exec): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[0] (remote-exec): docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[0] (remote-exec): See 'docker run --help'.
e[0me[0m
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[2] (remote-exec): docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[2] (remote-exec): See 'docker run --help'.
e[0me[0m
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[1] (remote-exec): docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
e[0me[0mmodule.mongo_install.null_resource.mongo_nodes[1] (remote-exec): See 'docker run --help'.
e[0me[0m
e[31m
e[1me[31mError: e[0me[0me[1merror executing "/home/triton-installer/tf-mongo_node_1.sh": Process exited with status 125e[0m

e[0me[0me[0m
e[31m
e[1me[31mError: e[0me[0me[1merror executing "/home/installer/tf-mongo_node_2.sh": Process exited with status 125e[0m

e[0me[0me[0m
e[31m
e[1me[31mError: e[0me[0me[1merror executing "/home/installer/tf-mongo_node_3.sh": Process exited with status 125e[0m

notice mongonodes tries to run during the check phase now

here is how a small snippet of the main.tf used to look

#Create volume with mongo.key
resource "null_resource" "mongo_key_volume" {
  count = "${length(var.mongo_hosts) * local.enable }"

  depends_on = ["null_resource.mongo_key_creator"]

  connection {
    host = "${element(var.mongo_hosts, count.index )}"
    user = "${var.user}"
    private_key = "${var.key}"
    script_path = "/home/${var.user}/tf-mongo_key_volume.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "mkdir -p /tmp/base/mongo"
    ]
  }

  provisioner "file" {
    source      = "${path.module}/resources"
    destination = "/tmp/base/mongo"
  }

  provisioner "remote-exec" {
    inline = [
      "docker volume create --name mongo-key",
      "sudo chmod 400 /tmp/base/mongo/resources/mongo.key",
      "docker run --rm -v mongo-key:/tmp/output \\",
              "-v /tmp/base/mongo/resources:/tmp/input \\",
              "${var.registry}/infrastructure/mongo:custom-3.6.8-security \\",
              "/bin/bash -c 'cp -T /tmp/input/mongo.key /tmp/output/mongo.key \\",
              "&& chown mongodb:mongodb /tmp/output/mongo.key'",
      "sudo rm -rf /tmp/base/mongo/resources"
    ]
  }
}




# Install mongo coordinator for cluster
resource "null_resource" "mongo_coordinator" {
  count = "${local.enable}"

  depends_on = ["null_resource.mongo_key_volume"]
 
  connection {
    host = "${element(var.mongo_hosts, 0)}"
    user = "${var.user}"
    private_key = "${var.key}"
    script_path = "/home/${var.user}/tf-mongo_coordinator.sh"
  }
  
  provisioner "remote-exec" {
    inline = [
      "mkdir -p /tmp/logging"
    ]
  }
  
  provisioner "file" {
    source      = "${path.module}/resources/client.conf.tpl"
    destination = "/tmp/logging/client.conf.tpl"
  }
  
  provisioner "remote-exec" {
    inline = [
      "export KUBE_IP=${element(var.kubelet_hosts,0)}",
      "export LOGGING_RESOURCES=/tmp/logging/",
      
      "for file in $(find $LOGGING_RESOURCES -name '*.tpl'); do name=$(basename $file); dir=$(dirname $file);name_no_ext=$(echo $name | sed -e 's/.tpl$//'); \\",
      "envsubst '\\",
      "$${KUBE_IP}' \\",
      "< $dir/$name > $dir/$name_no_ext; done",

    ]
  }

  provisioner "remote-exec" {
    inline = [
      "sudo mv /tmp/logging/client.conf /etc/rsyslog.d/",
      "sudo systemctl restart rsyslog",
      "sudo rm -rf /tmp/logging"
    ]
  }

  provisioner "remote-exec" {
    inline = [
      "docker volume create --name mongo-data",
      "docker volume create --name mongo-config",
      "docker network create --driver bridge mongo_net",
      "docker run -d --restart always --name mongo -p 27017:27017 -p 9216:9216 --log-driver syslog \\",
                  "--network=mongo_net \\",
                  "-e COORDINATOR=true \\",
                  "-e PRIMARY=${element(var.mongo_hosts, 0)} \\",
                  "-e SECONDARY_01=${element(var.mongo_hosts, 1)} \\",
                  "-e SECONDARY_02=${element(var.mongo_hosts, 2)} \\",
                  "-e REPLICASET=${var.replicaset} \\",
                  "-e MONGO_INITDB_ROOT_USERNAME=${var.mongo_admin_user} \\",
                  "-e MONGO_INITDB_ROOT_PASSWORD=\"${var.mongo_admin_pwd}\" \\",
                  "-v mongo-data:/data/db \\",
                  "-v mongo-config:/data/configdb \\",
                  "-v mongo-key:/var/private \\",
                  "${var.registry}/infrastructure/mongo:custom-3.6.8-security --replSet rstriton --noscripting --auth --logappend --bind_ip_all --keyFile /var/private/mongo.key"
    ]
  }
}

# Add nodes to mongo cluster
resource "null_resource" "mongo_nodes" {
  count = "${(length(var.mongo_hosts) - 1) * local.enable }"

  depends_on = ["null_resource.mongo_key_volume"]

  connection {
    host = "${element(var.mongo_hosts, count.index + 1)}"
    user = "${var.user}"
    private_key = "${var.key}"
    script_path = "/home/${var.user}/tf-mongo_node_${count.index + 1}.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "docker volume create --name mongo-data",
      "docker volume create --name mongo-config",
      "docker network create --driver bridge mongo_net",
      "docker run -d --restart always --name mongo -p 27017:27017 -p 9216:9216 --log-driver syslog \\",
                  "--network=mongo_net \\",
                  "-e COORDINATOR=false \\",
                  "-e PRIMARY=${element(var.mongo_hosts, 0)} \\",
                  "-e SECONDARY_01=${element(var.mongo_hosts, 1)} \\",
                  "-e SECONDARY_02=${element(var.mongo_hosts, 2)} \\",
                  "-e REPLICASET=${var.replicaset} \\",
                  "-e MONGO_INITDB_ROOT_USERNAME=${var.mongo_admin_user} \\",
                  "-e MONGO_INITDB_ROOT_PASSWORD=\"${var.mongo_admin_pwd}\" \\",
                  "-v mongo-data:/data/db \\",
                  "-v mongo-config:/data/configdb \\",
                  "-v mongo-key:/var/private \\",
                  "${var.registry}/infrastructure/mongo:custom-3.6.8-security --replSet rstriton --noscripting --auth --logappend --bind_ip_all --keyFile /var/private/mongo.key"
    ]
  }

}

Here is how it looks now

#Create volume with mongo.key
resource "null_resource" "mongo_key_volume" {
  count = length(var.mongo_hosts) * local.enable

  depends_on = [null_resource.mongo_key_creator]

  connection {
    host        = element(var.mongo_hosts, count.index)
    user        = var.user
    private_key = var.key
    script_path = "/home/${var.user}/tf-mongo_key_volume.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "mkdir -p /tmp/base/mongo",
    ]
  }

  provisioner "file" {
    source      = "${path.module}/resources"
    destination = "/tmp/base/mongo"
  }

  provisioner "remote-exec" {
    inline = [
      "docker volume create --name mongo-key",
      "sudo chmod 400 /tmp/base/mongo/resources/mongo.key",
      "docker run --rm -v mongo-key:/tmp/output \\",
      "-v /tmp/base/mongo/resources:/tmp/input \\",
      "${var.registry}/infrastructure/mongo:custom-3.6.8-security \\",
      "/bin/bash -c 'cp -T /tmp/input/mongo.key /tmp/output/mongo.key \\",
      "&& chown mongodb:mongodb /tmp/output/mongo.key'",
      "sudo rm -rf /tmp/base/mongo/resources",
    ]
  }
}

# Install mongo coordinator for cluster
resource "null_resource" "mongo_coordinator" {
  count = local.enable

  depends_on = [null_resource.mongo_key_volume]

  connection {
    host        = element(var.mongo_hosts, 0)
    user        = var.user
    private_key = var.key
    script_path = "/home/${var.user}/tf-mongo_coordinator.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "docker volume create --name mongo-data",
      "docker volume create --name mongo-config",
      "docker network create --driver bridge mongo_net",
      "docker run -d --restart always --name mongo -p 27017:27017 -p 9216:9216 \\",
      "--network=mongo_net \\",
      "-e COORDINATOR=true \\",
      "-e PRIMARY=${element(var.mongo_hosts, 0)} \\",
      "-e SECONDARY_01=${element(var.mongo_hosts, 1)} \\",
      "-e SECONDARY_02=${element(var.mongo_hosts, 2)} \\",
      "-e REPLICASET=${var.replicaset} \\",
      "-e MONGO_INITDB_ROOT_USERNAME=${var.mongo_admin_user} \\",
      "-e MONGO_INITDB_ROOT_PASSWORD=\"${var.mongo_admin_pwd}\" \\",
      "-v mongo-data:/data/db \\",
      "-v mongo-config:/data/configdb \\",
      "-v mongo-key:/var/private \\",
      "${var.registry}/infrastructure/mongo:custom-3.6.8-security --replSet rstriton --noscripting --auth --logappend --bind_ip_all --keyFile /var/private/mongo.key",
    ]
  }
}

# Add nodes to mongo cluster
resource "null_resource" "mongo_nodes" {
  count = length(var.mongo_hosts) - 1 * local.enable

  depends_on = [null_resource.mongo_key_volume]

  connection {
    host        = element(var.mongo_hosts, count.index + 1)
    user        = var.user
    private_key = var.key
    script_path = "/home/${var.user}/tf-mongo_node_${count.index + 1}.sh"
  }

  provisioner "remote-exec" {
    inline = [
      "docker volume create --name mongo-data",
      "docker volume create --name mongo-config",
      "docker network create --driver bridge mongo_net",
      "docker run -d --restart always --name mongo -p 27017:27017 -p 9216:9216 \\",
      "--network=mongo_net \\",
      "-e COORDINATOR=false \\",
      "-e PRIMARY=${element(var.mongo_hosts, 0)} \\",
      "-e SECONDARY_01=${element(var.mongo_hosts, 1)} \\",
      "-e SECONDARY_02=${element(var.mongo_hosts, 2)} \\",
      "-e REPLICASET=${var.replicaset} \\",
      "-e MONGO_INITDB_ROOT_USERNAME=${var.mongo_admin_user} \\",
      "-e MONGO_INITDB_ROOT_PASSWORD=\"${var.mongo_admin_pwd}\" \\",
      "-v mongo-data:/data/db \\",
      "-v mongo-config:/data/configdb \\",
      "-v mongo-key:/var/private \\",
      "${var.registry}/infrastructure/mongo:custom-3.6.8-security --replSet rstriton --noscripting --auth --logappend --bind_ip_all --keyFile /var/private/mongo.key",
    ]
  }
}

Hi @nicks1993!

Unfortunately with so much going on here it’s hard to get a sense of what might be going on. It seems like quite a few things have potentially changed here.

One important difference I noticed is that your “old” Terraform output doesn’t seem to mention module.mongo_install.null_resource.mongo_nodes at all, whereas in your new output it’s that resource that seems to be encountering the error with docker. Is your old output from a terraform apply that was only making updates to existing objects, whereas the new terraform apply is creating new objects?

It would help if you can say a little more about how this configuration is intended to work at a high level – what sequence of operations you are expecting, and how that differs from the sequence of operations Terraform is actually taking. As someone who is not expert with either Mongo or Docker it’s hard to trace through all of the various setup steps you’ve listed here and understand how they might relate to the error message you shared.