Handling Explicit Dependencies in Terraform

Terraform
1

I am working with the Splunk Terraform provider to deploy an integration between the Splunk Observability Platform and AWS. Splunk uses an AWS IAM role with trusted relationship using an externalID and splunk account id to authenticate to AWS accounts for deploying the integration.

The following resources define the configuration:

# AWS IAM roles and permissions to grant Splunk access for retrieving data or adding AWS CloudWatch metric streams

data "aws_iam_policy_document" "aws_splunk_policy_document" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "AWS"
      identifiers = [signalfx_aws_external_integration.aws_integration.signalfx_aws_account]
    }

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = [signalfx_aws_external_integration.aws_integration.external_id]
    }
  }
}


resource "aws_iam_role" "aws_splunk_role" {
  name               = "${local.integration_name}-cw-splunk-role"
  assume_role_policy = data.aws_iam_policy_document.aws_splunk_policy_document.json
}

resource "aws_iam_policy" "aws_splunk_policy" {
  name        = "${local.integration_name}-cw-splunk-policy"
  description = "AWS permissions required by the Splunk Observability Cloud"
  policy      = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        PERMISSIONS      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "*"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "aws_splunk_policy_attachment" {
  role       = aws_iam_role.aws_splunk_role.name
  policy_arn = aws_iam_policy.aws_splunk_policy.arn
}

# Splunk Observability Platform integrating resources
resource "signalfx_aws_external_integration" "aws_integration" {
  name = "test"
}

# Wait for aws_external_integration to be available in Splunk
resource "time_sleep" "wait_30_seconds" {
  depends_on = [signalfx_aws_external_integration.aws_integration]

  create_duration = "30s"
}

resource "signalfx_aws_integration" "aws_integration" {
  enabled                 = true
  integration_id          = signalfx_aws_external_integration.aws_integration.id
  external_id             = signalfx_aws_external_integration.aws_integration.external_id
  role_arn                = aws_iam_role.aws_splunk_role.arn
  regions                 = ["us-east-1"]
  poll_rate               = 300
  use_metric_streams_sync = true
  import_cloud_watch      = true
  enable_aws_usage        = true
  depends_on = [
    aws_iam_role.aws_splunk_role,
    time_sleep.wait_30_seconds
  ]
}

The full configuration can be found in the SignalFx Terraform provider documentation.

Issue:

The terraform apply command works fine, but the issue arises during terraform destroy.

Splunk attempts to remove deployed resources while destroying the integration, but Terraform deletes the IAM role before completing the deletion of Splunk resources. As a result, the cleanup fails, causing the entire terraform destroy process to fail.

aws_iam_role_policy_attachment.aws_splunk_policy_attachment: Destruction complete after 1s
aws_iam_policy.aws_splunk_policy: Destruction complete after 0s
aws_iam_role.aws_splunk_role: Destruction complete after 0s
signalfx_aws_integration.aws_integration: Destruction complete after 0s

signalfx_aws_external_integration.aws_integration: Still destroying... [id=Gi82K1QAIAA, 10s elapsed]
signalfx_aws_external_integration.aws_integration: Still destroying... [id=Gi82K1QAIAA, 20s elapsed]
signalfx_aws_external_integration.aws_integration: Still destroying... [id=Gi82K1QAIAA, 30s elapsed]
signalfx_aws_external_integration.aws_integration: Still destroying... [id=Gi82K1QAIAA, 40s elapsed]
signalfx_aws_external_integration.aws_integration: Still destroying... [id=Gi82K1QAIAA, 50s elapsed]
╷
│ Error: Error waiting for integration Gi82K1QAIAA state for use_metric_streams_sync to become disabled: unexpected state 'CANCELLATION_FAILED', wanted target ', DISABLED'. last error: %!s(<nil>)

What I’ve Tried:

I attempted to add the following dependencies, but encountered a cycle error message:

resource "signalfx_aws_external_integration" "aws_integration" {
  name = "test"
  depends_on = [aws_iam_role.aws_splunk_role]
}

error:

 Error: Cycle: signalfx_aws_external_integration.aws_integration, data.aws_iam_policy_document.aws_splunk_policy_document, aws_iam_role.aws_splunk_role

Question:

Is there a way to handle this dependency issue between resources? Or should it be managed within the SignalFx Terraform provider?

I also found an open issue related to cycle issue in the Terraform provider.

I would appreciate any support from the HashiCorp community!

2

Hi @meraj-kashi,

I’m not familiar with these resources, but this could be a design problem with the provider, or maybe a problem with the documented use and expectations of the provider.

In your config you have the aws_iam_role.aws_splunk_role depend on the signalfx_aws_external_integration, which means the external integration must be created before the aws_iam_role, and therefore must outlive the aws_iam_role during destroy. Since the signalfx_aws_external_integration must be created before any of the other resources, it stands to reason that it could be the last one remaining during the destroy operation.

If the actual dependencies are somehow reversed during the destroy operation though, then there is a logical cycle between the resources that Terraform can’t resolve, and your attempt to add depends_on just turns that implicit logical cycle into a visible error.

Since these sound like resources which are pretty fundamental to the usage of the signalfx provider, I would expect they have considered this problem somehow, so I would check their docs and issue tracker first. I don’t see any way to add any indirection here to try and move dependencies around, so it may take some architectural change with the resources in question.

2 Likes