AWS ECS Fargate issues with pulling container and IAM service scaling policy

Terraform Providers AWS
1

Hi community,

what terraform/aws provider versions:
Terraform v0.13.5

I’m tearing my hair out somewhat with a couple of issues related to a fargate terraform deployment pattern I am developing.

The first issue is with issues pulling the container from the ECR.

Status reason CannotPullContainerError: Error response from daemon: Get https://163213362226.dkr.ecr.ap-southeast-2.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

The ECS solution must run on private networks only meaning that the following was done to allow connectivity:

  1. Managed tenancy account with public and private subnets. Internet gateway as default gateway from public route table, NAT gateway as gateway for private route table. Security group for NAT gateway allowing access on port 443.
  2. Endpoint interfaces for ECR API and DKR and an S3 Gateway created as part of the account managed tenancy
  3. Security Groups for the endpoints provided as part of the account managed tenancy allowing port 443 from private subnets. Additional rule added to this SG to allow Port 443 from the service SG provided by the service terraform deployment
  4. NACL only on public subnets so irrelevant
  5. ECR has policy allowing access, service has policy granting access to the ECR - no IAM access errors experienced

The second issue is bizarre. I deploy a service autoscale role and policy based on the AWS managed ones they provide but with no “*” resources, it’s tightened up so that you can’t touch other services etc:

The role and policy deploys fine and is referenced in the service autoscaling as follows:

resource "aws_appautoscaling_target" "service_scaling_target" {
  min_capacity = var.service_min
  max_capacity = var.service_max
  resource_id = "service/${var.ecs_cluster_name}/${aws_ecs_service.service.name}"
  role_arn = aws_iam_role.ECSAutoscalingRole.arn
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace = "ecs"
}

This deploys without error and looks correct but on the next plan:

  # aws_appautoscaling_target.service_scaling_target will be updated in-place
  ~ resource "aws_appautoscaling_target" "service_scaling_target" {
        id                 = "service/ecs_test/helloworld"
        max_capacity       = 2
        min_capacity       = 1
        resource_id        = "service/ecs_test/helloworld"
      ~ role_arn           = "arn:aws:iam::163213362226:role/aws-service-role/ecs.application-autoscaling.amazonaws.com/AWSServiceRoleForApplicationAutoScaling_ECSService" -> "arn:aws:iam::163213362226:role/ECSAutoscalingRole-helloworld"
        scalable_dimension = "ecs:service:DesiredCount"
        service_namespace  = "ecs"
    }

The one I provide is showing up in the AWS console service autoscale role drop down but if select my role and save the service then when I go back in it reverts it every time.

The role and policy are:

resource "aws_iam_role" "ECSAutoscalingRole" {
  name = "ECSAutoscalingRole-${var.service_name}"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "ecs.application-autoscaling.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy" "ECSAutoscalingPolicy" {
  name = "ECSAutoscalingPolicy-${var.service_name}"
  path = "/"
  description = "Policy for ECS application autoscaling"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:UpdateServices",
        "ecs:DescribeServices"
      ],
      "Resource": "arn:aws:ecs:${var.aws_region}:${data.aws_caller_identity.current.account_id}:service/${var.ecs_cluster_name}/${var.service_name}"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricAlarm",
        "cloudwatch:DescribeAlarms"
      ],
      "Resource": "arn:aws:cloudwatch:${var.aws_region}:${data.aws_caller_identity.current.account_id}:alarm:${var.service_name}*"
    }
  ]
}
EOF
}

resource "aws_iam_policy_attachment" "ECSAutoscalePolicyAttach" {
  name = "ECS_RolePolicyAttach-${var.service_name}"
  roles = [aws_iam_role.ECSAutoscalingRole.name]
  policy_arn = aws_iam_policy.ECSAutoscalingPolicy.arn
}

Anyone able to assist with these issues please?

Thanks
Andy