Terraform create and delete route table indefinitely in an aws peering vpc

Terraform Providers AWS
1

I’m having strange Terraform behavior when I run it multiple times. The scenario is this: two VPCs in the same region and aws account and I want to create a peering between them. In the first round Terraform correctly creates the peering between the two vpcs and their respective routes but in the second round TF deletes the routes. This is my terraform code:

provider "aws" {
region                = "..."
access_key            = "..."
secret_key            = "..."
}

resource "aws_vpc" "vpc1" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_vpc" "vpc2" {
  cidr_block = "10.1.0.0/16"
}

resource "aws_subnet" "public1" {
  vpc_id     = aws_vpc.vpc1.id
  cidr_block = "10.0.1.0/24"
}

resource "aws_subnet" "private1" {
  vpc_id     = aws_vpc.vpc1.id
  cidr_block = "10.0.2.0/24"
}

resource "aws_subnet" "public2" {
  vpc_id     = aws_vpc.vpc2.id
  cidr_block = "10.1.1.0/24"
}

resource "aws_subnet" "private2" {
  vpc_id     = aws_vpc.vpc2.id
  cidr_block = "10.1.2.0/24"
}

resource "aws_internet_gateway" "igw1" {
  vpc_id = aws_vpc.vpc1.id
}

resource "aws_internet_gateway" "igw2" {
  vpc_id = aws_vpc.vpc2.id
}

resource "aws_route_table" "public_route_table1" {
  vpc_id = aws_vpc.vpc1.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw1.id
  }
}

resource "aws_route_table" "public_route_table2" {
  vpc_id = aws_vpc.vpc2.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw2.id
  }
}

resource "aws_route_table_association" "public1_rta" {
  subnet_id      = aws_subnet.public1.id
  route_table_id = aws_route_table.public_route_table1.id
}

resource "aws_route_table_association" "public2_rta" {
  subnet_id      = aws_subnet.public2.id
  route_table_id = aws_route_table.public_route_table2.id
}

resource "aws_vpc_peering_connection" "peer" {
  vpc_id      = aws_vpc.vpc1.id
  peer_vpc_id = aws_vpc.vpc2.id
  auto_accept = true
}

resource "aws_route" "peer1_to_2" {
  route_table_id         = aws_route_table.public_route_table1.id
  destination_cidr_block = aws_vpc.vpc2.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.peer.id

  depends_on = [aws_vpc_peering_connection.peer]
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route" "peer2_to_1" {
  route_table_id         = aws_route_table.public_route_table2.id
  destination_cidr_block = aws_vpc.vpc1.cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.peer.id

  depends_on = [aws_vpc_peering_connection.peer]
  lifecycle {
    create_before_destroy = true
  }
}

The first run TF creates peering and the correct route and two ec2 instances ping correctly. But if I run for the second time TF I have

# aws_route_table.public_route_table1 will be updated in-place
  ~ resource "aws_route_table" "public_route_table1" {
        id               = "rtb-0ad43422d1bc73f82"
      ~ route            = [
          - {
              - cidr_block                 = "0.0.0.0/0"
              - gateway_id                 = "igw-06ef35a787e283282"
                # (11 unchanged attributes hidden)
            },
          - {
              - cidr_block                 = "10.1.0.0/16"
              - vpc_peering_connection_id  = "pcx-0afbf2e62bf9a43d2"
                # (11 unchanged attributes hidden)
            },
          + {
              + cidr_block = "0.0.0.0/0"
              + gateway_id = "igw-06ef35a787e283282"
            },
        ]
        tags             = {}
        # (5 unchanged attributes hidden)
    }

  # aws_route_table.public_route_table2 will be updated in-place
  ~ resource "aws_route_table" "public_route_table2" {
        id               = "rtb-0f282179701ca2405"
      ~ route            = [
          - {
              - cidr_block                 = "0.0.0.0/0"
              - gateway_id                 = "igw-015a5ba053949c7cf"
                # (11 unchanged attributes hidden)
            },
          - {
              - cidr_block                 = "10.0.0.0/16"
              - vpc_peering_connection_id  = "pcx-0afbf2e62bf9a43d2"
                # (11 unchanged attributes hidden)
            },
          + {
              + cidr_block = "0.0.0.0/0"
              + gateway_id = "igw-015a5ba053949c7cf"
            },
        ]
        tags             = {}
        # (5 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

and after applied this changes running TF for the third time

# aws_route.peer1_to_2 will be created
  + resource "aws_route" "peer1_to_2" {
      + destination_cidr_block    = "10.1.0.0/16"
      + id                        = (known after apply)
      + instance_id               = (known after apply)
      + instance_owner_id         = (known after apply)
      + network_interface_id      = (known after apply)
      + origin                    = (known after apply)
      + route_table_id            = "rtb-0ad43422d1bc73f82"
      + state                     = (known after apply)
      + vpc_peering_connection_id = "pcx-0afbf2e62bf9a43d2"
    }

  # aws_route.peer2_to_1 will be created
  + resource "aws_route" "peer2_to_1" {
      + destination_cidr_block    = "10.0.0.0/16"
      + id                        = (known after apply)
      + instance_id               = (known after apply)
      + instance_owner_id         = (known after apply)
      + network_interface_id      = (known after apply)
      + origin                    = (known after apply)
      + route_table_id            = "rtb-0f282179701ca2405"
      + state                     = (known after apply)
      + vpc_peering_connection_id = "pcx-0afbf2e62bf9a43d2"
    }

Plan: 2 to add, 0 to change, 0 to destroy.

and so on. What’s wrong ?

I expected that after the firs apply terraform will not change any resource but instead it continues to apply and remove the resource aws_route. I’m using all updated (terraform and aws provider).

2

Hi @lucianosarra,

It seems that you are using both aws_route resources and a aws_route_table nested block route with the same route table, which contradicts the warning at the beginning of the aws_route documentation:

Terraform currently provides both a standalone Route resource and a Route Table resource with routes defined in-line. At this time you cannot use a Route Table with in-line routes in conjunction with any Route resources. Doing so will cause a conflict of rule settings and will overwrite rules.

The symptom you describe seems to match the consequence described in this warning: the route table’s “desired state” conflicts with the individual route desired states and so you have declared a contradiction that prevents Terraform from reaching convergence: there can’t be both just a single route and multiple separate routes at the same time.

I think you can avoid the problem by removing the route block from your aws_route_table resource and declaring that route using a separate aws_route resource instead.

You could also go the other way and delete the aws_route resources in favor of nested route blocks, if you’d prefer to manage all of the routes together and thus allow Terraform to detect and propose to delete any new routes that might be added outside of Terraform.

3

Thank you @apparentlymart for your replay. I can confirm that it is as you say: I used incorrectly resource aws_route_table. Here below I attached the correct snippet related to routing section which runs correctly

resource "aws_route_table" "public_route_table1" {
  vpc_id = aws_vpc.vpc1.id
}

resource "aws_route" "public_route_1" {
  route_table_id         = aws_route_table.public_route_table1.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.igw1.id
}

resource "aws_route_table_association" "public1_rta" {
  subnet_id      = aws_subnet.public1.id
  route_table_id = aws_route_table.public_route_table1.id
}

resource "aws_route_table" "public_route_table2" {
  vpc_id = aws_vpc.vpc2.id
}

resource "aws_route" "public_route_2" {
  route_table_id         = aws_route_table.public_route_table2.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.igw2.id
}

resource "aws_route_table_association" "public2_rta" {
  subnet_id      = aws_subnet.public2.id
  route_table_id = aws_route_table.public_route_table2.id
}

With this part fixed now I run the terraform script withou modification.