VPC Peering not working when using Provider with 'assume role' in aws credentials file

Terraform Version

$ terraform version
Terraform v0.12.1
+ provider.aws v2.13.0

Terraform Configuration Files

provider "aws" {
  alias   = "prod"
  region  = "${var.region}"
  profile = "prod_admin"
}
resource "aws_vpc_peering_connection" "peer" {
  vpc_id        = "${var.local_vpc_id}"
  peer_vpc_id   = "${var.peer_vpc_id}"  
  peer_owner_id = "${var.peer_vpc_owner}"
  peer_region   = "${var.region}"
  auto_accept   = false

  tags = {
    Side = "Requester"
  }
}
resource "aws_vpc_peering_connection_accepter" "peer" {
  provider                  = "aws.prod"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
  auto_accept               = true

  tags = {
    Side = "Accepter"
  }
}

# REMOTE ROUTE TABLES RULES #
#############################

resource aws_route "peer_public_1" {
  provider                  = "aws.prod"
  route_table_id            = "${var.peer_rt_pub_id}"
  destination_cidr_block    = "${var.local_vpc_cidr}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

resource aws_route "peer_private_1" {
  provider                  = "aws.prod"
  route_table_id            = "${var.peer_rt_priv1_id}"
  destination_cidr_block    = "${var.local_vpc_cidr}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

resource aws_route "peer_private_2" {
  provider                  = "aws.prod"
  route_table_id            = "${var.peer_rt_priv2_id}"
  destination_cidr_block    = "${var.local_vpc_cidr}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

# LOCAL ROUTE TABLES RULES #
############################

resource aws_route "local_public_1" {
  route_table_id            = "${var.local_rt_pub_id}"
  destination_cidr_block    = "${var.peer_vpc_cidr}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

resource aws_route "local_private_1" {
  route_table_id            = "${var.local_rt_priv1_id}"
  destination_cidr_block    = "${var.peer_vpc_cidr}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

resource aws_route "local_private_2" {
  route_table_id            = "${var.local_rt_priv2_id}"
  destination_cidr_block    = "${var.peer_vpc_cidr}"
  vpc_peering_connection_id = "${aws_vpc_peering_connection.peer.id}"
}

Crash Output

Error: Unable to accept VPC Peering Connection: OperationNotPermitted: User <REDACTED> cannot accept peering <REDACTED>
        status code: 400, request id: <REDACTED>

  on ../../common/modules/vpc_peering/main.tf line 39, in resource "aws_vpc_peering_connection_accepter" "peer":
  39: resource "aws_vpc_peering_connection_accepter" "peer" {

Error: Error creating route: InvalidRouteTableID.NotFound: The routeTable ID 'rtb-<REDACTED>' does not exist
        status code: 400, request id: <REDACTED>

  on ../../common/modules/vpc_peering/main.tf line 52, in resource "aws_route" "peer_public_1":
  52: resource aws_route "peer_public_1" {

Error: Error creating route: InvalidRouteTableID.NotFound: The routeTable ID 'rtb-<REDACTED>' does not exist
        status code: 400, request id: <REDACTED>

  on ../../common/modules/vpc_peering/main.tf line 59, in resource "aws_route" "peer_private_1":
  59: resource aws_route "peer_private_1" {

Error: Error creating route: InvalidRouteTableID.NotFound: The routeTable ID 'rtb-<REDACTED>' does not exist
        status code: 400, request id: <REDACTED>

  on ../../common/modules/vpc_peering/main.tf line 66, in resource "aws_route" "peer_private_2":
  66: resource aws_route "peer_private_2" {

Expected Behavior

The VPC peering should be correctly set up.

Actual Behavior

Apparently the provider is not working.

The User (AWS account ID) in the first error is not the correct account ID for the prod provider.
Instead it’s showing the User of the requester, which are the AWS credentials loaded to the system environment variables, and not the ones passed through the provider profile.

Also, the route tables that the errors state not to exist, do in fact exist in the accepter account (which is supposed to be invoked through the provider statement), which leads me to believe it’s actually trying to find them with the requester’s account.

Steps to Reproduce

terraform apply

Additional Context

(1) I would like to advise here, that the prod_admin profile in question here is configured as such in the ~/.aws/config file:

profile [prod_admin]
role_arn = arn:aws:iam::<REDACTED>:role/admin
source_profile = prod

It is in the ~/.aws/credentials file that the prod profile credentials are described.

(2) After manually accepting the VPC Peering in the AWS Console, only the route table not found errors persist, therefore I tried to import those aws_route resources, but with o avail:

$ terraform import -provider=aws.prod module.vpc_peering.aws_route.peer_public_1 rtb-<REDACTED>_10.13.0.0/16

module.vpc_peering.aws_route.peer_public_1: Importing from ID "rtb-<REDACTED>_10.13.0.0/16"...
module.vpc_peering.aws_route.peer_public_1: Import complete!
  Imported aws_route
module.vpc_peering.aws_route.peer_public_1: Refreshing state... [id=r-rtb-<REDACTED>]

Error: Cannot import non-existent remote object

While attempting to import an existing object to aws_route.peer_public_1, the
provider detected that no object exists with the given id. Only pre-existing
objects can be imported; check that the id is correct and that it is
associated with the provider's configured region or endpoint, or use
"terraform apply" to create a new remote object for this resource.

Observation 1: Note the extra “r-” in front of the route table id [id=r-rtb-<REDACTED>] when terraform tries to refresh the state.

Observation 2: Although the Imported aws_route message appears in green, when trying to apply the script again, the same 3 errors appear.

Hello Gio!

I wasn’t sure if you resolved this yet but I wanted to follow up. I tried your configuration and provider versions to triage and see if I could replicate the issue.

First, I checked my assume role configuration on what would be the equivalent of the prod account in your configuration. Below is my policy for the role, which is the same rules as AmazonVPCFullAccess.

resource "aws_iam_policy" "triage" {
  name = "TriageTestPolicy"

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
      {
            "Effect": "Allow",
            "Action": [
                "ec2:AcceptVpcPeeringConnection",
                "ec2:AcceptVpcEndpointConnections",
                "ec2:AllocateAddress",
                "ec2:AssignIpv6Addresses",
                "ec2:AssignPrivateIpAddresses",
                "ec2:AssociateAddress",
                "ec2:AssociateDhcpOptions",
                "ec2:AssociateRouteTable",
                "ec2:AssociateSubnetCidrBlock",
                "ec2:AssociateVpcCidrBlock",
                "ec2:AttachClassicLinkVpc",
                "ec2:AttachInternetGateway",
                "ec2:AttachNetworkInterface",
                "ec2:AttachVpnGateway",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateCustomerGateway",
                "ec2:CreateDefaultSubnet",
                "ec2:CreateDefaultVpc",
                "ec2:CreateDhcpOptions",
                "ec2:CreateEgressOnlyInternetGateway",
                "ec2:CreateFlowLogs",
                "ec2:CreateInternetGateway",
                "ec2:CreateNatGateway",
                "ec2:CreateNetworkAcl",
                "ec2:CreateNetworkAcl",
                "ec2:CreateNetworkAclEntry",
                "ec2:CreateNetworkInterface",
                "ec2:CreateNetworkInterfacePermission",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSecurityGroup",
                "ec2:CreateSubnet",
                "ec2:CreateTags",
                "ec2:CreateVpc",
                "ec2:CreateVpcEndpoint",
                "ec2:CreateVpcEndpointConnectionNotification",
                "ec2:CreateVpcEndpointServiceConfiguration",
                "ec2:CreateVpcPeeringConnection",
                "ec2:CreateVpnConnection",
                "ec2:CreateVpnConnectionRoute",
                "ec2:CreateVpnGateway",
                "ec2:DeleteCustomerGateway",
                "ec2:DeleteDhcpOptions",
                "ec2:DeleteEgressOnlyInternetGateway",
                "ec2:DeleteFlowLogs",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteNatGateway",
                "ec2:DeleteNetworkAcl",
                "ec2:DeleteNetworkAclEntry",
                "ec2:DeleteNetworkInterface",
                "ec2:DeleteNetworkInterfacePermission",
                "ec2:DeleteRoute",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteSubnet",
                "ec2:DeleteTags",
                "ec2:DeleteVpc",
                "ec2:DeleteVpcEndpoints",
                "ec2:DeleteVpcEndpointConnectionNotifications",
                "ec2:DeleteVpcEndpointServiceConfigurations",
                "ec2:DeleteVpcPeeringConnection",
                "ec2:DeleteVpnConnection",
                "ec2:DeleteVpnConnectionRoute",
                "ec2:DeleteVpnGateway",
                "ec2:DescribeAccountAttributes",
                "ec2:DescribeAddresses",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeClassicLinkInstances",
                "ec2:DescribeCustomerGateways",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeEgressOnlyInternetGateways",
                "ec2:DescribeFlowLogs",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeMovingAddresses",
                "ec2:DescribeNatGateways",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeNetworkInterfaceAttribute",
                "ec2:DescribeNetworkInterfacePermissions",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribePrefixLists",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroupReferences",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeStaleSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeTags",
                "ec2:DescribeVpcAttribute",
                "ec2:DescribeVpcClassicLink",
                "ec2:DescribeVpcClassicLinkDnsSupport",
                "ec2:DescribeVpcEndpointConnectionNotifications",
                "ec2:DescribeVpcEndpointConnections",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeVpcEndpointServiceConfigurations",
                "ec2:DescribeVpcEndpointServicePermissions",
                "ec2:DescribeVpcEndpointServices",
                "ec2:DescribeVpcPeeringConnections",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpnConnections",
                "ec2:DescribeVpnGateways",
                "ec2:DetachClassicLinkVpc",
                "ec2:DetachInternetGateway",
                "ec2:DetachNetworkInterface",
                "ec2:DetachVpnGateway",
                "ec2:DisableVgwRoutePropagation",
                "ec2:DisableVpcClassicLink",
                "ec2:DisableVpcClassicLinkDnsSupport",
                "ec2:DisassociateAddress",
                "ec2:DisassociateRouteTable",
                "ec2:DisassociateSubnetCidrBlock",
                "ec2:DisassociateVpcCidrBlock",
                "ec2:EnableVgwRoutePropagation",
                "ec2:EnableVpcClassicLink",
                "ec2:EnableVpcClassicLinkDnsSupport",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:ModifySubnetAttribute",
                "ec2:ModifyVpcAttribute",
                "ec2:ModifyVpcEndpoint",
                "ec2:ModifyVpcEndpointConnectionNotification",
                "ec2:ModifyVpcEndpointServiceConfiguration",
                "ec2:ModifyVpcEndpointServicePermissions",
                "ec2:ModifyVpcPeeringConnectionOptions",
                "ec2:ModifyVpcTenancy",
                "ec2:MoveAddressToVpc",
                "ec2:RejectVpcEndpointConnections",
                "ec2:RejectVpcPeeringConnection",
                "ec2:ReleaseAddress",
                "ec2:ReplaceNetworkAclAssociation",
                "ec2:ReplaceNetworkAclEntry",
                "ec2:ReplaceRoute",
                "ec2:ReplaceRouteTableAssociation",
                "ec2:ResetNetworkInterfaceAttribute",
                "ec2:RestoreAddressToClassic",
                "ec2:RevokeSecurityGroupEgress",
                "ec2:RevokeSecurityGroupIngress",
                "ec2:UnassignIpv6Addresses",
                "ec2:UnassignPrivateIpAddresses",
                "ec2:UpdateSecurityGroupRuleDescriptionsEgress",
                "ec2:UpdateSecurityGroupRuleDescriptionsIngress"
            ],
            "Resource": "*"
        }
    ]
}
EOF
}

I checked my policies on the requester side and made sure its Terraform user had AmazonVPCFullAccess as well.

I used the Role ARN and source credentials and added them to my ~/.aws/config, as you helpfully included.

Then, I used your configuration minus the routing table rules to check the peering connection. My configuration was cross-account and cross-region. The peering connection was successfully established.

The 400 error might be from a missing policy on the role, either from the user on the requester or accepter. Let me know if you find a root cause!

Associated Github Issue

@gio-salvador Have you tried setting the AWS_SDK_LOAD_CONFIG environment variable?
See https://github.com/terraform-providers/terraform-provider-aws/pull/8451.