I’ve been struggling with App Mesh failing to discover instances that I’ve put in a new consolidated private DNS namespace (after standing up the new namespace in parallel and migrating services to it from a collection of other namespaces). It says the service doesn’t exist. It works fine everywhere else I’ve used it, and yesterday I discovered why.
In my Terraform, I’ve been passing the name attribute from my aws_service_discovery_private_dns_namespace resource to my aws_appmesh_virtual_node resource. App Mesh uses the supplied value to do a DiscoverInstances call, which expects the HttpName ( not the Name ) of the namespace (see DiscoverInstances - AWS Cloud Map). Usually both of those attributes are the same, however, if you create two namespaces of different types with the same name, for example if you need to have both a public and private DNS namespace, or (as in our case) you need to transition from one namespace type to another and want to keep the same name, when you create the second namespace, AWS will append a unique value to the end of the HttpName attribute on each subsequent namespace to differentiate it from the first namespace when you’re calling DiscoverInstances. You can see this if you call ListNamespaces. Here’s an example of some redacted output hilighting the differences:
{
"Namespaces": [
{
"Id": "ns-xxxxxxxxxxxxxxxx",
"Arn": "arn:aws:servicediscovery:us-east-1:000000000000:namespace/ns-xxxxxxxxxxxxxxxx",
"Name": "example1",
"Type": "DNS_PRIVATE",
"Description": "Cloud Map HTTP+DNS namespace for example1 environment",
"Properties": {
"DnsProperties": {
"HostedZoneId": "Z00000000000000000000"
},
"HttpProperties": {
"HttpName": "example1-vxykwuw5nibpyamb"
}
},
"CreateDate": 1634147723.302
},
{
"Id": "ns-xxxxxxxxxxxxxxxx",
"Arn": "arn:aws:servicediscovery:us-east-1:000000000000:namespace/ns-xxxxxxxxxxxxxxxx",
"Name": "example1-service-mesh",
"Type": "HTTP",
"Description": "Cloud Map HTTP namespace for example1 environment",
"Properties": {
"DnsProperties": {},
"HttpProperties": {
"HttpName": "example1-service-mesh"
}
},
"CreateDate": 1633463863.434
}
}
}
This would normally be fine - if you’re passing the namespace name into a configuration to an app that’s going to call DiscoverInstances , you would give it the HttpName and all would work properly. Only issue is that none of the Cloud Map namespace resources in Terraform expose that attribute.
Just debating a few different ways to solve this and was hoping for input:
I could submit a bug report, but I don’t know if this necessarily counts as a bug. It’s definitely a shortcoming which can affect you if you aren’t aware of it. It wouldn’t be a huge amount of work to just add HttpName as another attribute, the same as HostedZoneId is exposed as hosted_zone on both of the DNS-type namespace resources, but I’m not sure if it’s worth the effort or if it would just make things more confusing. Ideally if we’re passing a namespace name to a service that’s going to consume it via the HTTP discovery API, we would want to give it the HttpName attribute value, since that’s the one it’s going to use.
I could also just submit a documentation PR to clarify and let people know this is a pitfall to be aware of - even though both the Terraform Provider and AWS are perfectly happy to let you create namespaces with the same name but different type, perhaps that’s something you just shouldn’t be doing at all, not even temporarily.
Or, I could submit a feature request that some form of pre-validation be added to prevent you from creating a namespace if one with the same name already exists.
Considering it took me multiple days of research and testing (and eventually trial and error to find the root cause, as the documentation available online was lacking), I’m hoping to prevent others from having to go through the same headaches.
How should I proceed on this?