Client initially connects using server DNS but then retries with IP address of server

I’m working in AWS where resources are ephemeral and IP addresses are dynamically allocated (not static). The issue we’re experiencing is that clients do not reconnect with the server if the server is restarted and gets a new IP address. I can see from the client logs that the initial connection with the server is made via the server’s DNS name but when the client tries to reconnect it uses the IP address. Our setup has a load balancer with the DNS name between the clients and the server. The DNS name is associated with the load balancer yet the clients still end up with the actual IP address of the server instance.

Is there any way to configure/force the client to always use the DNS name rather than the IP address? This is the preferred solution. I’ve tried many consul settings to get this type of behavior but have had no luck.

For reference, other solutions I’ve tried include:

  • Configuring a static IP on the nomad server – in a cloud environment this is problematic due to the ephemeral nature of compute resources and is definitely not the prefered solution
  • Setting DNS TTL to short setting – this is unreliable and results in waiting for clients to eventually show up. Also, not all OSes handle DNS TTL the same.
  • Configuring a 2nd IP address (EIN) on the server’s EC2 instance – the server was not happy binding to the 2nd IP address and this resulted in a slew of errors

The solution I’m using for now is to restart the client server on all nodes after IP-changes occur. This works but is obviously suboptimal.