It sounds like the downtime you are seeing is the time taken for the standby to detect the active server is no longer around, handle any leadership election, becoming active and then updating Consul with that detail (assuming you are using Consul for service discovery).
There will always be some level of delay for this (and in a production failure situation it could be a bit longer for the standby to detect that the active node has failed) but it should be pretty short.