Why might a consul client not be able to access Vault UI at https://vault.service.consul/ui

I have been learning how to use a Vault OSS HA cluster thanks to these examples https://github.com/hashicorp/terraform-aws-vault

I have an HA Vault cluster working with Consul HA storage, S3 backend for persistance, and KMS auto unseal in an AWS private subnet in the configuration.

I’m using an M4 Large NICE DCV graphical instance on the public subnet to interact with the Vault UI in firefox which works well.

In the browser I can only access the Vault UI with an IP address though, not with for example https://vault.service.consul/ui

I’d like to get this working, since it seems necessary for OIDC auth redirect to work after authenticating.

This graphical instance is running Amazon Linux 2, and has dnsmasq and consul client installed. It seems to show up as a member just fine.

[ec2-user@ip-10-4-101-54 ~]$ consul members
Node                 Address           Status  Type    Build  Protocol  DC              Segment
i-01f607c721c234874   alive   server  1.8.4  2         ap-southeast-2  <all>
i-0bdb564f86febe426    alive   server  1.8.4  2         ap-southeast-2  <all>
i-0f9ec9fc37771622d   alive   server  1.8.4  2         ap-southeast-2  <all>
i-03c6f2923c30b3fa5   alive   client  1.8.4  2         ap-southeast-2  <default>
i-0953fbd6c53420719   alive   client  1.8.4  2         ap-southeast-2  <default>
i-0dc315cd30736ad8c  ****  alive   client  1.8.4  2         ap-southeast-2  <default>
i-0edae3fed04e49e8a   alive   client  1.8.4  2         ap-southeast-2  <default>

But using dig doesn’t seem to aquire vault.service.consul

[ec2-user@ip-10-4-101-54 ~]$ dig vault.service.consul

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.amzn2.0.4 <<>> vault.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4769
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

; EDNS: version: 0, flags:; udp: 4096
;vault.service.consul.          IN      A

consul.                 0       IN      SOA     ns.consul. hostmaster.consul. 1605440976 3600 600 86400 0

;; Query time: 1 msec
;; WHEN: Sun Nov 15 11:49:36 UTC 2020
;; MSG SIZE  rcvd: 99

What kind of steps should I go through to ensure everything is configured correctly for consul client on this graphical bastion host to resolve the cluster Ip from the dns name?

I should also share the vault status if that provides any clues:

ubuntu@ip-10-4-1-131:~$ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.5.5
Cluster Name             vault-cluster-541caf47
Cluster ID               431693e2-4157-b83c-12b2-83dbf575ddc9
HA Enabled               true
HA Cluster     
HA Mode                  standby
Active Node Address

Thanks if you can provide any clues one what I should pay attention to to get the DNS to resolve.

I updated part of the solution here:

I can browse to https://vault.service.consul:8200/ui however firefox still has security warnings that I have to add an exception to. I wouldn’t think this should be necesary since the ami that built the graphical host with a web browser ran the command to update the certificate store:

sudo /tmp/terraform-aws-vault/modules/update-certificate-store/update-certificate-store --cert-file-path /opt/vault/tls/ca.crt.pem

Perhaps this is unrelated to the web browser’s certificate store, but it should use the os certificate store.