Why might a consul client not be able to access Vault UI at https://vault.service.consul/ui

I have been learning how to use a Vault OSS HA cluster thanks to these examples https://github.com/hashicorp/terraform-aws-vault

I have an HA Vault cluster working with Consul HA storage, S3 backend for persistance, and KMS auto unseal in an AWS private subnet in the configuration.

I’m using an M4 Large NICE DCV graphical instance on the public subnet to interact with the Vault UI in firefox which works well.

In the browser I can only access the Vault UI with an IP address though, not with for example https://vault.service.consul/ui

I’d like to get this working, since it seems necessary for OIDC auth redirect to work after authenticating.

This graphical instance is running Amazon Linux 2, and has dnsmasq and consul client installed. It seems to show up as a member just fine.

[ec2-user@ip-10-4-101-54 ~]$ consul members
Node                 Address           Status  Type    Build  Protocol  DC              Segment
i-01f607c721c234874  10.4.2.232:8301   alive   server  1.8.4  2         ap-southeast-2  <all>
i-0bdb564f86febe426  10.4.2.20:8301    alive   server  1.8.4  2         ap-southeast-2  <all>
i-0f9ec9fc37771622d  10.4.1.203:8301   alive   server  1.8.4  2         ap-southeast-2  <all>
i-03c6f2923c30b3fa5  10.4.1.194:8301   alive   client  1.8.4  2         ap-southeast-2  <default>
i-0953fbd6c53420719  10.4.2.240:8301   alive   client  1.8.4  2         ap-southeast-2  <default>
i-0dc315cd30736ad8c  **10.4.101.54:8301**  alive   client  1.8.4  2         ap-southeast-2  <default>
i-0edae3fed04e49e8a  10.4.1.131:8301   alive   client  1.8.4  2         ap-southeast-2  <default>

But using dig doesn’t seem to aquire vault.service.consul

[ec2-user@ip-10-4-101-54 ~]$ dig vault.service.consul

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.amzn2.0.4 <<>> vault.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 4769
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;vault.service.consul.          IN      A

;; AUTHORITY SECTION:
consul.                 0       IN      SOA     ns.consul. hostmaster.consul. 1605440976 3600 600 86400 0

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Nov 15 11:49:36 UTC 2020
;; MSG SIZE  rcvd: 99

What kind of steps should I go through to ensure everything is configured correctly for consul client on this graphical bastion host to resolve the cluster Ip from the dns name?

I should also share the vault status if that provides any clues:

ubuntu@ip-10-4-1-131:~$ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.5.5
Cluster Name             vault-cluster-541caf47
Cluster ID               431693e2-4157-b83c-12b2-83dbf575ddc9
HA Enabled               true
HA Cluster               https://10.4.2.240:8201
HA Mode                  standby
Active Node Address      https://10.4.2.240:8200

Thanks if you can provide any clues one what I should pay attention to to get the DNS to resolve.

I updated part of the solution here:

I can browse to https://vault.service.consul:8200/ui however firefox still has security warnings that I have to add an exception to. I wouldn’t think this should be necesary since the ami that built the graphical host with a web browser ran the command to update the certificate store:

sudo /tmp/terraform-aws-vault/modules/update-certificate-store/update-certificate-store --cert-file-path /opt/vault/tls/ca.crt.pem

Perhaps this is unrelated to the web browser’s certificate store, but it should use the os certificate store.