Thanks @salaxander!
We can generalize that solution to work for any number of appgw_dns_records elements by using a for expression:
[for r in local.appgw_dns_records : "${r.name}.${r.zone_name}"]
You can use an expression like the above in the place where you need the list of hostnames, and its result value should be the same as what @JamesStanley showed in the initial example.