Run-consul is slow in user data for Centos 7

Consul
1

I have a slowdown when trying to use run-consul in user data on centos 7 in AWS.

Because the script contains sudo commands, for some reason these commands are very slow and I’m not sure how to resolve it. Each time sudo runs its about 30 seconds, and the run-consul script takes 1:30 because of this, it contains three sudo commands.

I only observe this slowness in user data scripts, and perhaps that is just because the hostname hasn’t updated yet for the current session. Perhaps it may be as simple as restarting some service?

I am aware centos 7 has problems with sudo slowing down related to DNS and SELINUX, but the selinux package is current and I do add the correct hostname to /etc/hosts in the user data just before I start run-consul.

[root@ip-10-4-102-177 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1 ip-10-4-102-177.ap-southeast-2.compute.internal ip-10-4-102-177

This is a snippet of the user data script that exhibits the problem:

#!/bin/bash
# This script is meant to be run in the User Data of each EC2 Instance while it's booting. The script uses the
# run-consul script to configure and start Consul in client mode. Note that this script assumes it's running in an AMI
# built from the Packer template in examples/vault-consul-ami/vault-consul.json.

set -e

# Send the log output from this script to user-data.log, syslog, and the console
# From: https://alestic.com/2010/12/ec2-user-data-output/
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

# Log the given message. All logs are written to stderr with a timestamp.
function log {
 local -r message="$1"
 local -r timestamp=$(date +"%Y-%m-%d %H:%M:%S")
 >&2 echo -e "$timestamp $message"
}

function has_yum {
  [[ -n "$(command -v yum)" ]]
}

if $(has_yum); then
    hostname=$(hostname -s) # in centos, failed dns lookup can cause commands to slowdown
    echo "127.0.0.1 $hostname.${aws_domain} $hostname" | tee -a /etc/hosts
    hostnamectl set-hostname $hostname.${aws_domain} # Red hat recommends that the hostname uses the FQDN.  hostname -f to resolve the domain may not work at this point on boot, so we use a var.
fi

log "hostname: $(hostname)"
log "hostname: $(hostname -f) $(hostname -s)"

# These variables are passed in via Terraform template interpolation
/opt/consul/bin/run-consul --client --cluster-tag-key "${consul_cluster_tag_key}" --cluster-tag-value "${consul_cluster_tag_value}"

Strangely when I ssh in and use sudo, I don’t observe any slowdowns.