Network fingerprinting calling my home router

jcalonso · December 18, 2020, 9:27am

Hello!,

I have a single node Nomad instance (v1.0.1) running on my home server. I was debugging some issues with Jobs not resolving Consul DNS properly and found an odd value in the unique.network.ip-address when looking in the clients UI info page:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type" /> <meta content="no-cache" http-equiv="Pragma" /> <title>Waiting...</title> <script type="text/javascript"> var pageName = '/'; top.location.replace(pageName); </script> </head> <body> </body> </html>

In fact, the same html appears in all unique.network.* values and platform.*

After some investigation, found that the HTML correspond to the default redirect page of my home router when hitting a non- existent endpoint. I believe the NetworkFingerprint some how ends calling my home router ip (192.168.1.254) to fetch some details.

Nomad startup logs show some more details:

Dec 18 09:03:45 nomadnode01 systemd[1]: Started Nomad Client.
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> Loaded configuration from /etc/nomad.d/client.hcl, /etc/nomad.d/server.hcl
Dec 18 09:03:45 nomadnode01 nomad[6979]: ==> Starting Nomad agent...
Dec 18 09:03:46 nomadnode01 nomad[6979]: ==> Nomad agent configuration:
Dec 18 09:03:46 nomadnode01 nomad[6979]:        Advertise Addrs: HTTP: 192.168.1.11:4646; RPC: 192.168.1.11:4647; Serf: 192.168.1.11:4648
Dec 18 09:03:46 nomadnode01 nomad[6979]:             Bind Addrs: HTTP: 192.168.1.11:4646; RPC: 192.168.1.11:4647; Serf: 192.168.1.11:4648
Dec 18 09:03:46 nomadnode01 nomad[6979]:                 Client: true
Dec 18 09:03:46 nomadnode01 nomad[6979]:              Log Level: DEBUG
Dec 18 09:03:46 nomadnode01 nomad[6979]:                 Region: global (DC: dc01)
Dec 18 09:03:46 nomadnode01 nomad[6979]:                 Server: true
Dec 18 09:03:46 nomadnode01 nomad[6979]:                Version: 1.0.1
Dec 18 09:03:46 nomadnode01 nomad[6979]: ==> Nomad agent started! Log data will stream in below:
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.775Z [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/lib/nomad/storage/server/plugins
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=/var/lib/nomad/storage/server/plugins
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.776Z [INFO]  agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.794Z [INFO]  nomad.raft: restored from snapshot: id=7-8351-1608257027253
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:192.168.1.11:4647 Address:192.168.1.11:4647}]"
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:46.469Z [INFO]  client.gc: marking allocation for GC: alloc_id=5042c56f-fa29-467d-e6ad-7be5e42e4a07
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad.raft: entering follower state: follower="Node at 192.168.1.11:4647 [Follower]" leader=
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad: serf: EventMemberJoin: client.global 192.168.1.11
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  nomad: starting scheduling worker(s): num_workers=4 schedulers=[service, batch, system, _core]
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [WARN]  nomad: serf: Failed to re-join any previously known node
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.811Z [INFO]  client: using state directory: state_dir=/var/lib/nomad/storage/server/client
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.812Z [INFO]  nomad: adding server: server="client.global (Addr: 192.168.1.11:4647) (DC: dc01)"
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.812Z [INFO]  client: using alloc directory: alloc_dir=/var/lib/nomad/storage/server/alloc
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=[arch, bridge, cgroup, cni, consul, cpu, host, memory, network, nomad, signal, storage, vault, env_azure, env_aws, env_gce]
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.813Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=cgroup period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [INFO]  client.fingerprint_mgr.consul: consul agent is available
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr.cpu: detected cpu frequency: MHz=3900
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.815Z [DEBUG] client.fingerprint_mgr.cpu: detected core count: cores=4
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: link speed detected: interface=enp0s31f6 mbits=100
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=192.168.1.11
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=2806:102e:8:2e65::9
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=2806:102e:8:2e65:e2d5:5eff:fe00:8c97
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.817Z [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=enp0s31f6 IP=fdac:6175:1a7e:dd00:e2d5:5eff:fe00:8c97
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.818Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=lo
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.818Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/lo/speed
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.818Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.819Z [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/sbin/ethtool device=docker0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.819Z [DEBUG] client.fingerprint_mgr.network: unable to read link speed: path=/sys/class/net/docker0/speed
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.819Z [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: mbits=1000
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.823Z [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault period=15s
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:45.858Z [DEBUG] consul.sync: sync complete: registered_services=3 deregistered_services=0 registered_checks=3 deregistered_checks=0
Dec 18 09:03:46 nomadnode01 nomad[6979]:     2020-12-18T09:03:46.025Z [DEBUG] client.fingerprint_mgr.env_aws: read an empty value: attribute="network/interfaces/macs/<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Dec 18 09:03:46 nomadnode01 nomad[6979]: <html xmlns="http://www.w3.org/1999/xhtml">
Dec 18 09:03:46 nomadnode01 nomad[6979]: <head>
Dec 18 09:03:46 nomadnode01 nomad[6979]: <meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
Dec 18 09:03:46 nomadnode01 nomad[6979]: <meta content="no-cache" http-equiv="Pragma" />
Dec 18 09:03:46 nomadnode01 nomad[6979]: <title>Waiting...</title>
Dec 18 09:03:46 nomadnode01 nomad[6979]: <script type="text/javascript">
Dec 18 09:03:46 nomadnode01 nomad[6979]: var pageName = '/';
Dec 18 09:03:46 nomadnode01 nomad[6979]: top.location.replace(pageName);
Dec 18 09:03:46 nomadnode01 nomad[6979]: </script>
Dec 18 09:03:46 nomadnode01 nomad[6979]: </head>
Dec 18 09:03:46 nomadnode01 nomad[6979]: <body> </body>
Dec 18 09:03:46 nomadnode01 nomad[6979]: </html>/ipv6s"

In the last line you can see its interpolating what it think its the mac address in here:

github.com

hashicorp/nomad/blob/d3a51279af4c8303cc64a311143c909e2732f5b8/client/fingerprint/env_aws.go#L153


				Device: "eth0",
				IP:     val,
				CIDR:   val + "/32",
				MBits:  f.throughput(request, ec2meta, val),
			},
		}
	}
	// copy over IPv6 network specific information
	if val, ok := response.Attributes["unique.platform.aws.mac"]; ok && val != "" {
		k := "network/interfaces/macs/" + val + "/ipv6s"
		addrsStr, err := ec2meta.GetMetadata(k)
		addrsStr = strings.TrimSpace(addrsStr)
		if addrsStr == "" {
			f.logger.Debug("read an empty value", "attribute", k)
		} else if awsErr, ok := err.(awserr.RequestFailure); ok {
			f.logger.Debug("could not read attribute value", "attribute", k, "error", awsErr)
		} else if awsErr, ok := err.(awserr.Error); ok {
			// if it's a URL error, assume we're not in an AWS environment
			// TODO: better way to detect AWS? Check xen virtualization?
			if _, ok := awsErr.OrigErr().(*url.Error); ok {

to obtain the ipv6 address.

I tried disabling ipv6 but the problem persist.

A wild guess is that the fingerprinting it’s using the first address in my server routes to gather information, then it calls a non- existent endpoint and my router returns a 200 OK and the Html page pasted above.

Example:

curl http://192.168.1.254/im-a-dumb-home-router -v
*   Trying 192.168.1.254...
* TCP_NODELAY set
* Connected to 192.168.1.254 (192.168.1.254) port 80 (#0)
> GET /im-a-dumb-home-router HTTP/1.1
> Host: 192.168.1.254
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-control:no-cache, no-store, max-age=0
< Content-Type:text/html
< Pragma:no-cache
< Transfer-Encoding:chunked
< X-Frame-Options:SAMEORIGIN
< Connection:Keep-Alive
<
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type" />
<meta content="no-cache" http-equiv="Pragma" />
<title>Waiting...</title>
<script type="text/javascript">
var pageName = '/';
top.location.replace(pageName);
</script>
</head>
<body> </body>
</html>
* Connection #0 to host 192.168.1.254 left intact

(200 status code when it should be a 404)

Any workaround that doesn’t involve changing the router ?

Thanks!

jcalonso · December 18, 2020, 10:16am

@apollo13 from Gitter, recommended adding the envFingerprinters to the client denyList.

Like:

options = {
  "fingerprint.denylist" = "env_aws,env_gce,env_azure"
 }

Now Nomad its able to determine the ip-address correctly.

tgross · December 18, 2020, 5:00pm

@jcalonso and @apollo13 have the right workaround. That being said, blasting the HTML into the logs like that is definitely a bug in my mind. Would you mind filing a GitHub issue describing this?

jcalonso · December 19, 2020, 6:06am

No problem!,