My Vault installation(1.3.0) isn’t emitting metrics. Given below is my config stored under /etc/vault.d/telemetry.json. How I can debug the issue?
{
"telemetry": {
"statsd_address": "127.0.0.1:8125",
"disable_hostname": true
}
}
Please note that I have statsd server running on localhost:8125 using Telegraf StatsD input plugin, and routing the statsd output to a file using Telegraf File output plugin, and I don’t see any telemetry data in the output. Other metrics (e.g. CPU) are showing fine.
I even tried dogstatsd_addr but didn’t work either.
Given below is Telegraf config:
[[inputs.statsd]]
protocol = "udp"
service_address = ":8125"
delete_gauges = true
delete_counters = true
delete_sets = true
delete_timings = true
percentiles = [50.0, 90.0, 99.0, 99.9, 99.95, 100.0]
metric_separator = "_"
datadog_extensions = true
allowed_pending_messages = 10000
percentile_limit = 1000
[[outputs.file]]
files = ["stdout", "/var/log/metrics.out"]
data_format = "wavefront"
Logs from Journelctl
Jan 05 19:20:43 device-sampling-deep-dev-csl-3 systemd[1]: Stopping "Tool for managing secrets"...
Jan 05 19:20:43 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01-05T19:20:43.997Z [INFO] storage.consul: shutting down consul backend
Jan 05 19:20:44 device-sampling-deep-dev-csl-3 vault[35528]: ==> Vault shutdown triggered
Jan 05 19:20:44 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01-05T19:20:44.041Z [INFO] core: marked as sealed
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01-05T19:20:51.351Z [INFO] core: stopping cluster listeners
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01-05T19:20:51.351Z [INFO] core.cluster-listener: forwarding rpc listeners stopped
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01- 05T19:20:51.490Z [INFO] core.cluster-listener: rpc listeners successfully shut down
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01-05T19:20:51.490Z [INFO] core: cluster listeners successfully shut down
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01-05T19:20:51.493Z [WARN] storage.consul: concurrent sealed state change notify dropped
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[35528]: 2020-01-05T19:20:51.494Z [INFO] core: vault is sealed
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 systemd[1]: Stopped "Tool for managing secrets".
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 systemd[1]: Started "Tool for managing secrets".
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: ==> Vault server configuration:
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Api Address: http://10.84.36.231:8200
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Cgo: disabled
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Cluster Address: https://10.84.36.231:8201
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Log Level: info
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Mlock: supported: true, enabled: false
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Recovery Mode: false
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Storage: consul (HA available)
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: Version: Vault v1.3.0
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: ==> Vault server started! Log data will stream in below:
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: 2020-01-05T19:20:51.769Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: 2020-01-05T19:20:51.769Z [WARN] storage.consul: appending trailing forward slash to path
Jan 05 19:20:51 device-sampling-deep-dev-csl-3 vault[39499]: 2020-01-05T19:20:51.771Z [WARN] no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
Jan 05 19:21:25 device-sampling-deep-dev-csl-3 vault[39499]: 2020-01-05T19:21:25.804Z [INFO] core.cluster-listener: starting listener: listener_address=0.0.0.0:8201
Jan 05 19:21:25 device-sampling-deep-dev-csl-3 vault[39499]: 2020-01-05T19:21:25.804Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
Jan 05 19:21:25 device-sampling-deep-dev-csl-3 vault[39499]: 2020-01-05T19:21:25.804Z [INFO] core: vault is unsealed
Jan 05 19:21:25 device-sampling-deep-dev-csl-3 vault[39499]: 2020-01-05T19:21:25.804Z [INFO] core: entering standby mode