Error core: failed to lookup token: error=failed to read entry, dial tcp [::1]:8500: getsockopt: connection refused

Vault
,
1

Hi All,

We are performing load test on our application using Jmeter, our application uses consul and vault as a backend service for reading/storing application configuration related data. While performing load testing, our application queries the vault for authentication data and this happens for each incoming request. Initially it runs fine for some duration (10 to 15 minutes) and I can see the success response in Jmete, but eventually after sometime the responses starts failing for all the requests. I see the following error in the vault log for each request but do not see any error/exception in the consul log.

Error in Vault log

[ERROR] core: failed to lookup token: error=failed to read entry: Get http://localhost:8500/v1/kv/<application>/vault/sys/token/id/87f7b82131cb8fa1ef71aa52579f155d4cf9f095: dial tcp [::1]:8500: getsockopt: connection refused

As of now the load is 100 request (users) in each 10 milliseconds with a ramp-up period of 60 seconds. And this executes over a loop. What could be the cause of this error? Is it due to the limited connection to port 8500

Below is my vault and consul configuration
Vault

backend "consul" {
  address = "localhost:8500"
  path = "app/vault/"
}

listener "tcp" {
 address = "10.88.97.216:8200"
 cluster_address = "10.88.97.216:8201"
 tls_disable = 0
 tls_min_version = "tls12"
 tls_cert_file = "/var/certs/vault.crt"
 tls_key_file = "/var/certs/vault.key"
}

Consul

{
  "data_dir": "/var/consul",
  "log_level": "info",
  "server": true,
  "leave_on_terminate": true,
  "ui": true,
  "client_addr": "127.0.0.1",
  "ports": {
    "dns": 53,
    "serf_lan": 8301,
    "serf_wan" : 8302
  },
  "disable_update_check": true,
  "enable_script_checks": true,
  "disable_remote_exec": false,
  "domain": "primehome",
  "limits": {
    "http_max_conns_per_client": 1000,
    "rpc_max_conns_per_client": 1000
  },

  "service": {
      "name": "nginx-consul-https",
      "port": 443,
      "checks": [{
			"http": "https://localhost/nginx_status",
			"tls_skip_verify": true,
            "interval": "10s",
            "timeout": "5s",
            "status": "passing"
      }]
  }
}

I have also configured the http_max_conns_per_client & rpc_max_conns_per_client, thinking that it might be due to the limited connection perclicent. But still I am seeing this error in vault log.

2

What is the actual testing plan, APIs, parameters that you’re using?