Error with nginx as reverse proxy

Consul
1

Hello,

I encountered this error messages in nginx as reverse proxy with consul:

2022/01/13 16:36:52 [error] 741#741: *425422 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: consul.service.xxxx, request: "GET /v1/kv/terraform/os-integ-manage-tenants.tfstate/.lock?consistent=&index=17011479 HTTP/1.1", upstream: "https://127.0.0.1:8501/v1/kv/terraform/os-integ-manage-tenants.tfstate/.lock?consistent=&index=17011479", host: "consul.service.xxxx"
2022/01/13 16:37:59 [error] 741#741: *425430 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: consul.service.xxxx, request: "GET /v1/kv/terraform/os-integ-manage-tenants.tfstate/.lock?consistent=&index=17011500 HTTP/1.1", upstream: "https://127.0.0.1:8501/v1/kv/terraform/os-integ-manage-tenants.tfstate/.lock?consistent=&index=17011500", host: "consul.service.xxxx"
2022/01/13 16:39:07 [error] 741#741: *425422 upstream timed out (110: Connection timed out) while reading response header from upstream, client: XX.XX.XX.XX, server: consul.service.xxxx, request: "GET /v1/kv/terraform/os-integ-manage-tenants.tfstate/.lock?consistent=&index=17011524 HTTP/1.1", upstream: "https://127.0.0.1:8501/v1/kv/terraform/os-integ-manage-tenants.tfstate/.lock?consistent=&index=17011524", host: "consul.service.xxxx"
2022/01/13 16:47:18 [warn] 741#741: *425901 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/8/15/0000000158 while reading upstream, client: XX.XX.XX.XX, server: consul.service.xxxx, request: "GET /v1/kv/terraform/os-integ-manage-tenants.tfstate HTTP/1.1", upstream: "https://127.0.0.1:8501/v1/kv/terraform/os-integ-manage-tenants.tfstate", host: "consul.service.xxxx"
2022/01/13 16:47:20 [warn] 741#741: *425901 an upstream response is buffered to a temporary file /var/cache/nginx/proxy_temp/9/15/0000000159 while reading upstream, client: XX.XX.XX.XX, server: consul.service.xxxx, request: "GET /v1/kv/terraform/os-integ-manage-tenants.tfstate HTTP/1.1", upstream: "https://127.0.0.1:8501/v1/kv/terraform/os-integ-manage-tenants.tfstate", host: "consul.service.xxxx"
2022/01/13 16:47:20 [warn] 741#741: *425901 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000160, client: XX.XX.XX.XX, server: consul.service.xxxx, request: "PUT /v1/txn HTTP/1.1", host: "consul.service.xxxx"
2022/01/13 16:47:20 [error] 741#741: *425901 SSL_write() failed (32: Broken pipe) while sending request to upstream, client: XX.XX.XX.XX, server: consul.service.xxxx, request: "PUT /v1/txn HTTP/1.1", upstream: "https://127.0.0.1:8501/v1/txn", host: "consul.service.xxxx"

In my nginx config, I have:

location / {
      add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

      proxy_connect_timeout     600;
      proxy_send_timeout        600;
      proxy_read_timeout        600;
      send_timeout              600;

      proxy_buffering off;
      proxy_pass https://127.0.0.1:8501;
    }

Any idea of the issue ?
I am using consul 1.9.9.

Thanks.

2

Nothing related to Nginx in fact. My tfstate size was greater than the default limit (512ko).

I increased the limit in the consul config with:

"limits": {
  "kv_max_value_size": 1024000,
  "txn_max_req_len": 1024000
},
1 Like
3

Thank you for posting the solution, @smutel !