Nomad UI behind nginx load balancer, no logs

Nomad
1

I have 3 nomad servers running the UI that I want to put behind an nginx load balancer.

The following works except I am unable to stream any logs in the browser (streaming logs from the nomad cli works fine; bypassing nginx and hitting host1:4646 also works fine).

47
471908×722 26.6 KB

In the nginx logs I see this error:

2019/08/22 16:15:08 [error] 19249#19249: *3 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.17.30.18, server: proxy-sb-sand-nomad.example.com, request: “GET /v1/allocation/b9cc5171-3b96-aa29-9568-9dc187dc174c?index=51191 HTTP/1.1”, upstream: “https://10.47.80.6:4646/v1/allocation/b9cc5171-3b96-aa29-9568-9dc187dc174c?index=51191”, host: “nomad.example.com”, referrer: “https://nomad.example.com/ui/allocations/b9cc5171-3b96-aa29-9568-9dc187dc174c/fabio/logs”

nginx config

server {
  listen       *:443 ssl;
  server_name  nomad.example.com;
  ssl on;
  ssl_certificate           /etc/ssl/certs/xxxxx;
  ssl_certificate_key       /etc/ssl/private/xxxxx;
  ssl_session_cache         shared:SSL:10m;
  ssl_session_timeout       5m;
  ssl_protocols             TLSv1.2;
  ssl_ciphers               xxxxxxx
  client_max_body_size 10m;
  access_log            /var/log/nginx/ngproxy.access.log mdformat;
  error_log             /var/log/nginx/ngproxy.error.log;
  add_header              Strict-Transport-Security max-age=31536000;

  location / {
    proxy_pass            https://nomad-upstreams;
    proxy_read_timeout    90s;
    proxy_connect_timeout 1s;
    proxy_redirect        off;
    proxy_set_header      Host $http_host;
    proxy_set_header      X-Forwarded-Proto $scheme;
    proxy_set_header      X-Forwarded-Port $server_port;
    proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header      Upgrade $http_upgrade;
    proxy_set_header      Connection upgrade;

  }
  access_log syslog:server=127.0.0.1,facility=local1 logstash;
}

Has anyone else successfully got nomad working behind an nginx load balancer?

2

Hi @spuder,

Looks like you have everything configured well except for one detail:

proxy_buffering needs to be set to off

This is because logging and file system requests require HTTP streaming and when proxy_buffering is enabled, NGINX waits for a complete response.

This is also something Nomad can handle automatically in the future via the X-Accel-Buffer response header.

Hope this fixes things for you!

1 Like