We are noticing an issue with the waypoint-entrypoint where it always exits with an exit code 2 and does not seem to be propagating the SIGTERM to the process that it is executing. This surfaces in Nomad with the following error when stopping an allocation:
Terminated Exit Code: 2, Exit Message: “Docker container exited with non-zero exit code: 2”
I’ve been playing a bit with this on my local machine and have managed to reproduce this with just docker. Here is an example of running without the waypoint-entrypoint and the exit is graceful:
» docker run -it --rm --name demo_django_web ${DEMO_IMAGE} /app/scripts/web-manual.sh
[2023-04-10 14:46:42 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-04-10 14:46:42 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2023-04-10 14:46:42 +0000] [1] [INFO] Using worker: gthread
[2023-04-10 14:46:42 +0000] [7] [INFO] Booting worker with pid: 7
[2023-04-10 14:46:43 +0000] [8] [INFO] Booting worker with pid: 8
[2023-04-10 14:46:46 +0000] [1] [INFO] Handling signal: term
{"name": "gunicorn.error", "processName": "MainProcess", "threadName": "MainThread", "levelname": "INFO", "message": "Worker exiting (pid: 7)", "timestamp": "2023-04-10T14:46:47.182976+00:00"}
{"name": "gunicorn.error", "processName": "MainProcess", "threadName": "MainThread", "levelname": "INFO", "message": "Worker exiting (pid: 7)", "timestamp": "2023-04-10T14:46:47.182976+00:00"}
{"name": "gunicorn.error", "processName": "MainProcess", "threadName": "MainThread", "levelname": "INFO", "message": "Worker exiting (pid: 8)", "timestamp": "2023-04-10T14:46:47.213390+00:00"}
{"name": "gunicorn.error", "processName": "MainProcess", "threadName": "MainThread", "levelname": "INFO", "message": "Worker exiting (pid: 8)", "timestamp": "2023-04-10T14:46:47.213390+00:00"}
[2023-04-10 14:46:47 +0000] [1] [INFO] Shutting down: Master
» echo $?
0
As you can see, the gunicorn process recieves the SIGTERM, exits gracefully and the exit code is 0.
If I run that exact same process, but with a waypoint-entrypoint, I see the following:
» docker run -it --rm --name demo_django_web ${DEMO_IMAGE} /app/waypoint-entrypoint /app/scripts/web-manual.sh
2023-04-10T14:47:43.359Z [INFO] entrypoint: entrypoint starting: deployment_id="" instance_id=01GXNT65ZZAZRWHX65ZQR9YCB2 args=["/app/scripts/web-manual.sh"]
2023-04-10T14:47:43.359Z [INFO] entrypoint: entrypoint version: full_string=v0.11.0 version=v0.11.0 prerelease="" metadata="" revision=""
2023-04-10T14:47:43.359Z [DEBUG] entrypoint.child: waiting for stateChildReady to flip to true
2023-04-10T14:47:43.359Z [DEBUG] entrypoint.child: starting child command watch loop
2023-04-10T14:47:43.359Z [DEBUG] entrypoint.child: child command received
2023-04-10T14:47:43.359Z [INFO] entrypoint.child: starting child process: args=["/app/scripts/web-manual.sh"] cmd=/app/scripts/web-manual.sh
[2023-04-10 14:47:43 +0000] [16] [INFO] Starting gunicorn 20.1.0
[2023-04-10 14:47:43 +0000] [16] [INFO] Listening at: http://0.0.0.0:8000 (16)
[2023-04-10 14:47:43 +0000] [16] [INFO] Using worker: gthread
[2023-04-10 14:47:43 +0000] [17] [INFO] Booting worker with pid: 17
[2023-04-10 14:47:43 +0000] [18] [INFO] Booting worker with pid: 18
» echo $? 2 ↵
2
This time, the gunicorn process does not receive the SIGTERM and the exit code for the container is 2. Interestingly, the container shuts down almost immediately so it’s not waiting for the web process to finish.
If anyone has any explanation for this or how we can successfully propagate the SIGTERM to the process that waypoint-entrypoint runs, we would be very grateful.
Cheers,
Dan Wanek