We’re using envconsul in an init container to inject secrets (from vault) into containers at runtime. Under normal conditions, this is working fine. However when a situation arises where envconsul needs to restart the child process (secret has changed, misconfiguration, etc), the child process is coming up, but immediately crashing due to a port conflict - I’m assuming from the previous child process still holding the port.
Here is an excerpt from logs:
2021/03/22 20:23:09.048062 [INFO] (runner) running
2021/03/22 20:23:09.048067 [DEBUG] Found KV2 secret
2021/03/22 20:23:09.048210 [DEBUG] (runner) setting SECRET_API_KEY from vault.read(secret/path1...)
2021/03/22 20:23:09.048223 [DEBUG] Found KV2 secret
2021/03/22 20:23:09.048271 [DEBUG] (runner) setting ANOTHER_API_KEY from vault.read(secret/path2...)
2021/03/22 20:23:09.048281 [DEBUG] Found KV2 secret
2021/03/22 20:23:09.048326 [DEBUG] (runner) setting LAST_KEY from vault.read(secret/path3...)
2021/03/22 20:23:09.048437 [INFO] (runner) stopping existing child process
2021/03/22 20:23:09.048443 [DEBUG] (runner) stopping child process
2021/03/22 20:23:09.048446 [INFO] (child) stopping process
2021/03/22 20:23:09.052123 [DEBUG] (cli) receiving signal "child exited"
2021/03/22 20:23:09.052341 [INFO] (child) spawning: my-app
2021/03/22 20:23:09.052463 [DEBUG] (cli) receiving signal "child exited"
2021/03/22 20:23:09.058494 [DEBUG] (cli) receiving signal "child exited"
INFO[0000] Using redis store: redis://redis-url:6379
INFO[0000] Application listening on: 8080
Main error: listen tcp :8080: bind: address already in use
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7438e4]
goroutine 1 [running]:
main.main()
/go/src/github.com/<removed>/main.go:376 +0xd0d4
2021/03/22 20:23:09.201875 [INFO] (runner) running
2021/03/22 20:23:09.201898 [DEBUG] Found KV2 secret
2021/03/22 20:23:09.201885 [DEBUG] (cli) receiving signal "child exited"
2021/03/22 20:23:09.202041 [INFO] (cli) subprocess exited
2021/03/22 20:23:09.202071 [INFO] (runner) stopping
2021/03/22 20:23:09.202162 [DEBUG] (runner) stopping watcher
2021/03/22 20:23:09.202187 [DEBUG] (watcher) stopping all views
2021/03/22 20:23:09.202383 [DEBUG] (runner) stopping child process
2021/03/22 20:23:09.202447 [INFO] (child) stopping process
2021/03/22 20:23:09.202467 [DEBUG] (child) Kill() called but process dead; not waiting for splay.
2021/03/22 20:23:09.202486 [ERR] (cli) unexpected exit from subprocess (2)
You can see a mix of envconsul logs and the application logs. Envconsul tries to restart the application, and you see it start to come up, but immediately crash due to port 8080 already being in use.
I’ve posted a sanitized envconsul config below:
vault {
<redacted>
}
pristine = false
upcase = true
secret {
<redacted>
}
...
I’ve turned logging to debug via the command line ("-log-level=debug"), and I’ve tried “-exec-kill-signal=SIGKILL” and “-exec-kill-signal=SIGINT” in case it was a signal issue. I saw someone post a gist with a similar issue (envconsul-errors-example-1 · GitHub). I’m wondering if anybody else has noticed behavior like this and has any suggestions.