Http calls from secrets engine hang after 10th connection

Developing a secrets engine that makes a web request to a third-party platform, I’m seeing different behavior running from a Vault plugin vs running the same code in standalone. This is a standard http.DefaultClient.Do(), defer res.Body.Close(), ioutil.ReadAll(res.Body) pattern. I can run this on its own in a loop as long as I like, but when I call this from a secrets engine on a write operation, it works as expected the first 10 times. On the 11th write, however, it does not make an outgoing connection and ultimately times out. It seems to me like Vault is changing some http connection parameters and not releasing the connections as expected. Replacing the web call with a static return value again allows an unlimited number of calls, but using my own http client with different MaxIdle… parameters does not. Anyone encounter this or find a workaround? Thanks.

Hi! Is the code you’re working on opensource where you could point us in its direction? If not, could you copy paste the code you have where you’re encountering the issue? I have some idea of what you’re describing, but I’m not certain how Vault is involved if it’s using the net/http library directly. Thanks!

Thanks Becca! Our code is not open source, but I’ve pasted below the code with some parameters redacted. In a standalone app, I can call “run” ad infinitum, but calling it from a write handler in vault, again I can only make a few calls before it seems to exhaust the connection pool and all future calls time out. Appreciate any insight!

package main

import (
“io/ioutil”
“net/http”
“strings”
“time”
)

func run(host string, time string, template string, cn string, password string, appkey string, creds string) byte {
url := “https://” + host + “…”
payload := strings.NewReader("{“timestamp”: “” + time + “”,“TemplateName”: “” + template + “”,“cn”:"" + cn + “”,“password”: “” + password + “”}")
req, _ := http.NewRequest(“POST”, url, payload)
req.Header.Add(“content-type”, “application/json”)
req.Header.Add(“x-keyfactor-appkey”, appkey)
req.Header.Add(“authorization”, "Basic "+creds)
res, _ := http.DefaultClient.Do(req)
defer res.Body.Close()
body, _ := ioutil.ReadAll(res.Body)
return body
}

func main() {
cn := “test1”
location, _ := time.LoadLocation(“UTC”)
t := time.Now().In(location)
timestamp := t.Format(“2006-01-02T15:04:05”)
result := run(“192.168.41.82”, timestamp, “UserServer”, cn, “123456789”, “…”, “…”)
print(string(result))
}

@tyrannosaurus-becks I guess your comment also elicits the question - is there a preferred alternative to using the net/http library? Thanks again.

Hi! Thanks for sharing that code! It’s super helpful.

In looking at it, I actually don’t think the behavior you’re seeing is related to Vault. There are a couple of reasons I say that. For one, Vault communicates with plugins using GRPC; so in other words, Vault and the plugin run as entirely separate processes. Also, Vault doesn’t do anything to manipulate the http library. I’ve seen magic before where some Go applications changed the default transport for the http library, or the default client, but Vault doesn’t touch it at all. In fact, in Vault we often prefer to use something else entirely, this: https://github.com/hashicorp/go-cleanhttp.

I’m thinking what you’re seeing is unlikely to be related to Vault. That doesn’t make it invalid at all, I simply wonder if you’d see the same behavior running the code directly from a little “main” method you made on your own machine. If that’s the case, it may help to debug it that way because it simplifies the issue and reduces the number of moving parts.

Let me know if that doesn’t sound right. If it doesn’t, if you could point me towards how you’re comparing “standalone” to “plugin” behavior, maybe I could duplicate it too.

Thanks again Becca. What I mean by standalone is exactly what you described - you can see a simple main method in the code I gave, and just putting the call to “run(…)” in a loop works just fine for thousands of requests. It’s only when that “run(…)” function is called FROM a vault plugin that I see it hanging after just a few calls. One of your SEs I was working with saw that behavior as well in his instance. I was able to work around the issue by calling http.DefaultClient.CloseIdleConnections() each time, but then we’re losing all the benefits of connection pooling - potentially worse than not having pooling in the first place - so it would be preferable if we didn’t have to do that. I’ll look in to the Hashicorp cleanhttp library; that seems promising. Thanks again!

1 Like