Suddenly Receiving 500 Error Stating "Unable to parse connection url" When using mongodb-atlas vault plugin

Vault
1

Hello, we have been using the mongodb-atlas vault plugin for quite some time, and we have been running into a strange error since last week. For the past few days, we have randomly been receiving the following error from vault when attempting to generate temporary credentials with mongodb:

{"errors":["1 error occurred:\n\t* unable to parse connection url\n\n"]}

For whatever reason, this error eventually goes away if we keep attempting to generate the credentials, but the error has always shown up later, still happening completely randomly. This is happening across multiple different database mounts, across different atlas clusters, so it does not appear to be localized to a particular atlas instance.

Some background on our vault installation, We have a vault EC2 target group in AWS made up of 5 replicas, 4 being in performance standby mode, with an ELB load balancer in front of the target group.

Here is an example of one of the statement we used to initialize the atlas plugin in our vault installation:

vault secrets enable -path=database/prod/mongodb/client1 database

vault write database/prod/mongodb/client1/config/our-mongodb-instance \
plugin_name=mongodbatlas-database-plugin \
allowed_roles="ro,rw,admin" \
project_id="11111" \
public_key="aaaaa" \
private_key="123123123123123"

vault write database/prod/mongodb/client1/roles/rw  \
db_name=our-mongodb-instance \
creation_statements='{"database_name": "admin","roles":[{"databaseName":"admin","roleName":"rw"}],"scopes":[{"name":"vault-user","type":"CLUSTER"}]}' \
default_ttl="12m" \
max_ttl="7h";

As said above, this mongodb configuration has been working fine for months, and the errors mentioned above seem to happen randomly, leading me to believe that this is a load-balancing issue with requests perhaps being rerouted to the wrong server, but we only have a single server that’s not in “standby” mode, leading me to assume that vault should be correctly forwarding requests to the active server.

I don’t fully understand the chain of events that occur when you request a temporary credential using the mongodb-atlas plugin, but it seems to me like something is getting lost in translation, and the final connection request that gets sent to mongodb is getting corrupted at some point. If anyone could provide any insight that would help me debug this issue, I would greatly appreciate it.

Edit: Here is some information about my vault cluster, outputted from vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.13.1
Build Date               2023-03-23T12:51:35Z
Storage Type             raft
Cluster Name             vault-cluster-11111111
Cluster ID               <redacted>
HA Enabled               true
HA Cluster               https://<redacted>:8201
HA Mode                  standby
Active Node Address      https://<redacted>:8200
Raft Committed Index     468594
Raft Applied Index       468594
2

did you manage to solve? I have the same 500 with the same response message, but I don’t have the same behavior you have with “sometimes it works”

3

I think I know what happened. Netowork.

MongoAtlas vault plugin connects to https://cloud.mongodb.com/

if the request is initiated from a vault machine where it does not have this connection, it fails dramatically

4

This was a while ago so I don’t remember all the details now, but I believe our issue was a load balancing one, we had a few servers running vault on EC2, and some of them decided to go down, so we’d receive “random” error messages which were really just the request being routed to a downed server.

Confidentiality Notice: This email, including any attachments, is for the sole use of the individual to whom it is addressed, and may contain sensitive, confidential or privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error, please notify the sender by reply e-mail and destroy this message and its attachments.

1 Like