Boundary MySQL connection - "connection closed" - session still active

Hi everyone,

I am doing some stress testing in the Boundary+Vault setup I have deployed.
I am having issues sometimes when connecting to a database target (MySQ RDS) using a Vault Credential Store. Ie: I can connect normally once, second time, maybe, third time, maybe… and so on. The error is a normal MySQL access denied:

ERROR 1045 (28000): Access denied for user 'v-token-toke-test-XjNrj088dQE'@'10.237.99.236' (using password: YES)

This weird part is that after getting that error, the session in Boundary will remain active (even though I could not connect) and I will only get this error when connecting through Boundary. I can still connect to the database target using the same credentials generated by Vault+Boundary from the worker nodes.
How does Boundary compare for valid credentials when doing the proxy? Is there a limit on the connections used by Boundary or one Vault token or something like that?
Why would I get access denied ONLY when connecting with the Boundary proxy, and when connecting from the worker nodes using the SAME credentials directly with a mysql client, the connection works fine? The generated/relayed credentials are properly created on the database, the worker nodes can use the credential, but the credentials fail when connecting ONLY with the Boundary proxy.
I am currently using 0.10.1 and have moved up from 0.7.5. Could database migrations be causing this (I had no errors)?
Can anyone please point me in which direction to debug?

TIA

Still pulling my hair over this…

I wrote a script that would go through one Boundary target multiple times, and connecting once through the proxy open by Boundary, and next, connect to a SSH tunnel that’s port forwarding to the very same database host, using the same credentials that I’ve gotten from Boundary.
The script is the following:

#!/bin/bash
SUCCESS=0
TCP="ttcp_NvPtCblYtv"
while [ 1 ]; do
  PORT=$(awk -v min=40000 -v max=50000 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')
  boundary connect -target-id $TCP --listen-port $PORT > output &
  sleep 3
  TMP_BOUNDARY_USER=$(cat output|grep username|cut -d'"' -f4)
  TMP_BOUNDARY_PASS=$(cat output|grep password|cut -d'"' -f4)

  echo "Trying $TMP_BOUNDARY_USER / $TMP_BOUNDARY_PASS (Port $PORT)"
  echo "Using Boundary proxy: "
  mysql -h 127.0.0.1 -u$TMP_BOUNDARY_USER -P$PORT -p"$TMP_BOUNDARY_PASS" -e"SELECT CURRENT_USER();"
  echo -e "\n"
  echo "Using tunnel: "
  mysql -h 127.0.0.1 -u$TMP_BOUNDARY_USER -P3306 -p"$TMP_BOUNDARY_PASS" -e"SELECT CURRENT_USER();"
  echo -e "\n"
done

The output is the following:


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_HM6@%  |
+----------------+


Trying infosec_AnI / ZL5yejY3Ejdk-XAdakYJ (Port 42197)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_AnI@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_AnI@%  |
+----------------+


Trying infosec_T2P / B-Ecbs-y4MtcnOeeynOG (Port 40082)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_T2P@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_T2P@%  |
+----------------+


Trying infosec_tJy / TX1-dqkfi-17-H4U3xgb (Port 43026)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_tJy@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_tJy@%  |
+----------------+


Trying infosec_VkX / MgyDfcPM4cDvxtVzB-4u (Port 47318)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_VkX'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_VkX@%  |
+----------------+


Trying infosec_1Fp / R3PGR5re-uqGtkPj2muT (Port 46655)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_1Fp@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_1Fp@%  |
+----------------+


Trying infosec_l7A / h1gY-a3RIq01l3eqHkQo (Port 49626)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_l7A'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_l7A@%  |
+----------------+


Trying infosec_EMz / I4IWCgXTTFfa-wIe3BJD (Port 43919)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_EMz@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_EMz@%  |
+----------------+


Trying infosec_e0Z / JBKjYJFjLBF8MJF1-kqj (Port 43278)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_e0Z'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_e0Z@%  |
+----------------+


Trying infosec_hgs / 38s9uoo6fCVjksTFRI-s (Port 42575)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_hgs@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_hgs@%  |
+----------------+


Trying infosec_xWo / Nz5yE6a-1TvH53qgCWzk (Port 41917)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_xWo@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_xWo@%  |
+----------------+


Trying infosec_14i / OupAZQ01CZuaBauN-szg (Port 41256)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_14i'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_14i@%  |
+----------------+


Trying infosec_4Ed / JGYZTHd-t14S73F5NBcY (Port 45625)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_4Ed@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_4Ed@%  |
+----------------+


Trying infosec_Ozj / 5rMuWAl-ujX7LEweHuwN (Port 43556)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_Ozj'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_Ozj@%  |
+----------------+


Trying infosec_uQ8 / l-r32x5Yg3smjD2tuecy (Port 42849)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_uQ8'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_uQ8@%  |
+----------------+


Trying infosec_oND / TWMZeMHdkCQV-QifmP-5 (Port 42186)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_oND'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_oND@%  |
+----------------+

As you can see, the SSH tunnel always works, whereas the Boundary proxy doesn’t work every time as the tunnel does. Why would that be?! :pensive:

Forgot to mention that the tunnel is going to the worker, and then to the database, much like Boundary is doing. This is the only worker that is running currently.

Which version of Boundary? If 0.10.x, there is an issue in 0.10.0+ when a session has multiple connections, which will be fixed in 0.10.3. Otherwise you could try the latest 0.9 series and see if that helps.