Boundary MySQL connection - "connection closed" - session still active

Boundary
1

Hi everyone,

I am doing some stress testing in the Boundary+Vault setup I have deployed.
I am having issues sometimes when connecting to a database target (MySQ RDS) using a Vault Credential Store. Ie: I can connect normally once, second time, maybe, third time, maybe… and so on. The error is a normal MySQL access denied:

ERROR 1045 (28000): Access denied for user 'v-token-toke-test-XjNrj088dQE'@'10.237.99.236' (using password: YES)

This weird part is that after getting that error, the session in Boundary will remain active (even though I could not connect) and I will only get this error when connecting through Boundary. I can still connect to the database target using the same credentials generated by Vault+Boundary from the worker nodes.
How does Boundary compare for valid credentials when doing the proxy? Is there a limit on the connections used by Boundary or one Vault token or something like that?
Why would I get access denied ONLY when connecting with the Boundary proxy, and when connecting from the worker nodes using the SAME credentials directly with a mysql client, the connection works fine? The generated/relayed credentials are properly created on the database, the worker nodes can use the credential, but the credentials fail when connecting ONLY with the Boundary proxy.
I am currently using 0.10.1 and have moved up from 0.7.5. Could database migrations be causing this (I had no errors)?
Can anyone please point me in which direction to debug?

TIA

2

Still pulling my hair over this…

3

I wrote a script that would go through one Boundary target multiple times, and connecting once through the proxy open by Boundary, and next, connect to a SSH tunnel that’s port forwarding to the very same database host, using the same credentials that I’ve gotten from Boundary.
The script is the following:

#!/bin/bash
SUCCESS=0
TCP="ttcp_NvPtCblYtv"
while [ 1 ]; do
  PORT=$(awk -v min=40000 -v max=50000 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')
  boundary connect -target-id $TCP --listen-port $PORT > output &
  sleep 3
  TMP_BOUNDARY_USER=$(cat output|grep username|cut -d'"' -f4)
  TMP_BOUNDARY_PASS=$(cat output|grep password|cut -d'"' -f4)

  echo "Trying $TMP_BOUNDARY_USER / $TMP_BOUNDARY_PASS (Port $PORT)"
  echo "Using Boundary proxy: "
  mysql -h 127.0.0.1 -u$TMP_BOUNDARY_USER -P$PORT -p"$TMP_BOUNDARY_PASS" -e"SELECT CURRENT_USER();"
  echo -e "\n"
  echo "Using tunnel: "
  mysql -h 127.0.0.1 -u$TMP_BOUNDARY_USER -P3306 -p"$TMP_BOUNDARY_PASS" -e"SELECT CURRENT_USER();"
  echo -e "\n"
done

The output is the following:


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_HM6@%  |
+----------------+


Trying infosec_AnI / ZL5yejY3Ejdk-XAdakYJ (Port 42197)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_AnI@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_AnI@%  |
+----------------+


Trying infosec_T2P / B-Ecbs-y4MtcnOeeynOG (Port 40082)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_T2P@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_T2P@%  |
+----------------+


Trying infosec_tJy / TX1-dqkfi-17-H4U3xgb (Port 43026)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_tJy@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_tJy@%  |
+----------------+


Trying infosec_VkX / MgyDfcPM4cDvxtVzB-4u (Port 47318)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_VkX'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_VkX@%  |
+----------------+


Trying infosec_1Fp / R3PGR5re-uqGtkPj2muT (Port 46655)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_1Fp@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_1Fp@%  |
+----------------+


Trying infosec_l7A / h1gY-a3RIq01l3eqHkQo (Port 49626)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_l7A'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_l7A@%  |
+----------------+


Trying infosec_EMz / I4IWCgXTTFfa-wIe3BJD (Port 43919)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_EMz@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_EMz@%  |
+----------------+


Trying infosec_e0Z / JBKjYJFjLBF8MJF1-kqj (Port 43278)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_e0Z'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_e0Z@%  |
+----------------+


Trying infosec_hgs / 38s9uoo6fCVjksTFRI-s (Port 42575)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_hgs@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_hgs@%  |
+----------------+


Trying infosec_xWo / Nz5yE6a-1TvH53qgCWzk (Port 41917)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_xWo@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_xWo@%  |
+----------------+


Trying infosec_14i / OupAZQ01CZuaBauN-szg (Port 41256)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_14i'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_14i@%  |
+----------------+


Trying infosec_4Ed / JGYZTHd-t14S73F5NBcY (Port 45625)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_4Ed@%  |
+----------------+


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_4Ed@%  |
+----------------+


Trying infosec_Ozj / 5rMuWAl-ujX7LEweHuwN (Port 43556)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_Ozj'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_Ozj@%  |
+----------------+


Trying infosec_uQ8 / l-r32x5Yg3smjD2tuecy (Port 42849)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_uQ8'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_uQ8@%  |
+----------------+


Trying infosec_oND / TWMZeMHdkCQV-QifmP-5 (Port 42186)
Using Boundary proxy:
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'infosec_oND'@'10.237.99.236' (using password: YES)


Using tunnel:
mysql: [Warning] Using a password on the command line interface can be insecure.
+----------------+
| CURRENT_USER() |
+----------------+
| infosec_oND@%  |
+----------------+

As you can see, the SSH tunnel always works, whereas the Boundary proxy doesn’t work every time as the tunnel does. Why would that be?! :pensive:

4

Forgot to mention that the tunnel is going to the worker, and then to the database, much like Boundary is doing. This is the only worker that is running currently.

5

Which version of Boundary? If 0.10.x, there is an issue in 0.10.0+ when a session has multiple connections, which will be fixed in 0.10.3. Otherwise you could try the latest 0.9 series and see if that helps.