Boundary connection to MySQL RDS doesn't work everytime

Hi everyone,

I am onboarding quite a few database instances to Boundary, and it’s using a Credential store from Vault.
All is good apart from when it’s not :slight_smile:
I don’t know what I could be missing, but now that I have onboarded a few, I am seeing this weird behaviour that sometimes the proxying works and sometimes it does not, even with the same database target instance, and the same worker.
It seems to me completely random and I don’t understand why.
For instance, this is for the same worker, and the same database target, just a few minutes apart.
When credentials work (using the proxy) I can connect normally to localhost and get to the target and I get this in the worker logs:

Aug 17 14:30:23 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“HMRhFndKcK”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“msg”:"session successfully activated”,“session_id”:“s_FdVxLITvXH”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:30:23.570402091Z”}
Aug 17 14:30:23 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“NonPiBuRSk”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“connection_id”:“sc_QUtIu7KJiQ”,“msg”:"connection successfully authorized”,“session_id”:“s_FdVxLITvXH”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:30:23.583013302Z”}

When not, when doing the same on the mysql cli client:

ERROR 1045 (28000): Access denied for user ‘v-token-toke-ase-Ixr24szF8LFeYVy’@‘10.237.94.12’ (using password: YES)

And the worker logs:

Aug 17 14:31:53 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“OdFGfgi1bm”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“msg”:"session successfully activated”,“session_id”:“s_jbhBV2zAkm”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:31:53.938718247Z”}
Aug 17 14:31:53 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“icPAzPi11y”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“connection_id”:“sc_usjcqRpItA”,“msg”:"connection successfully authorized”,“session_id”:“s_jbhBV2zAkm”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:31:53.980205045Z”}
Aug 17 14:31:54 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“1aL7cpgisd”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“connection_id”:“sc_usjcqRpItA”,“msg”:"connection closed”,“session_id”:“s_jbhBV2zAkm”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:31:54.295401486Z”}

The funny part is that if I go to the worker and from there connect directly to the instance using the credentials generated by Vault, I can connect normally. The generated user is there (with @‘%’). I just can’t when connecting from the proxy, and that happens completely randomly. Sometimes the credentials work when proxying through Boundary, sometimes they don’t, using the exact same target. Not using Boundary, the credentials work. The user is created in the database by Vault and I can connect using it, just not using the created Boundary session.

I’m quite puzzled. Does anyone know what could happening?

Thanks

Also worth mentioning that recently I have upgraded from 0.7.5 to 0.9.1.

Hey @raphaelschneider ,
I’m also trying to connect to an RDS instance using boundary. This is how I’ve configured my target and host set.

However when I’m trying to run this command :-
boundary connect -target-id ttcp_2A9pAHhmoc -exec mysql -- --port {{boundary.port}} -u livspace -p

I’m getting this error :-1:

Can you kindly share how you have configured to connect to the correct host address instead of localhost?
Thank you in advance!

Hi @AbhilashaLiv,

I have configured everything from Terraform, so I can’t help much, but the ‘livspace’@‘localhost’ you see on the mysql client error doesn’t mean you’re connecting to localhost (which you are, in part, because Boundary opens the connection on localhost, but proxies to the Host target you’ve configured). The “@‘localhost’” part means that you’re connecting from localhost (which you are). Therefore, the user livspace you’re using needs to have ‘localhost’ on mysql.user on the host column, or ‘%’, as that will make the user accessible from any source address.

Cheers

I have upgraded to 0.10.1 and this strange behaviour is still the same. Sometimes the connection goes through and I can successfully connect, sometimes it fails (connection closed on the logs). User is normally created on database target, just can’t connect through Boundary. Can connect directly with the user created. Same database target, same worker.

Any help please?

PS: Even though the connection fails, the session shows as active in Boundary.

ERROR 1045 (28000): Access denied for user ‘v-token-toke-[REDACTED]-tPzHNl9eJvt’@‘[REDACTED]’ (using password: YES)

@jbrandhorst @jeff sorry to bug you directly… but any ideas?

Hi Raphael. Sorry that you’re having trouble with connecting to MySQL. I can assure you that both the dev and support teams monitor this board closely and we’ll get back to you as soon as we can. Sorry for the delay but I’m afraid I don’t have to bandwidth to help debug this right now.

1 Like