Boundary connection to MySQL RDS doesn't work everytime

[raphaelschneider]

Hi everyone,

I am onboarding quite a few database instances to Boundary, and it’s using a Credential store from Vault.
All is good apart from when it’s not
I don’t know what I could be missing, but now that I have onboarded a few, I am seeing this weird behaviour that sometimes the proxying works and sometimes it does not, even with the same database target instance, and the same worker.
It seems to me completely random and I don’t understand why.
For instance, this is for the same worker, and the same database target, just a few minutes apart.
When credentials work (using the proxy) I can connect normally to localhost and get to the target and I get this in the worker logs:

Aug 17 14:30:23 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“HMRhFndKcK”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“msg”:"session successfully activated”,“session_id”:“s_FdVxLITvXH”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:30:23.570402091Z”}
Aug 17 14:30:23 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“NonPiBuRSk”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“connection_id”:“sc_QUtIu7KJiQ”,“msg”:"connection successfully authorized”,“session_id”:“s_FdVxLITvXH”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:30:23.583013302Z”}

When not, when doing the same on the mysql cli client:

ERROR 1045 (28000): Access denied for user ‘v-token-toke-ase-Ixr24szF8LFeYVy’@‘10.237.94.12’ (using password: YES)

And the worker logs:

Aug 17 14:31:53 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“OdFGfgi1bm”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“msg”:"session successfully activated”,“session_id”:“s_jbhBV2zAkm”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:31:53.938718247Z”}
Aug 17 14:31:53 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“icPAzPi11y”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“connection_id”:“sc_usjcqRpItA”,“msg”:"connection successfully authorized”,“session_id”:“s_jbhBV2zAkm”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:31:53.980205045Z”}
Aug 17 14:31:54 ip-10-237-94-12.eu-central-1.compute.internal boundary[15126]: {“id”:“1aL7cpgisd”,“source”:“https://hashicorp.com/boundary/ip-10-237-94-12.eu-central-1.compute.internal/worker",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:“worker.(Worker).handleProxy”,“data”:{“connection_id”:“sc_usjcqRpItA”,“msg”:"connection closed”,“session_id”:“s_jbhBV2zAkm”}},“datacontentype”:“application/cloudevents”,“time”:“2022-08-17T14:31:54.295401486Z”}

The funny part is that if I go to the worker and from there connect directly to the instance using the credentials generated by Vault, I can connect normally. The generated user is there (with @‘%’). I just can’t when connecting from the proxy, and that happens completely randomly. Sometimes the credentials work when proxying through Boundary, sometimes they don’t, using the exact same target. Not using Boundary, the credentials work. The user is created in the database by Vault and I can connect using it, just not using the created Boundary session.

I’m quite puzzled. Does anyone know what could happening?

Thanks

[raphaelschneider]

Also worth mentioning that recently I have upgraded from 0.7.5 to 0.9.1.

[AbhilashaLiv]

Hey @raphaelschneider ,
I’m also trying to connect to an RDS instance using boundary. This is how I’ve configured my target and host set.

However when I’m trying to run this command :-
boundary connect -target-id ttcp_2A9pAHhmoc -exec mysql -- --port {{boundary.port}} -u livspace -p

I’m getting this error

Can you kindly share how you have configured to connect to the correct host address instead of localhost?
Thank you in advance!

[raphaelschneider]

Hi @AbhilashaLiv,

I have configured everything from Terraform, so I can’t help much, but the ‘livspace’@‘localhost’ you see on the mysql client error doesn’t mean you’re connecting to localhost (which you are, in part, because Boundary opens the connection on localhost, but proxies to the Host target you’ve configured). The “@‘localhost’” part means that you’re connecting from localhost (which you are). Therefore, the user livspace you’re using needs to have ‘localhost’ on mysql.user on the host column, or ‘%’, as that will make the user accessible from any source address.

Cheers

[raphaelschneider]

I have upgraded to 0.10.1 and this strange behaviour is still the same. Sometimes the connection goes through and I can successfully connect, sometimes it fails (connection closed on the logs). User is normally created on database target, just can’t connect through Boundary. Can connect directly with the user created. Same database target, same worker.

Any help please?

[raphaelschneider]

PS: Even though the connection fails, the session shows as active in Boundary.

ERROR 1045 (28000): Access denied for user ‘v-token-toke-[REDACTED]-tPzHNl9eJvt’@‘[REDACTED]’ (using password: YES)

@jbrandhorst @jeff sorry to bug you directly… but any ideas?

[jbrandhorst]

Hi Raphael. Sorry that you’re having trouble with connecting to MySQL. I can assure you that both the dev and support teams monitor this board closely and we’ll get back to you as soon as we can. Sorry for the delay but I’m afraid I don’t have to bandwidth to help debug this right now.

[ian-taylor-catapult]

Running into the exact same problem. Anyone have any ideas?

[jeff]

@ian-taylor-catapult Also with MySQL?

There are two things I can think of:

1. A timing factor where the creds are not valid yet within some members of the RDS cluster. I don’t know how RDS manages creds and if they are eventually consistent like most of AWS IAM or if they are strongly consistent, but if it’s the former then perhaps if you’re going through Boundary the connection is being made too quickly after the creds are generated and before some members of the cluster have had them synced, which could explain why sometimes it works and sometimes it says it’s an invalid password.

2. The creds are being revoked before you actually use them. This could happen if the session has a connection count limit on it and sometimes that limit is being hit before the creds are used, in which case Boundary will have gone and revoked them already. Postgres for instance uses a minimum of two connections even if you think you’re making one. I don’t know why sometimes this would be a problem and sometimes not, but I don’t know the details of the MySQL protocol. You could try to figure out if this is an issue by watching Vault logs to see if the creds are being revoked.

[ian-taylor-catapult]

@jeff So I’m not using the vault integration or temporary credentials. We’re currently in the POC stage trying to connect to an AWS hosted RDS MySQL database with root credentials using HCP hosted workers and properly configured Boundary hosts, targets, and AWS networking. Pictures to follow…

I’m just getting stonewalled at trying to connect to the RDS database in the first place. It seems like this is problem? Looks like a random port getting assigned to connect through to localhost (shown in the below picture).

Any thoughts? Connection to AWS EC2 is working properly, but for some reason RDS is a struggle.

[ian-taylor-catapult]

Never mind, figured it out. Kind of obtuse though. Command that works to follow below…

boundary connect -target-id <your-target-id> -exec mysql -- --port {{boundary.port}} -h {{boundary.ip}} -u <username> -p

The way exec works is not the way I thought. You need to provide the IP / host directly to the MySQL command to get it to point to the right place.

[jeff]

Glad you figured it out! Yes, exec is very generic – it’s designed to simply run what you want, but to allow you to template in the address. If it helps, look at the boundary connect -h output for -listen-address and -listen-port.

[martinhristov90]

Hello @ian-taylor-catapult ,

Hope you are doing well!

I think the issue originated because of the following:

Only connection options that are relevant to the selected transport protocol are used or checked. Other connection options are ignored. For example, with --host=localhost on Unix, the client attempts to connect to the local server using a Unix socket file, even if a --port or -P option is given to specify a TCP/IP port number.

To ensure that the client makes a TCP/IP connection to the local server, use --host or -h to specify a host name value of 127.0.0.1 (instead of localhost), or the IP address or name of the local server. You can also specify the transport protocol explicitly, even for localhost, by using the --protocol=TCP option.

This is actually a quote from the following page of the MySQL docs.

So, In case you do not specify a host via -h option, the mysql client tries to connect via Unix socket, which Boundary is not listening to.
When the hostname is provided via -h option, it instructs the mysql client to use TCP/IP socket that Boundary is listening on.