Error: Configuring Oracle Vault DB Plugin

bhardwaj.nithin131 · September 18, 2021, 8:34pm

Hi Team,

We are exploring the opportunity of using HashiCorp Vault to rotate the credentials of DB Users.

Used the following link as reference for doing the POC

As part of the POC,

Installed Docker version 20.10.8 on MacOS Big Sur Version 11.5.2
Running Oracle 12.2.0.1 inside the docker container . Used this image to install Oracle
Docker Hub
Launched a centos container using the image centos:latest. Installed Vault 1.8.2 in Centos Conatiner
Installed Oracle Instant Client instantclient-basic-linux.x64-12.2.0.1.0.zip in the Centos
Container from the page Instant Client for Linux x86-64 (64-bit) | Oracle India
Downloaded the Oracle Plugin 0.4.2 (vault-plugin-database-oracle_0.4.2_linux_amd64.zip)
from the following page vault-plugin-database-oracle v0.4.2 Binaries | HashiCorp Releases
Registered the plugin with Vault Plugin Catalog
While writing connection configuration i am getting the following error

Error writing data to database/config/oraconn: Error making API request.

URL: PUT http://127.0.0.1:443/v1/database/config/oraconn

Code: 400. Errors:

error creating database object: invalid database version: 2 errors occurred:
fork/exec /root/vault/plugins/vault-plugin-database-oracle: operation not permitted
fork/exec /root/vault/plugins/vault-plugin-database-oracle: operation not permitted

Error can be reproduced by doing the following steps

Installing Docker version 20.10.8 on MacOS
Running Oracle 12.2.0.1 inside the docker container . Used this image to install Oracle
Docker Hub and ran the container using the following command

docker run -d -p 1521:1521 --name oracle store/oracle/database-enterprise:12.2.0.1

Create a user in Oracle by logging as SYS and giving the user the Admin privileges
Launching a centos container using the image centos:latest
4 . Login to the centos container by using the command
docker run -it -p 443:443 centos:latest
Execute the commands in the attachment in sequence.Commands.txt (4.9 KB)
After getting the error, executed the below command as well
sudo setcap cap_ipc_lock=+ep /root/vault/plugins/vault-plugin-database-oracle

Tried older versions of plugin also, but still facing the same issue.
Can anyone please help me in resolving this issue.

aram · September 19, 2021, 8:29am

Run ldd as the vault user on the oracle plugin executable to make sure it found the oracle libraries. Most likely it didn’t find them. You probably need to add the LD_LIBRARY_PATH to the service file that starts vault to set the oci ldd library file.

Second, host.docker.internal:1521 sounds wrong for a hostname, it should be the name of the container (as long as their on their own network). It would be easier if you made both of them network host for testing purposes so you can just do 127.0.0.1:1521.

Lastly, a piece of advice … don’t screw around with ports and tls. Although this works it’s just going to cause you issues and/or confusion later on.

listener "tcp" {
  address          = "0.0.0.0:443"
  tls_disable      = "true"
}

storage "file" {
  path    = "/root/vault/data"
}

Leave vault on 8200, there is no reason to move it. If you need to have it on 443 (with https - tls on) or 80 (http - tls off) then setup a nginx forward proxy, quick and easy.

bhardwaj.nithin131 · September 19, 2021, 10:59am

Hi Aram,

Thanks for your inputs.

Made the following changes

Changed the port to 8200.Following is the new config file

listener “tcp” {
address = “0.0.0.0:8200”
tls_disable = “true”
}

storage “file” {
path = “/root/vault/data”
}

plugin_directory="/root/vault/plugins"
api_addr = “http://127.0.0.1:8200”
disable_mlock = “true”
ui = “true”

Created a Vault Service File(vault.service) with the following configuration in the path /etc/systemd/system and as mentioned by you added the ld Library path to this file under service section

[Unit]

Description=vault service

Requires=network-online.target

After=network-online.target

ConditionFileNotEmpty=/root/vault/configs/vault-config.hcl

[Service]

User=root

Group=root

Restart=on-failure

ExecStart=/root/vault/bin/vault server -config=/root/vault/configs/vault-config.hcl

StandardOutput=/root/vault/logs/vault-output.log

StandardError=/root/vault/logs/vault-error.log

LimitMEMLOCK=infinity

ExecReload=/bin/kill -HUP $MAINPID

KillSignal=SIGTERM

LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2

[Install]

WantedBy=multi-user.target

When i am trying to create the connection configuration got following error

Error writing data to database/config/oraconn: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/database/config/oraconn

Code: 400. Errors:

error creating database object: invalid database version: 2 errors occurred:
Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply

needs to be recompiled to support the latest protocol.

Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply

needs to be recompiled to support the latest protocol.

After this error, i executed the command
setcap cap_ipc_lock=+ep /root/vault/plugins/vault-plugin-database-oracle
Again tried creating the connection configuration using the following comand
vault write database/config/oraconn
plugin_name=vault-plugin-database-oracle
allowed_roles="*"
connection_url=’{{username}}/{{password}}@host.docker.internal:1521/ORCLPDB1.localdomain’
username=‘username’
password=‘password’

Again ended up getting the same error

Error writing data to database/config/oraconn: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/database/config/oraconn

Code: 400. Errors:

error creating database object: invalid database version: 2 errors occurred:
fork/exec /root/vault/plugins/vault-plugin-database-oracle: operation not permitted
fork/exec /root/vault/plugins/vault-plugin-database-oracle: operation not permitted

As per your suggestion executed the command ldd vault-plugin-database-oracle and got following is the output
linux-vdso.so.1 (0x00007ffe94571000)
libclntsh.so.19.1 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff1d9b61000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff1d979c000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff1d9d81000)
I was able to access the oracle db running on a separate container in the same system using sqlplus from the centos container by mentioning the host name as host.docker.internal.

Can you please suggest the reason why i am getting this error.Did i do anything wrong

aram · September 19, 2021, 3:04pm

The proper systemd way of adding an environment variable to systemd is:

[Service]
Environment="LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2"

For clarity – in situations where the unit file is shipped with the app, you should create an override file. You can then upgrade your package and unit file without having to update it again. Since vault doesn’t ship with a unit file it is moot in this case, but just included here for completeness.

bhardwaj.nithin131 · September 20, 2021, 6:38am

Hi Aram,

Thanks for the inputs and suggestions.
Modified the vault.service file as below by adding environment variable in service section

[Unit]

Description=vault service

Requires=network-online.target

After=network-online.target

ConditionFileNotEmpty=/root/vault/configs/vault-config.hcl

[Service]

User=root

Group=root

Restart=on-failure

ExecStart=/root/vault/bin/vault server -config=/root/vault/configs/vault-config.hcl

StandardOutput=/root/vault/logs/vault-output.log

StandardError=/root/vault/logs/vault-error.log

LimitMEMLOCK=infinity

ExecReload=/bin/kill -HUP $MAINPID

KillSignal=SIGTERM

Environment=“LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2”

[Install]

WantedBy=multi-user.target

Still facing the same error

Error writing data to database/config/oraconn: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/database/config/oraconn

Code: 400. Errors:

error creating database object: invalid database version: 2 errors occurred:
fork/exec /root/vault/plugins/vault-plugin-database-oracle: operation not permitted
fork/exec /root/vault/plugins/vault-plugin-database-oracle: operation not permitted

Can you please help me in resolving this issue

aram · September 20, 2021, 7:07am

One note, using ‘root’ to run Vault is discouraged. See: Production Hardening | Vault - HashiCorp Learn

“operation not permitted” sounds like your token doesn’t have the proper permission. Or that the OS cannot fork/execute the registered plugin.

Check your vault logs to see any detail … set log_level to “DEBUG” if you need more information. A HUP signal will reload the config. Retry the ‘vault write’ to get the related messages.

You can use /sys/plugins/reload/backend - HTTP API | Vault by HashiCorp to reload the plugin as well to see any messages tells you anything about the plugin.

bhardwaj.nithin131 · September 20, 2021, 10:03am

Hi Aram,

To check whether it is due to permission issue i am running it as root. Will not do this in production.

Tried reloading the plugin. Got the following response.

{
“request_id”: “214aa92c-024c-7dba-39b9-17022d652dbb”,
“lease_id”: “”,
“renewable”: false,
“lease_duration”: 0,
“data”: {
“reload_id”: “214aa92c-024c-7dba-39b9-17022d652dbb”
},
“wrap_info”: null,
“warnings”: null,
“auth”: null
}

I have set the log_level as “debug” in the config file and following is the log i got

2021-09-20T09:50:17.392Z [DEBUG] storage.cache: creating LRU cache: size=0

2021-09-20T09:50:17.392Z [DEBUG] cluster listener addresses synthesized: cluster_addresses=[0.0.0.0:8201]

2021-09-20T09:50:17.394Z [DEBUG] would have sent systemd notification (systemd not present): notification=READY=1

2021-09-20T09:50:23.476Z [INFO] core: security barrier not initialized

2021-09-20T09:50:23.476Z [INFO] core: seal configuration missing, not initialized

2021-09-20T09:50:23.476Z [INFO] core: security barrier not initialized

2021-09-20T09:50:34.509Z [INFO] core: security barrier not initialized

2021-09-20T09:50:34.510Z [INFO] core: security barrier not initialized

2021-09-20T09:50:34.510Z [INFO] core: seal configuration missing, not initialized

2021-09-20T09:50:37.597Z [INFO] core: security barrier not initialized

2021-09-20T09:50:37.959Z [INFO] core: security barrier not initialized

2021-09-20T09:50:37.959Z [INFO] core: seal configuration missing, not initialized

2021-09-20T09:50:37.985Z [INFO] core: security barrier not initialized

2021-09-20T09:50:37.985Z [INFO] core: seal configuration missing, not initialized

2021-09-20T09:50:40.958Z [INFO] core: security barrier not initialized

2021-09-20T09:50:40.958Z [INFO] core: security barrier initialized: stored=1 shares=1 threshold=1

2021-09-20T09:50:40.959Z [DEBUG] core: cluster name not found/set, generating new

2021-09-20T09:50:40.959Z [DEBUG] core: cluster name set: name=vault-cluster-680e334c

2021-09-20T09:50:40.959Z [DEBUG] core: cluster ID not found, generating new

2021-09-20T09:50:40.959Z [DEBUG] core: cluster ID set: id=487683b1-89a3-afe6-c2a5-4ea1ae43c526

2021-09-20T09:50:40.959Z [INFO] core: post-unseal setup starting

2021-09-20T09:50:40.959Z [DEBUG] core: clearing forwarding clients

2021-09-20T09:50:40.959Z [DEBUG] core: done clearing forwarding clients

2021-09-20T09:50:40.959Z [DEBUG] core: persisting feature flags

2021-09-20T09:50:40.967Z [INFO] core: loaded wrapping token key

2021-09-20T09:50:40.967Z [INFO] core: upgrading plugin information: plugins=

2021-09-20T09:50:40.967Z [INFO] core: successfully setup plugin catalog: plugin-directory=/root/vault/plugins

2021-09-20T09:50:40.967Z [INFO] core: no mounts; adding default mount table

2021-09-20T09:50:40.968Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/

2021-09-20T09:50:40.968Z [INFO] core: successfully mounted backend: type=system path=sys/

2021-09-20T09:50:40.968Z [INFO] core: successfully mounted backend: type=identity path=identity/

2021-09-20T09:50:40.970Z [INFO] core: successfully enabled credential backend: type=token path=token/

2021-09-20T09:50:40.971Z [INFO] rollback: starting rollback manager

2021-09-20T09:50:40.971Z [INFO] core: restoring leases

2021-09-20T09:50:40.971Z [DEBUG] expiration: collecting leases

2021-09-20T09:50:40.971Z [DEBUG] expiration: leases collected: num_existing=0

2021-09-20T09:50:40.971Z [INFO] expiration: lease restore complete

2021-09-20T09:50:40.971Z [DEBUG] identity: loading entities

2021-09-20T09:50:40.971Z [DEBUG] identity: entities collected: num_existing=0

2021-09-20T09:50:40.972Z [INFO] identity: entities restored

2021-09-20T09:50:40.972Z [DEBUG] identity: identity loading groups

2021-09-20T09:50:40.972Z [DEBUG] identity: groups collected: num_existing=0

2021-09-20T09:50:40.972Z [INFO] identity: groups restored

2021-09-20T09:50:40.972Z [INFO] core: post-unseal setup complete

2021-09-20T09:50:40.972Z [INFO] core: root token generated

2021-09-20T09:50:40.972Z [INFO] core: pre-seal teardown starting

2021-09-20T09:50:40.973Z [DEBUG] expiration: stop triggered

2021-09-20T09:50:40.973Z [DEBUG] expiration: finished stopping

2021-09-20T09:50:40.973Z [INFO] rollback: stopping rollback manager

2021-09-20T09:50:40.973Z [INFO] core: usage gauge collection is disabled

2021-09-20T09:50:40.973Z [INFO] core: pre-seal teardown complete

2021-09-20T09:50:52.988Z [DEBUG] core: unseal key supplied: migrate=false

2021-09-20T09:50:52.988Z [DEBUG] core: starting cluster listeners

2021-09-20T09:50:52.988Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=0.0.0.0:8201

2021-09-20T09:50:52.988Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201

2021-09-20T09:50:52.988Z [INFO] core: post-unseal setup starting

2021-09-20T09:50:52.988Z [DEBUG] core: clearing forwarding clients

2021-09-20T09:50:52.988Z [DEBUG] core: done clearing forwarding clients

2021-09-20T09:50:52.988Z [DEBUG] core: persisting feature flags

2021-09-20T09:50:52.989Z [INFO] core: loaded wrapping token key

2021-09-20T09:50:52.989Z [INFO] core: upgrading plugin information: plugins=

2021-09-20T09:50:52.989Z [INFO] core: successfully setup plugin catalog: plugin-directory=/root/vault/plugins

2021-09-20T09:50:52.989Z [INFO] core: successfully mounted backend: type=system path=sys/

2021-09-20T09:50:52.989Z [INFO] core: successfully mounted backend: type=identity path=identity/

2021-09-20T09:50:52.989Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/

2021-09-20T09:50:52.990Z [INFO] core: successfully enabled credential backend: type=token path=token/

2021-09-20T09:50:52.991Z [INFO] rollback: starting rollback manager

2021-09-20T09:50:52.991Z [INFO] core: restoring leases

2021-09-20T09:50:52.991Z [DEBUG] expiration: collecting leases

2021-09-20T09:50:52.991Z [DEBUG] expiration: leases collected: num_existing=0

2021-09-20T09:50:52.991Z [DEBUG] identity: loading entities

2021-09-20T09:50:52.991Z [DEBUG] identity: entities collected: num_existing=0

2021-09-20T09:50:52.991Z [INFO] expiration: lease restore complete

2021-09-20T09:50:52.992Z [INFO] identity: entities restored

2021-09-20T09:50:52.992Z [DEBUG] identity: identity loading groups

2021-09-20T09:50:52.992Z [DEBUG] identity: groups collected: num_existing=0

2021-09-20T09:50:52.992Z [INFO] identity: groups restored

2021-09-20T09:50:52.992Z [DEBUG] core: request forwarding setup function

2021-09-20T09:50:52.992Z [DEBUG] core: clearing forwarding clients

2021-09-20T09:50:52.992Z [DEBUG] core: done clearing forwarding clients

2021-09-20T09:50:52.992Z [DEBUG] core: request forwarding not setup

2021-09-20T09:50:52.992Z [DEBUG] core: leaving request forwarding setup function

2021-09-20T09:50:52.992Z [INFO] core: usage gauge collection is disabled

2021-09-20T09:50:52.992Z [INFO] core: post-unseal setup complete

2021-09-20T09:50:52.992Z [INFO] core: vault is unsealed

2021-09-20T09:51:07.899Z [INFO] secrets.database.database_b902502f: initializing database rotation queue

2021-09-20T09:51:07.900Z [INFO] core: successful mount: namespace="" path=database/ type=database

2021-09-20T09:51:07.910Z [INFO] secrets.database.database_b902502f: populating role rotation queue

2021-09-20T09:51:07.910Z [INFO] secrets.database.database_b902502f: starting periodic ticker

2021-09-20T09:52:59.068Z [INFO] secrets.database.database_b902502f.vault-plugin-database-oracle: configuring client automatic mTLS

2021-09-20T09:52:59.097Z [DEBUG] secrets.database.database_b902502f.vault-plugin-database-oracle: starting plugin: path=/root/vault/plugins/vault-plugin-database-oracle args=["/root/vault/plugins/vault-plugin-database-oracle"]

2021-09-20T09:52:59.170Z [DEBUG] secrets.database.database_b902502f.vault-plugin-database-oracle: starting plugin: path=/root/vault/plugins/vault-plugin-database-oracle args=["/root/vault/plugins/vault-plugin-database-oracle"]

aram · September 20, 2021, 10:25am

Sounds like you have a policy permission then. Fix that and you should be good.

bhardwaj.nithin131 · September 20, 2021, 10:27am

Currently i am using the root token which means i should be able use the plugin right

bhardwaj.nithin131 · September 23, 2021, 12:08pm

Hi can anyone provide me any suggestion on why oracle-db plugin is not working.

AntonioM · October 13, 2022, 4:27pm

You must point LD_LIBRARY_PATH to the “lib” dir, like this

Environment=“LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2/lib”