Jenkins: "Access denied to Vault Secrets at 'path'" vs CLI

[sf-greensill]

Hi there,

I am following a tutorial which prints out a secret value to a txt file in Jenkins.
'echo $SECRET > secret.txt'

Jenkins has the vault plugin installed.

The credentials have been entered into Jenkins for an approle user. When the job is run, Jenkins authenticates with Vault. I have tried with a bad password and that failed the pipeline.

When I run the pipeline with the correct credentials I get the message:

Access denied to Vault Secrets at 'secret/dev/appsecret'

I have checked this user, there is a policy in place to allow access to this path and I have used the that user in the Vault cli to successfully get the secret.

vault login <approle-user-token>
Success! You are now authenticated.
vault kv get -field=username secret/dev/appsecret
dbUser

The policy is:
path "auth/approle/login" { capabilities = [ "create", "read" ] }
# Read test data
path "secret/dev/*" { capabilities = [ "read" ] }

I have tried everything that I can think of but must be missing something. Any insight would be gratefully received, thank you.

[sf-greensill]

This is an update.

When I modify the secret path either in the request or in the policy path the error returned states:
FATAL: Vault credentials not found for '<PATH>'

[conradf7]

+1. I am following the same tutorial (probably from Linkedin) and I run into the exact same issue.

I can confirm everything works fine when I directly ssh to the Vault server container; then login via the jenkins role ; and the get secret/dev/appsecret works just fine.

But the same configuration (same role-id and secret-id) does NOT work via the jenkins server.

Console output:
Started by user Vault User
Running as SYSTEM
Building in workspace /var/jenkins_home/workspace/proj1
Access denied to Vault Secrets at 'secret/dev/appsecret’

What’s causing this deny ?

[Wolfsrudel]

Can you provide a link to the tutorial if it’s not from learn.hashicorp.com?

[conradf7]

Unfortunately it’s in Linkedin-Learning (Learning Hashicorp Vault - by David Swersky), so that’s a paid / proprietary subscription

[Wolfsrudel]

Can we reproduce all steps?

[conradf7]

It’s a little tough to reproduce all the steps since this is the “last-mile” … having Jenkins as the client to use the role-id and secret-id to access the secret.

But I will try to present the important points.

First, from the Vault server side perspective :

Note that I logged into the actual Vault container and did the following:

1. Printed the VAULT_SERVER env variable (needed to enter into the client app, Jenkins)
2. Logged in via the “jenkins” approle. Notice the output , which you can ignore since it is not important, including the login token which is NOT used in the client
3. Once logged in as “jenkins”, the secret can be accessed, in this case, it’s the “username” key , with “dbUser” as the value

So as stated earlier, the direct interaction on the Vault server works just fine as expected.

However, the problems occur on the Jenkins client side.

You can enter the relevant information in JENKINS Server , as follows :

The Jenkins’ “vault credential” is configured essentially with the same role_id and secret_id (as was shown earlier in the vault server session).

But when you try to run the job , which includes echo (printing) the $USERNAME key-value, you unfortunately get the error that was originally posted , as shown in the “Console Output” image below:

As you can see … the echo shows up empty, meaning Jenkins -> Vault invocation returns blanks when trying to access the secret/dev/appsecret
(which I think should be the equivalent of invoking:
“vault kv get secret/dev/appsecret”
which does in fact work fine, as shown earlier on the vault server, while logged into vault as jenkins approle)

So, in theory, this sounds very straightforward. In fact the Jenkins job successfully accesses the vault server (http://vaultserver:8200) on the bridge network as indicated, but for some strange reason is denied access to the vault secret despite using the same approle creds.

Ideas ?

[conradf7]

I found a solution. It was definitely not mentioned in the author’s tutorial. (and frankly not mentioned in other tutorials). But it was due to the engine version, which needs to be changed to “1” even though the default in Jenkins is apparently version “2”

And granted the issue was on the Jenkins side ; but still, given this is a popular use case for Vault, it would have been helpful for the Vault people to make this more obvious, given how many hours and hours new users must have wasted because of this seemingly subtle, but opaque issue.

Anyhow, here’s an image showing the single change that worked :

Tutorials don’t state this, but you will need to click on “Advanced settings” in the Vault Plugin, and then CHANGE from the default to
K/V Engine version: 1

Again, the bridge network and vault is running in dev mode, but for whatever other strange reason, it works as if in “normal” vault server mode.

BTW, it works for both the approle as well as using a temporary token (issued for the jenkins role). I verified the vault engine “version 1” change works in both scenarios, as you would expect it to.

Hope this helps the next person to save a ton of needless perplexing and debugging time.

Cheers

[Wolfsrudel]

Great! Thank you for sharing the solution!

[sf-greensill]

Hi @conradf7,

You were right, I am following the LinkedIn Learning tutorial and indeed your solution worked in my case as well.

For added confusion, it is worth noting that in the module: Adding policies to Vault Our intrepid instructor states at 1:03

One thing to note, the dev server we started uses the KV Secrets Engine version two.

Thank you.

[conradf7]

Hi @sf-greensill , glad it worked for you too!

I refused to let this issue go, mainly because it was so perplexing, given that we did all the “right” things. Multiple times! And it just refused to work as you know.

And yes, the instructor didn’t change the default when demonstrating. Yet it somehow worked for him. So that leads me to believe that back then (notice his instruction was back in late 2018) it’s quite possible the “Default” in Vault-plugin meant something else in that earlier version of both Vault server and Jenkins’ vault-plugin.

Moreover, what is really confusing to me is - Version 1 is stated by Hashicorp to be “Normal” servers. See their comments in the HCL shown in:
(https://www.vaultproject.io/intro/getting-started/policies/)

BTW, that article clued me in to trying the switch to version 1 “normal server” mode, mainly due to correlating that article HCL to that of the policy language used by the tutorial author.

So bottom line, even though we are running a “dev” vault server in the tutorial, however in jenkins’ vault-plugin we need to switch default kv version to “normal server” mode (version 1), I guess to match the policy language used.

So yes, it’s confusing for sure!

Oh well, it works! Given today’s situation, I’m grateful for the little things these days

[sf-greensill]

Hi @conradf7,

Thanks for explaining how you found what was causing the problem.  It’s trouble shooting issues like these which I find counter intuitively help with the learning process.

You may have found you didn’t have the kv type path secret/... doesn’t exist on start up either as it did in the tutorial. It doesn’t look to me, like the instructor is responding any more in the QA section.

Thanks again for posting your answer, it’s very helpful!