Cannot enable plugin ELF architecture: EM_X86_64 (current architecture: amd64)

Vault
1

Hi,

I wrote a plugin that should be added to the vault server in golang.

I’m compiling the plugin using this command: GOARCH=amd64 GOOS=linux go

The code compiled successfully and I managed to write it to the vault server.

When I’m trying to enable it I’m receiving this error:

secrets.system.system_c3126942: error occurred during enable mount: path=<my_path>/
  error=
  | invalid backend version: 2 errors occurred:
  | \t* Unrecognized remote plugin message: 
  | This usually means
  |   the plugin was not compiled for this architecture,
  |   the plugin is missing dynamic-link libraries necessary to run,
  |   the plugin is not executable by this process due to file permissions, or
  |   the plugin failed to negotiate the initial go-plugin protocol handshake

OS version:

cat /etc/os-release 
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Go version: go version go1.19.4 linux/amd64

2

That’s an incomplete command - I assume you’re actually running some flavour of go build. Are you setting GOARCH and GOOS because you’re actually compiling on one platform to run on another, or just redundantly specifying the platform you’re already on?

You’ve only provided one set of OS and platform details - do these refer to the compilation or runtime environment, or both?

The error you have quoted is a really generic one - you can see that it is saying that something went wrong, and then speculating about possible causes, so the next thing to do is to investigate which one it might be.

Vault plugins are regular executables. You should attempt to execute your compiled plugin - in the target server environment - from the command line, with no parameters. A Vault plugin should respond with the message

This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

which proves the plugin code was at least able to start - but I suspect in your environment it will fail before that, and display some kind of relevant error.

3

I received this message when I ran the code:

This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

The compilation is done on the same machine that I’m trying to execute it.

4

You need to execute it in the target server environment for this to be a useful test.

You imply here:

that that’s a different server.

5

The message was from the target server which the vault install on it.

6

OK… FYI, you’ve really distracted from the actual problem a lot, by bringing up GOOS, GOARCH, and architectures, if you’re compiling the plugin on the same machine the Vault server is running on.

Of all the quoted possible problems,

is now seeming quite likely.

  • Look into that
  • Consider raising the Vault server’s log level to TRACE which might provide more information
  • Try starting a separate Vault dev server using vault server -dev -dev-plugin-dir=/path/to/directory/containing/plugin, and enabling the plugin within that, to test the plugin separately from your existing Vault server setup
7

This is the stack trace when I enabled the TRACE

2023-03-23T07:57:38.591Z [DEBUG] secrets.plugin.plugin_bc2a940b.plugin.plugin: 2023/03/23 07:57:38 cannot allocate memory: metadata=true
2023-03-23T07:57:38.592Z [ERROR] secrets.system.system_c3126942: error occurred during enable mount: path=plugin
...
2023-03-23T08:06:17.822Z [TRACE] activity: writing segment on timer expiration

The new error is in addition to the previous errors.

8

I guess something is running out of memory, or some other memory-related resource, then.

9

I have 2GB RAM and my memory is 10% use.
When I’m using dev mode the plugin is enabled successfully.

BTW: on Ubuntu 22.04 it’s running as expected.

10

Do you have any idea how to proceed from here?

11

I am also facing the same issue, and did’nt found anything in logs. I am getting the expected message when I am running the plugin directly. It gives me


* invalid backend version: 2 errors occurred:
        * Unrecognized remote plugin message:
Failed to read any lines from plugin's stdout
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake

Additional notes about plugin:
  Path: /app/bin/vault-plugin-auth-kubernetes
  Mode: -rwxr-xr-x
  Owner: 1001 [vault] (current: 1001 [vault])
  Group: 1001 [?] (current: 1001 [?])
  ELF architecture: EM_X86_64 (current architecture: amd64)

Any ideas on how to solve it.

12

its a glibc issue, you need to patch the binary with

find something around.

$patchelf --set-interpreter /opt/glibc/lib/ld-linux-x86-64.so.2 --set-rpath /opt/glibc/lib ‘db plugin path’

U need to have glibc available at /opt path, please do some googling around building the required glibc.