Using Azure Key Vault KMS for Boundary

Boundary
1

Hi there Everyone,

I’m testing out Boundary using Azure Key Vault as the KMS. I was able to successfully:

  • Successfully run database init script to run an create the database schema in the PostgreSQL backend
  • Get boundary to generate the recovery, root and worker keys in Azure Key Vault

When I try to start boundary, however, it fails with the following error “error registering aws host plugin” :

Error initializing controller: error registering aws host plugin: error creating host plugin: 
host.(Repository).CreatePlugin: unable to get oplog wrapper: kms.GetWrapper: error loading 
root key for scope global: kms.loadRoot: error looking up root key versions for scope global: 
kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.

(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error 
unwrapping value: rpc error: code = Unknown desc = keyvault.BaseClient#UnwrapKey: Failure 
responding to request: StatusCode=404 -- Original Error: autorest/azure: error response 
cannot be parsed: {"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" 
\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html 
xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" 
content=\"text/html; charset=iso-8859-1\"/>\r\n<title>404 - File or directory not found.
</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}\r\nfieldset{padding:0 
15px 10px 15px;} \r\nh1{font-size:2.4em;margin:0;color:#FFF;}\r\nh2{font-si" '\x00' '\x00'} 
error: invalid character '<' looking for beginning of value

When I examine the calls boundary is making, I can see the following sequence:

  • SUCCESSFULLY Get the Root Key from Key Vault
GET https://kv-boundary-nprod0.vault.azure.net/keys/root/?api-version=7.1 HTTP/1.1
Host: kv-boundary-nprod0.vault.azure.net
User-Agent: Go/go1.17.10 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v62.0.0 keyvault/7.1
Accept-Encoding: gzip
  • SUCCESSFULLY Get the Worker Key from Key Vault
GET https://kv-boundary-nprod0.vault.azure.net/keys/worker/?api-version=7.1 HTTP/1.1
Host: kv-boundary-nprod0.vault.azure.net
User-Agent: Go/go1.17.10 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v62.0.0 keyvault/7.1
Accept-Encoding: gzip
  • SUCCESSFULLY Get the Recovery Key from Key Vault
GET https://kv-boundary-nprod0.vault.azure.net/keys/recovery/?api-version=7.1 HTTP/1.1
Host: kv-boundary-nprod0.vault.azure.net
User-Agent: Go/go1.17.10 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v62.0.0 keyvault/7.1
Accept-Encoding: gzip
  • Fail While attempting to get global_root key from KeyVault:
POST https://kv-boundary-nprod0.vault.azure.net/keys/root/global_root/unwrapkey?api-version=7.1 HTTP/1.1
Host: kv-boundary-nprod0.vault.azure.net
User-Agent: Go/go1.17.10 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v62.0.0 keyvault/7.1
Content-Length: 33
Content-Type: application/json; charset=utf-8
Accept-Encoding: gzip

{"alg":"RSA-OAEP-256","value":""}

The above call Boundary makes is unusual. The Azure REST API suggests, the URI for getting keys as follows. Which means boundary is passing the word global_root as the version of the root key

https://kv-boundary-nprod0.vault.azure.net/keys/root/$VERSION/unwrapkey

Given the malformed request, Azure responds with an HTML document and a 404:

"error": "kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: 
error #301: error unwrapping value: rpc error: code = Unknown desc = 
keyvault.BaseClient#UnwrapKey: Failure responding to request: StatusCode=404 -- Original 
Error: autorest/azure: error response cannot be parsed: {\"\u003c!DOCTYPE html PUBLIC 
\\\"-//W3C//DTD XHTML 1.0 Strict//EN\\\" \\\"http://www.w3.org/TR/xhtml1/DTD/xhtml1-
strict.dtd\\\"\u003e\\r\\n\u003chtml 
xmlns=\\\"http://www.w3.org/1999/xhtml\\\"\u003e\\r\\n\u003chead\u003e\\r\\n\u003cme
ta http-equiv=\\\"Content-Type\\\" content=\\\"text/html; charset=iso-8859-
1\\\"/\u003e\\r\\n\u003ctitle\u003e404 - File or directory not 
found.\u003c/title\u003e\\r\\n\u003cstyle type=\\\"text/css\\\"\u003e\\r\\n\u003c!--
\\r\\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-
serif;background:#EEEEEE;}\\r\\nfieldset{padding:0 15px 10px 15px;} \\r\\nh1{font-
size:2.4em;margin:0;color:#FFF;}\\r\\nh2{font-si\" '\\x00' '\\x00'} error: invalid character 
'\u003c' looking for beginning of value",

Questions:

  • Why is boundary trying to load aws host plugin in Azure while using Azure Key Vault as a KMS ?
  • Why is boundary sending a malformed request with global_root as the key version to Azure Key Vault ?

Any help would be truly appreciated - Thanks !

2

What does your Boundary controller/worker config look like?

3

Hi there @omkensey ,

My config looks like below with the azure creds passed as env variables:

# Azure Creds will be passed via environment variables:
#
# AZURE_TENANT_ID
# AZURE_CLIENT_ID
# AZURE_CLIENT_SECRET
# AZUREKEYVAULT_WRAPPER_KEY_NAME

disable_mlock = true
log_level     = "trace"

worker {
  name        = "poc-worker"
  description = "A worker for a POC"
}

controller {
  name        = "env://HOSTNAME"
  description = "A controller for a demo!"
  database {
    url = "env://BOUNDARY_POSTGRES_URL"
  }
}

# API config
listener "tcp" {
  purpose              = "api"
  tls_disable          = true
  cors_enabled         = true
  cors_allowed_origins = ["*"]
}

listener "tcp" {
  purpose     = "cluster"
  tls_disable = true
}

listener "tcp" {
  purpose     = "proxy"
  tls_disable = true
}

kms "azurekeyvault" {
  purpose  = "root"
  key_name = "root"
}

kms "azurekeyvault" {
  purpose  = "worker-auth"
  key_name = "worker"
}

kms "azurekeyvault" {
  purpose  = "recovery"
  key_name = "recovery"
}
4

What’s the value of AZUREKEYVAULT_WRAPPER_KEY_NAME? Should that variable be named AZUREKEYVAULT_WRAPPER_VAULT_NAME?

5

Hi there @omkensey ,

Apologies, you’re right, that is what I use. My startup script looks like this:

$env:AZURE_TENANT_ID="<TENANT_ID>"
$env:AZURE_CLIENT_ID="<CLIENT_ID>"
$env:AZURE_CLIENT_SECRET="<CLIENT_SECRET>"
$env:AZUREKEYVAULT_WRAPPER_VAULT_NAME="<KEYVAULT_NAME>"
$env:BOUNDARY_POSTGRES_URL="postgresql://user:password@server:5432/postgres?sslmode=require"
$env:HOSTNAME="mycontroller"

boundary server -config C:\scripts\boundary\config.hcl

And the server attempts to start up:

C:\scripts\boundary> $env:HOSTNAME="mycontroller"
C:\scripts\boundary> boundary server -config C:\scripts\boundary\config.hcl
==> Boundary server configuration:

               Azure Environment: AzurePublicCloud
               Azure Environment: AzurePublicCloud
               Azure Environment: AzurePublicCloud
                  Azure Key Name: recovery
                  Azure Key Name: recovery
                  Azure Key Name: recovery
                Azure Vault Name: <vault name>
                Azure Vault Name: <vault name>
                Azure Vault Name: <vault name>
                             Cgo: disabled
  Controller Public Cluster Addr: 127.0.0.1:9201
                      Listener 1: tcp (addr: "127.0.0.1:9200", cors_allowed_headers: "[]", cors_allowed_origins: "[*]", cors_enabled: "true", max_request_duration: "1m30s", purpose: "api")
                      Listener 2: tcp (addr: "127.0.0.1:9201", max_request_duration: "1m30s", purpose: "cluster")
                      Listener 3: tcp (addr: "127.0.0.1:9202", max_request_duration: "1m30s", purpose: "proxy")
                       Log Level: trace
                           Mlock: supported: false, enabled: false
                         Version: Boundary v0.8.1
                     Version Sha: dd272536554f47c2dcf52cf842da9d7e703dec83
        Worker Public Proxy Addr: 127.0.0.1:9202

==> Boundary server started! Log data will stream in below:

{"id":"7TNhHFQBzv","source":"https://hashicorp.com/boundary/mycontroller/poc-worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"github.com/hashicorp/boundary/internal/observability/event.(*HclogLoggerAdapter).writeEvent","data":{"@original-log-level":"none","@original-log-name":"azurekeyvault","msg":"configuring client automatic mTLS","purpose":"root"}},"datacontentype":"application/cloudevents","time":"2022-05-20T23:52:36.4811986+04:00"}
{"id":"tPafSsaVjj","source":"https://hashicorp.com/boundary/mycontroller/poc-worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"github.com/hashicorp/boundary/internal/observability/event.(*HclogLoggerAdapter).writeEvent","data":{"@original-log-level":"none","@original-log-name":"azurekeyvault","args":["C:\\Users\\<username>\\AppData\\Local\\Temp\\1448580041\\boundary-plugin-kms-azurekeyvault.exe.gz.exe"],"msg":"starting plugin","path":"C:\\Users\\<username>\\AppData\\Local\\Temp\\1448580041\\boundary-plugin-kms-azurekeyvault.exe.gz.exe","purpose":"root"}},"datacontentype":"application/cloudevents","time":"2022-05-20T23:52:36.4811986+04:00"}

And then quickly dies there after:

Error initializing controller: error registering aws host plugin: error creating host plugin: host.(Repository).CreatePlugin: unable to get oplog wrapper: kms.GetWrapper: error loading root key for scope global: kms.loadRoot: error looking up root key versions for scope global: kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: rpc error: code = Unknown desc = keyvault.BaseClient#UnwrapKey: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: error response cannot be parsed: {"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>\r\n<title>404 - File or directory not found.</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}\r\nfieldset{padding:0 15px 10px 15px;} \r\nh1{font-size:2.4em;margin:0;color:#FFF;}\r\nh2{font-si" '\x00' '\x00'} error: invalid character '<' looking for beginning of value
6

Hi there,

I’ve worked this out. Whilst performing the boundary database init - some of the environment variables used to expose the Azure AD creds to the boundary were set incorrectly. Once correctly set - the application starts up without a problem ! Thanks.

1 Like