Dynamically generate Service Principal with Azure secret backend

I am trying to generate dynamic service principal in Azure, this is documented here ( Azure - Secrets Engine | Vault | HashiCorp Developer ).

I have created a Service Principal (SP) with ‘Owner’ role assigned at the subscription level. My understanding is that I can then give this SP to Vault, and have it dynamically create short-lived SP’s that have ‘Contributor’ role assigned at a Resource Group level.

The terraform code for this is below,

resource "vault_azure_secret_backend" "azure" {
  subscription_id = var.subscription_id
  tenant_id       = var.tenant_id
  client_id       = var.client_id
  client_secret   = var.client_secret
  path            = var.vault_azure_path_name
}

locals {
  azure_roles = [
    for rg in var.azure_rg : {
      role_name = "Contributor"
      scope     = "/subscriptions/${var.subscription_id}/resourceGroups/${rg}"
    }
  ]
}

resource "vault_azure_secret_backend_role" "producer" {
  backend = vault_azure_secret_backend.azure.path
  role    = "${var.vault_azure_path_name}-role"
  ttl     = 300
  max_ttl = 600

  dynamic "azure_roles" {
    for_each = local.azure_roles
    content {
      role_name = azure_roles.value.role_name
      scope     = azure_roles.value.scope
    }
  }
}

If I read on the role, I got the following error. I am assuming it is saying my SP (with Owner role) has ‘Insufficient privileges to complete the operation’?

 % vault read dynamic-azure-creds-producer/creds/dynamic-azure-creds-producer-role 
Error reading dynamic-azure-creds-producer/creds/dynamic-azure-creds-producer-role: Error making API request.

URL: GET http://127.0.0.1:8200/v1/dynamic-azure-creds-producer/creds/dynamic-azure-creds-producer-role
Code: 500. Errors:

* 1 error occurred:
	* provider#CreateApplication: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Authorization_RequestDenied" Message="Insufficient privileges to complete the operation." InnerError={"client-request-id":"a94314ed-3ea1-43e6-9b34-a5eb0158523d","date":"2023-06-05T08:08:27","request-id":"a94314ed-3ea1-43e6-9b34-a5eb0158523d"}

But below shows that my SP seems to be fine. (identities were scraped with xxx)

% az login --service-principal -u xxx-4c38-bc18-4029fccafb23 -p xxxyim3qHZLsNTmp~86pI7zfagoQdAt --tenant xxx-43c6-bbc4-ed6a172ad522



[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "xxx-43c6-bbc4-ed6a172ad522",
    "id": "xxx-4b84-9054-68a6ea5f838b",
    "isDefault": true,
    "managedByTenants": [],
    "name": "Azure subscription 1",
    "state": "Enabled",
    "tenantId": "xxx-43c6-bbc4-ed6a172ad522",
    "user": {
      "name": "xxx-4c38-bc18-4029fccafb23",
      "type": "servicePrincipal"
    }
  }
]

% az role assignment list --assignee xxx-4c38-bc18-4029fccafb23 --query "[].{Role:roleDefinitionName, Scope:scope}"
[
  {
    "Role": "Owner",
    "Scope": "/subscriptions/xxx-fbad-4b84-9054-68a6ea5f838b"
  }
]

I looked at the Azure AD sign-in logs, but cannot find anything related to my test logins.

Is there a way to find out exactly what API calls Vault is making to Azure? I used ‘-log-level=trace’ when running the Vault server, but no useful information was logged.

Has anyone seen this issue before? ( Vault v1.13.2, built 2023-04-25T13:02:50Z )?

Thanks,
Chris