Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource when running packer build


I am trying to create an KMS Key encrypted image using packer. I have created a key and I am providing the key in the packer build command using this.

packer build -machine-readable -var "key_name=$key_fqdn" -var "account_file=$GOOGLE_APPLICATION_CREDENTIALS" -var "zone=$zone" -var "ami=$ami" -var "project_name=$project_name" -var "ami_name=$ami_name" build.json

By build.json looks like this.

    "builders": [
    "type": "googlecompute",
    "project_id": "{{user `project_name`}}",
    "source_image": "{{user `ami`}}",
    "ssh_username": "packer",
    "subnetwork": "default",
    "account_file": "{{user `account_file`}}",
    "image_name": "{{user `ami_name`}}",
    "zone": "{{user `zone`}}",
    "machine_type": "n2-standard-8",
    "disk_size": "50",
    "image_encryption_key": {
        "kmsKeyName": "{{user `key_name`}}"

However, I am getting error like this.

1621956469,,ui,error,==> googlecompute: Error waiting for image: googleapi: Error 400:
 Cloud KMS error when using key projects/my-project/locations/us-central1/keyRings/my-keyring/cryptoKeys/my-key/cryptoKeyVersions/1: 
Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 
'projects/my-project/locations/us-central1/keyRings/my-keyring/cryptoKeys/my-key' (or it may not exist).%!(PACKER_COMMA) kmsPermissionDenied

Though, My key has permission for cloudkms.cryptoKeyVersions.useToEncrypt

Is packer using some other service account in this case ?