Hyper-V VMCX builder fails to modify memory

I’ve successfully made a VM from an ISO and exported it.
Now trying to build a VM from a vmcx file fails when trying to modify memory.
Error:
Debug mode enabled. Builds will not be parallelized.
hyperv-vmcx: output will be in this color.

==> hyperv-vmcx: Creating build directory... 
==> hyperv-vmcx: Pausing after run of step 'StepCreateBuildDir'. Press enter to continue.
==> hyperv-vmcx: Pausing after run of step 'StepOutputDir'. Press enter to continue.
==> hyperv-vmcx: Pausing after run of step 'StepCreateFloppy'. Press enter to continue.
==> hyperv-vmcx: Pausing after run of step 'StepHTTPServer'. Press enter to continue.
==> hyperv-vmcx: Creating switch 'vSwitch' if required...
==> hyperv-vmcx:     switch 'vSwitch' already exists. Will not delete on cleanup...
==> hyperv-vmcx: Pausing after run of step 'StepCreateSwitch'. Press enter to continue.
==> hyperv-vmcx: Cloning virtual machine...
==> hyperv-vmcx: Error cloning virtual machine: PowerShell error: Hyper-V\Set-VMMemory : Failed to modify device 'Memory'.
==> hyperv-vmcx: 'Win10Ent2019LTSC_GOLD' failed to modify device 'Memory'. (Virtual machine ID 38C534CD-4FB7-485F-8632-D15D78C753E8)
==> hyperv-vmcx: At C:\Users\usx00014\AppData\Local\Temp\2\powershell147704452.ps1:44 char:1
==> hyperv-vmcx: + Hyper-V\Set-VMMemory -VM $compatibilityReport.VM -StartupBytes $memor ...
==> hyperv-vmcx: + 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
==> hyperv-vmcx:     + CategoryInfo          : InvalidOperation: (:) [Set-VMMemory], VirtualizationException
==> hyperv-vmcx:     + FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.SetVMMemory
==> hyperv-vmcx: keep_registered set. Skipping unregister/deletion of VM.
==> hyperv-vmcx: Pausing before cleanup of step 'StepCreateSwitch'. Press enter to continue.
==> hyperv-vmcx: Pausing before cleanup of step 'StepHTTPServer'. Press enter to continue.
==> hyperv-vmcx: Pausing before cleanup of step 'StepCreateFloppy'. Press enter to continue.
==> hyperv-vmcx: Pausing before cleanup of step 'StepOutputDir'. Press enter to continue.
==> hyperv-vmcx: Deleting output directory...
==> hyperv-vmcx: Pausing before cleanup of step 'StepCreateBuildDir'. Press enter to continue.
==> hyperv-vmcx: Deleting build directory...
==> hyperv-vmcx: Error deleting build directory: remove C:\Users\usx00014\AppData\Local\Temp\2\hyperv771964783\Virtual Machines\38C534CD-4FB7-485F-8632-D15D78C753E8.vmcx: The process cannot access the file because it is being used by another process.
Build 'hyperv-vmcx' errored: Error cloning virtual machine: PowerShell error: Hyper-V\Set-VMMemory : Failed to modify device 'Memory'.
'Win10Ent2019LTSC_GOLD' failed to modify device 'Memory'. (Virtual machine ID 38C534CD-4FB7-485F-8632-D15D78C753E8)
At C:\Users\usx00014\AppData\Local\Temp\2\powershell147704452.ps1:44 char:1
+ Hyper-V\Set-VMMemory -VM $compatibilityReport.VM -StartupBytes $memor ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (:) [Set-VMMemory], VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.SetVMMemory

==> Some builds didn't complete successfully and had errors:
--> hyperv-vmcx: Error cloning virtual machine: PowerShell error: Hyper-V\Set-VMMemory : Failed to modify device 'Memory'.
'Win10Ent2019LTSC_GOLD' failed to modify device 'Memory'. (Virtual machine ID 38C534CD-4FB7-485F-8632-D15D78C753E8)
At C:\Users\usx00014\AppData\Local\Temp\2\powershell147704452.ps1:44 char:1
+ Hyper-V\Set-VMMemory -VM $compatibilityReport.VM -StartupBytes $memor ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (:) [Set-VMMemory], VirtualizationException
+ FullyQualifiedErrorId : InvalidState,Microsoft.HyperV.PowerShell.Commands.SetVMMemory

==> Builds finished but no artifacts were created.

I have tried passing in user value for memory as a string as below, but also tried with memory value hardcoded with 2 as an int with the same error.

{
  "variables": {
  "vm_name": "Windows10Ent_01",
  "ram": "4196",
  "cpu": "2",
  "admin_password": "",
  "domain_username": "",
  "domain_password": "",
  "unattend_folder": "Win10Ent2019LTSC_GOLD",
  "vlan_id": "none",
  "scripts_dir": "\\\\devops-hyperv-2\\VDI\\Golden_Images\\Desktop\\Packer\\scripts",
  "os_ver": "",
  "collection_name": "{{user `group_name`}}_{{user `os_ver`}}",
  "group_name": "",
  "users_file": "\\\\devops-hyperv-2\\VDI\\Golden_Images\\Desktop\\Packer\\user_lists\\userlist.txt"
},

"builders": [
{
  "vm_name": "{{user `vm_name`}}",
  "boot_wait": "6m",
  "communicator": "winrm",
  "cpus": "{{user `cpu`}}",
  "memory": "{{user `ram`}}",
  "winrm_username": "Admin",
  "winrm_password": "{{user `admin_password`}}",
  "winrm_timeout": "4h",
  "type": "hyperv-vmcx",
  "clone_from_vmcx_path": "\\\\devops-hyperv-2\\VDI\\Golden_Images\\Desktop\\Win10Ent2019LTSC_GOLD",
  "switch_name": "vSwitch",
  "vlan_id": "{{user `vlan_id`}}",
  "keep_registered": "true",
  "output_directory": "\\\\devops-hyperv-2\\VDI\\Instantiated_Images\\Desktop\\{{user `group_name`}}\\{{user `vm_name`}}",
  "shutdown_command": "{{user `scripts_dir`}}\\ShutdownForSysprep.bat {{user `vm_name`}} {{user `admin_password`}} {{user `domain_username`}} {{user `domain_password`}} {{user `unattend_folder`}}"
  }
],

"provisioners": [
  {
  "scripts": [
    "{{user `scripts_dir`}}\\enable-rdp.bat",
    "{{user `scripts_dir`}}\\pin-powershell.bat",
    "{{user `scripts_dir`}}\\compile-dotnet-assemblies.bat",
    "{{user `scripts_dir`}}\\uac-enable.bat",
    "{{user `scripts_dir`}}\\dis-updates.bat"
  ],
  "type": "windows-shell"
  },
{
  "scripts": [
    "{{user `scripts_dir`}}\\set-powerplan.ps1",
    "{{user `scripts_dir`}}\\vdi.ps1",
    "{{user `scripts_dir`}}\\Complete-setup.ps1"
  ],
  "type": "powershell",
  "environment_vars": [
    "VM_NAME={{.vm_name}}",
    "GROUP_NAME={{.group_name}}",
    "COLLECTION_NAME={{.collection_name}}"
    ]
  }
 ] 
}