One of the errors is implying that packer is trying to use the path string as the checksum itself. This is why the default vault starts with file: to specify that the checksum is stored in a file. The iso_filename variable value is only a path. Have you modified the variable default values at all? Are you 100% sure that you are running the packer command from the root of the git repo? EG the following succeeds from the same path that your running packer from?
Alternatively, you can specify the shasum and the absolute path to the iso on the command line.
packer build -force -only=base.vmware-iso.macOS_11_base -var iso_filename=/Users/blake/packer-macos-11/install_bits/macOS_1101_installer.iso -var iso_file_checksum=sha256:7319b89db22a6e0dabbbb40b0a9899d93cb571645152985d3204c4f52b399438 macOS_11.pkr.hcl
Seems like something external to packer and this template is causing the issue you are seeing. You should investigate things like macOS access controls and maybe test giving the packer binary full disk access? Big Sur has lots of new restrictions and you could be hitting those as well.