Packer source/build condition

I have a template with vmware-iso and amazon-ebs sources.
I’d like the always build the amazon-ebs but conditionally build the vmware-iso ones.
How can I accomplish this without having to create 2 separate templates and 2 separate build jobs?