SSH Agent Forwarding not working with Ansible Provisioner

I am using Ansible Provisioner with Packer and one of the task in playbook is to clone a github private repository with Ansbile git module. I am getting permission denied error. I tried to modify the ansible.cfg file and added

ssh_args = -o ForwardAgent=yes

But still I am unable to clone the repo. Can someone please help here?