I've followed several different guides on how to configure two factor authentication on my ssh server using the google's libpam-google-authenticator. I've been able to get it to work but when I try and log in from my desktop or any other device over the ssh I have to give it my ssh-key, user password, and then the authenticator token. But in my sshd_config file I have:

AuthenticationMethods publickey,keyboard-interactive  

ChallengeResponseAuthentication yes  

PasswordAuthentication no   

UsePAM yes

And in my /etc/pam.d/sshd I added

auth required pam_google_authenticator.so

So I think it should not ask for password as well. How can I fix it?