You can add an IP address check to the authorized_keys file on the server, but it’s not there by default. (Do man sshd for more info.)
A passphrase protects your private key. The idea is that if someone was to get a hold of your private key file, it would be useless without also knowing the passphrase. If you don’t use a passphrase, then getting a copy of the private key file will give someone the ability to log into the server.