scp uses ssh as transport, regardless if it uses SCP or SFTP (see "History, SCP and SFTP" in this answer for insight). In ~/.ssh/config you can define two Hosts pointing to the same Hostname and Port, one using, the other not using the key; then use one host with scp, the other with ssh. Still, if the key leaks out then an attacker will be able to ssh, you are only limiting yourself. Even if you restrict the key to (e.g.) command="internal-sftp" on the server, it won't be enough.