Answer by Kamil Maciorowski for Does ssh remember private keys?

Does ssh remember private keys?

The answer depends on the implementation of ssh you're using and on your config. Here I will show a possible setup involving ssh from OpenSSH; for this particular setup the simplified answer is yes.

Possible setup

First thing: -i /the/key by itself does not prevent ssh from using other keys. It prevents ssh from falling back to the default keys (~/.ssh/id_rsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ecdsa_sk, ~/.ssh/id_ed25519, ~/.ssh/id_ed25519_sk), but if your config (~/.ssh/config and /etc/ssh/ssh_config) specifies additional key(s) (IdentityFile directive(s)) then it will be as if you used multiple -i options: multiple keys will be tried one by one until one is accepted or the list is depleted.

In your case, if the right key came from the config, it would be used also after the reboot. This did not happen, so there must have been something else.

Another source of keys (even if you use -i) may be an agent. Please read "understanding ssh-agent" in this other answer of mine. It does not have to be exactly ssh-agent, there are other programs that can take its role. Anyway, there may be an agent running.

Let's assume there is an agent running. When you use ssh -i with a key (e.g. with the right key in your case), ssh may or may not add the key to the agent. This is governed by the AddKeysToAgent option:

Specifies whether keys should be automatically added to a running ssh-agent(1). If this option is set to yes and a key is loaded from a file, the key and its passphrase are added to the agent with the default lifetime, as if by ssh-add(1).

The default value for AddKeysToAgent is no, but if the admin put AddKeysToAgent yes into /etc/ssh/ssh_config (or if this value came with the distro) or if you put the line into your ~/.ssh/config (and maybe forgot), then the value may be yes for your ssh.

Let's assume the value is yes and thus the right key has been added to the agent. If you use ssh again, this time with -i /wrong/key, then the key from the agent may or may not be used. This is governed by the IdentitiesOnly option:

Specifies that ssh(1) should only use the configured authentication identity and certificate files (either the default files, or those explicitly configured in the ssh_config files or passed on the ssh(1) command-line), even if ssh-agent(1) or a PKCS11Provider or SecurityKeyProvider offers more identities.

If this option is no (the default) then keys stored in the agent will be used. (Note there are other possible providers, but I won't elaborate).

All this means that if there is an agent running and if AddKeysToAgent is yes (or similar) and if IdentitiesOnly is no then what you observed is the expected behavior. In this case the simplified answer to your question is yes. The non-simplified strict answer is: no, sshitself does not remember private keys (but ssh-agent does).

How to confirm

To see if there is an agent, run:

env | grep -E '^SSH_AUTH_SOCK|^SSH_AGENT_PID'

If SSH_AUTH_SOCK exists in the environment then most likely an agent is running. It is possible your desktop environment runs an agent for you every time you log in to it and you were not aware up to this point. If SSH_AGENT_PID exists in the environment then you can investigate further with e.g. ps -u -p "$SSH_AGENT_PID".

You can see the relevant config of ssh (from OpenSSH) by running ssh -G:

ssh -G -i /tmp/hoge git@github.com | grep -iE 'identityfile|addkeystoagent|identitiesonly'

Note that the output may include different IdentityFile lines, depending on whether or not /tmp/hoge exists. The file existed when you observed the "issue", so make sure it exists now when you're using ssh -G.

ssh -G will not show you which keys have been added to the agent. To see them, query the agent:

ssh-add -l

Reboot and replicate the "issue", invoking ssh-add -l before and after each ssh. This way you will know if this answer explains what happens in your particular case.

Even if this is not (or not exactly) what happens in your particular case, the scenario is plausible and the answer may be useful for others.