"you're not exactly avoiding a shell on the server side (not even with SFTP!)" -- Except when the SSH server uses
Subsystem sftp internal-sftp. In this case one can even use SFTP to fix the remote user's broken startup scripts.