Note that at least on Linux, deploying a keylogger to exfiltrate the password for a SSH key is not hard either. Even if a keylogger is somehow not possible, you can probably still replace key binaries with patched versions or change desktop shortcuts to launch modified programs.