I just ran into this issue after recently disabling IPv6 on my machines, a fix is to add "AddressFamily inet" to the /etc/ssh/sshd_config file and reload the ssh service.