The problem is likely an intervening firewall with a timeout.
You can try telling ssh to keep the connection active by appending to ~/.ssh/config:
Host *
ServerAliveInterval 90
It could also be intermittent failures of the network. From the sshd_config man page regarding TCPKeepAlive (on by default):
> this means that connections will die if the route is down temporarily, and some people find it annoying.