Yes for $n=3$. This is due to Waldhausen, who proved that the mapping class group of a closed aspherical oriented Haken 3-manifold is identified with $\text{Out}(\pi_1)$; in this case that is $GL_3 \Bbb Z$. There is a proof of Waldhausen's theorem in Hempel's book on 3-manifolds.
Open for $n=4$. Mapping class groups of 4-manifolds are hardly understood. There is some work by Danny Ruberman in the simply connected case showing that they are extremely large, so I might anticipate things to go badly even for the 4-torus. But who knows.
Very false for $n>4$.