Because if one knows the coefficients of $A$ only up to a given precision, a small variation in them will cause a huge variation in the coefficients of $A^{-1}$, hence, presumably, in the solution $x=A^{-1}b$. Alternatively, even if $A^{-1}$ is known with an absolute precision, if one knows the coefficients of $b$ only up to a given precision, a small variation in them will cause a notable variation in the coefficients of the solution $x=A^{-1}b$ since some coefficients of $A^{-1}$ are large. In real life, both effects are often conspiring. See the definition of the condition number of a matrix.