You are right, the covariance of a random matrix $X$ is given by $E[(X-EX)(X-EX)']$, however note that in a context of regression analysis we are interested, inter alia, in the variance of the coefficients estimators. Consider a model $Y=X\beta + \epsilon$, with $var(\epsilon) = \sigma^2I$. \begin{align} var(\hat{\beta}|X)&=var((X'X)^{-1}X'y|X)\\\ &= (X'X)^{-1}X'var(y|X)X(X'X)^{-1}\\\ &= (X'X)^{-1}X'E(\epsilon\epsilon')X(X'X)^{-1}\\\ & = \sigma^2(X'X)^{-1}XX(X'X)^{-1}=\sigma^2 (X'X)^{-1}. \end{align}