For the forward direction: The case $n=2$ is easy. Assume $n$ is an odd prime. We know $(n-1)!$ is a product of units mod $n$. If you pair up the elements which are not their own inverses mod $n$ they cancel. You are left with a product of elements which are their own inverses. But the only such elements are $1$ and $n-1$ since there can be at most two square roots of $1$ over any field.
For the backward direction: If $n$ is not prime then one of the numbers $1,\dots,n-1$ is a zero divisor mod $n$ so the product cannot be congruent to $-1$ mod $n$.