The correct one is $$ \vec a^T \vec a = \|\vec a\|^2. $$ The intuition behind this is that the quantity on the left is quadratic with respect to $\vec a$, so the right hand side must be quadratic as well (which would not be the case for $\|\vec a\|$): Take real $t>0$, then $$ (t\vec a)^T(t\vec a) = t^2 \vec a^T \vec a = t^2 \|\vec a\|^2 = \|t \vec a\|^2 $$ holds for all $t>0$. This computation would not work with the incorrect $\vec a^T \vec a = \|\vec a\|$.