Angle between two 3D vectors measured in a specific direction Two normalized 3D vectors, $\vec a$ and $\vec b$, lie in the plane with the normal $\vec n$. By how much should $\vec a$ be rotated anti-clockwise around $\vec n$ to line up with $\vec b$? The $acos$ of the dot product of $\vec a$ and $\vec b$ is not quite it, as that only returns the shortest angle, i.e. the rotation is either clockwise or anti-clockwise.

If $\vec a$, $\vec b$, $\vec n$ are oriented like standard basis vectors $i,j,k$, then the shortest angle is counterclockwise. Otherwise, it is clockwise. This suggests the following algorithm:

1. Compute the triple product of $\vec a$, $\vec b$, $\vec n$, in this order.

2. If the triple product is positive, the answer is $\arccos \frac{\vec a\cdot\vec b }{|\vec a||\vec b|}$. Otherwise it is $2\pi- \arccos \frac{\vec a\cdot\vec b }{|\vec a||\vec b|}$.




(The triple product is zero only when the three vectors lie in the same planar, which is not the case here).