How to rotate two vectors (2d), where their angle is larger than 180. The rotation matrix $$\begin{bmatrix} \cos\theta & -\sin \theta\\\ \sin\theta & \cos\theta \end{bmatrix}$$ cannot process the case that the angle between two vectors is larger than $180$ degrees. (counter-clockwise rotation).

The matrix $M(\theta)$, where $$M(\theta) = \left[ \begin{array}{cc} \cos\theta & -\sin\theta \\\ \sin\theta & \cos\theta \end{array}\right]$$

Can take any value of $\theta$ whatsoever. For example, let $\theta = 270^{\circ}$ then we have $\sin\theta = -1$ and $\cos\theta =0$ giving $$M(270^{\circ}) = \left[ \begin{array}{cc} 0 & 1 \\\ -1 & 0 \end{array}\right]$$ The vector $[1,0]^{\top}$ gets sent to $[0,-1]^{\top}$ while the vector $[0,1]^{\top}$ to $[1,0]^{\top}$. Notice, moreover, that $\det(M(\theta)) = 1$ for all $\theta$. This is exactly an anti-clockwise rotation of $270^{\circ}$.