Nonlinear optimization with rotation matrix constraint I'm trying to optimize the equation || R - W || = minimum where W is a predetermined 3x3 matrix and R is the 3x3 matrix that I'm trying to optimize, with the constraint that it must be a rotation matrix. My knowledge of linear algebra is pretty limited, so I'm not even sure where to begin on approaching this problem?

The answer is $R=UV',$ where $W=USV'$ is the singular value decomposition of $W.$

Assume the norm in $||R-W||^2$ is the Frobenius norm, so $||R-W||^2=\mbox{Trace}((R-W)'(R-W))= \mbox{Trace}(R'R-R'W-W'R+W'W).$ Since $R'R=I$ and $W'W$ is constant, the problem is reduced to maximizing $\mbox{Trace}(WR')= \mbox{Trace}(USV'R')$, where $W=USV'$ is the SVD of $W,$ so $S$ is non-negative and diagonal while $U$ and $V$ are orthogonal matrices. Letting $Q=V'R'U,$ and using the trace property $\mbox{Trace}(AB)=\mbox{Trace}(BA),$ the last expression is $\mbox{Trace}(SQ),$ and we want to maximize this over all rotations $Q.$ Multiplying $S$ by $Q$ rotates the rows of the diagonal matrix $S,$ and thus preserves their length. The length of each row of $S$ is the corresponding diagonal element of $S,$ thus any $Q$ other than $I$ can only reduce the trace of $SQ.$ Thus $Q=I,$ so $V'R'U=I$, so $R=UV'.$