What's the derivative of F over R? $F(R) = \|A-RY\|^2_F$ what should $\partial F \over \partial R$ looks like? Since $\partial \|X\|^2_F \over \partial X$ is $2X$, so $\partial \|A-RY\|_F^2$ should be $2(A-RY){ \partial RY \over \partial R} $? What will $\partial RY \over \partial R$ looks like? Thanks

Assume that $||U||_F^2=trace(FF^T)$, the Frobenius norm. Here $F(R)=trace((A-RY)(A-RY)^T)$. Its derivative is $DF_R:H\rightarrow trace((-HY)(A-RY)^T+(A-RY)(-HY)^T)=trace(-2HY(A-RY)^T)=trace(-2Y(A-RY)^TH)$.

If you seek a gradient, then we consider the gradient $\
abla_R(F)$ associated to the scalar product $(U,V)=trace(U^TV)$. It is defined as follows: for every $H$, $(\
abla_R(F),H)=DF_R(H)$. Thus $\
abla_R(F)=-2(A-RY)Y^T$.