Does $HOX = OX$ imply $HO^TX = O^TX$? Suppose $X \in \mathbb{R}^{n \times p}$ has rank $p$, and let $H = X(X^TX)^{-1}X^T$ be the corresponding projection matrix. If there is an orthogonal matrix $O \in \mathbb{R}^{n \times n}$ such that $HOX = OX$, I am wondering if its inverse satisfies this identity as well, i.e., $$HO^TX = O^TX. \tag{*}$$ This problem stems from a statement from a statistics text book, claiming that under the conditions listed above, the set of orthogonal matrices $$\mathcal{G} = \\{O \in \mathcal{O}: HOX = OX\\}$$ forms a group (I am assuming the group operation is naturally defined as matrix multiplication). To verify this, it is necessary to show $(*)$ holds so that every member in $\mathcal{G}$ has its inversion element also in $\mathcal{G}$. This seemingly trivial fact turns out quite challenging to me. Is it really technical or might $\mathcal{G}$ fail to be a group?

Phrased like that, it looks hard. But one can see that $$\tag1 \mathcal G=\\{O:\ O\mathcal X=\mathcal X\\}, $$ where $\mathcal X$ is the range of $X$.

Now the equality $O\mathcal X=\mathcal X$ is the same as $O^T\mathcal X=\mathcal X$, and if $U,V\in\mathcal G$ then $UV\mathcal X=U\mathcal X=\mathcal X$. So $\mathcal G$ is a group.

* * *

_Proof of $(1)$_

If $O\mathcal X=\mathcal X$, then for any $v\in\mathbb R^p$, there exists $w\in\mathbb R^p$ with $OXv=Xw$; as $HX=X$, we get $HOXv=HXw=Xw=OXv$. As this works for all $v$, we get $HOX=OX$.

Conversely, if $HOX=OX$, then for any $v\in\mathbb R^p$ we have $OXv=HOXv\in\mathcal X$. So $O\mathcal X\subset \mathcal X$. As $O$ preserves dimension (being injective), $O\mathcal X=\mathcal X$.