Before answering your question, few comments:
a) It is not possible to invert a matrix that has not an equal number of rows and columns; so it is not possible to invert $R$ which not a square matrix.
b) Even if the matrix has an equal number of rows and columns it is not always possible to compute the inverse: it must also be full rank, or its determinant must but nonzero. If $rank(X)=K$ then $X$ has an inverse (and conversely).
c) It is well known that $rank(AB) \leq \min(rank(A),rank(B))$.
These rules imply that your matrix $M = FRXR'F'$ is $(N\times N)$ and has an inverse matrix if $rank(M)=N$, this requires that (i) $N\leq K$ (see condition c), that (ii) $rank(F)=N$ and (iii) $rank(R'XR)=N$. Under these conditions
$$(FRXR'F')^{-1}=(F')^{-1}(RXR')^{-1}(F)^{-1}.$$