For example, let $$R=\begin{pmatrix}r_{11}&r_{12}&r_{13}\\\ r_{21}&r_{22}&r_{23}\\\ r_{31}&r_{32}&r_{33}\end{pmatrix}$$ be the $3\times 3$ rotation matrix, $$\vec{t}=\begin{pmatrix}t_1\\\t_2\\\t_3\end{pmatrix}$$ be the $3\times 1$ translation vector, $s$ be the scale factor. If the order is rotation, translation, then scaling, the matrix should be $$A=s\begin{pmatrix}r_{11}&r_{12}&r_{13}&t_1\\\ r_{21}&r_{22}&r_{23}&t_2\\\ r_{31}&r_{32}&r_{33}&t_3\\\ 0&0&0&1\end{pmatrix}$$
If otherwise, the order is rotation, scaling, then translation, the matrix should be
$$A=\begin{pmatrix}sr_{11}&sr_{12}&sr_{13}&t_1\\\ sr_{21}&sr_{22}&sr_{23}&t_2\\\ sr_{31}&sr_{32}&sr_{33}&t_3\\\ 0&0&0&1\end{pmatrix}$$
Then $$\begin{pmatrix}x_1'\\\x_2'\\\x_3'\\\1\end{pmatrix}=A\begin{pmatrix}x_1\\\x_2\\\x_3\\\1\end{pmatrix}$$