$G'$ will be $(\Bbb R^*, .)$ here $$ \begin{pmatrix} x & y \\\ 0 & z \\\ \end{pmatrix} $$ $\mapsto \frac xz$
Look:
Always there might not be a general way. But here as the matrices are upper triangular so we can easily see $x\
ot = 0$ & $y\
ot = 0$. Again $x,y\in \Bbb R$ So 1st attempt will be in $(\Bbb R^*, .)$. One more thing do the multiplication of the matrices. You will get some idea from the $(1,1)$ & $(2,2)$ place & u have to observe this because your normal subgroup is specified mainly on these two places.
See $ $$ \begin{pmatrix} x & y \\\ 0 & z \\\ \end{pmatrix} $$ $$ \begin{pmatrix} a & b \\\ 0 & c \\\ \end{pmatrix} $$ =$$ \begin{pmatrix} xa & * \\\ 0 & zc \\\ \end{pmatrix} $$ $ So what idea do u get from here?