Yes you doing in the right way.
Given $z+\frac{20i}{w}=5+i$ and $w+\frac{12i}{z}=-4+10i$ $$zw+32i-\frac{240}{zw}=-30+46i$$ $$(zw)^2+(30-14i)zw-240=0$$
Now use the quadratic formula $x=\frac{-b\pm \sqrt{b^2-4ac}}{2a}$ and we get
Here $a=1,b=(30-14i),c=-240$ $$=\frac{-(30-14i)\pm \sqrt{(30-14i)^2-4(1)(-240)}}{2(1)}$$ $$=\frac{-2(15-7i)\pm \sqrt{(2(15-7i))^2-4(1)(-240)}}{2(1)}$$ $$=\frac{-2(15-7i)\pm 2\sqrt{(15-7i)^2+240}}{2(1)}$$ Note that $\sqrt{(15-7i)^2+240}=\sqrt{416-210i}=\sqrt{(5i-21)^2}=5i-21$ $$7i-15\pm\sqrt{(15-7i)^2+240}=6+2i,12i-36$$
Now $$|zw|^2=6^2+2^2=40$$ and $$|zw|^2=12^2+36^2>40$$ So, the smallest value is $40$