Speaker is a mother and the father is being spoken to if we disregard other sexual orientations.
Suppose $x$ is the speaker and $y$ is being spoken to. $x$ must be a parent. Suppose $x$ is the father. Then $x$ is referring to himself as he says "father of my son". So $x$ is the only child of $y$'s parents i.e. $x=y$ and $x$ is talking to himself. Therefore $x$ can't be the father $\implies x$ is the mother. Now the only child of $y$'s parents is $y$. So $y$ is the father.