the sailboat is at $a$ and the motor-boat is at $b$.
you are using $c$ for distance.
$c = \sqrt {a^2 + b^2}\\\ \frac {dc}{dt} = \frac {a \frac {da}{dt} + b\frac{db}{dt}}{c}$
$a<0$ (to the left of the origin) and $\frac {da}{dt} > 0$ (headed toward the origin)
$b>0$ (above of the origin) and $\frac {db}{dt} < 0$ (headed toward the origin).
$\frac {dc}{dt} < 0$ since the boats are getting closer together $c$ is getting smaller. (and both terms in the numerator are $<0$ while the denominator is strictly positive)