Why not simply $$ ax+by+c=0 \implies ar\cos\theta+br\sin\theta+c=0\implies r=-\frac{c}{a\cos\theta+b\sin\theta} $$
Anyway, you should perform the derivative, obtaining $$ \frac{r}{\sqrt{r'^2+r^2}}-\frac{r''}{\sqrt{r'^2+r^2}}+\frac{r'(2r'r''+2rr')}{2\sqrt{(r'^2+r^2)^3}} = 0 $$ and further simplify to $$ -\frac{r(rr''-2r'^2-r^2)}{\sqrt{(r'^2+r^2)^3}} = 0 $$