I recommend looking up the formula on Wikipedia, which makes clear that the $r^2$ in your formula is not the squared radius of the sphere. Rearranging gives the last expression in the article of $$V_{\text{cap}}=\pi \frac{h^2}{3}(3r_\text{radius}-h)$$ Here $h$ is the depth by which the sphere is submerged.
* * *
The mass of the whole buoy is $\frac 4 3 \pi r_\text{radius}^3 * \frac 1 4 \rho_{\text{water}}$.
The mass of water displaced is $V_{\text{cap}} \rho_{\text{water}}$.
When are these equal? You should find an equation $h^3-3h^2+1=0$. This equation has three real roots, of which the physical one is the one in $[0,2]$, namely $0.652704\ldots$. (This has an exact expression involving complex numbers but I suspect you were meant to approximate the solution somehow.)