Since $X,Y$ are both almost surely positive, then: $$\begin{align}\mathsf P(X>\surd Y) {}&= \mathsf P(X^2>Y) \\\ &= \int_0^1\int_0^{x^2} (x+y)~\mathsf dy~\mathsf d x \end{align}$$
From this you should immediately anticipate a _rational_ result.
* * *
However, it looks like you tried directly: $$\begin{align}\mathsf P(X>\surd Y) {}&= \int_0^1\int_{\surd y}^1 (x+y)~\mathsf dx~\mathsf d y \end{align}$$
This is _slightly_ harder to do by hand, because _fractions_. But if you persevere and take care, it does give the same result.