You should assume a binomial of the form whose root you desire. In this case, you should have supposed the root is $$\sqrt x -\sqrt y$$ instead.
To be specific, the problem in the above calculation is in your step ii, where you set $$-\sqrt{12}=\sqrt{xy}.$$ But this is impossible if you're dealing only with real numbers. It seems you need to note that the symbol $\sqrt{}$ denotes a function which, by definition, assumes nonnegative values. Thus, you can see that your equation is false, for it says a negative number is equal to a nonnegative one. That's a contradiction.