There seems to be some missing context here. If $x$ is a free variable, and $a,b$ are distinct ground terms, then $x$ cannot be unified with both $a,b$. That is why (under the given assumed context) unification is not possible between $P(x,x)$ and $P(a,b)$.
However if $a,b$ were free variables, then they could be unified with $x$ (and each other), and in that context $P(x,x)$ and $P(a,b)$ would unify successfully.