First,
$$3(a-b)^2=3\left(a^2-2ab+b^2\right)=3a^2-6ab+3b^2$$
by simple algebra. Now recall that we’re assuming that $\sqrt3=\frac{a}b$, so that $a^2=3b^2$. Thus,
$$3a^2-6ab+3b^2=3\cdot 3b^2-6ab+a^2\;,$$
where we’ve substituted $3b^2$ for $a^2$ in the first term and $a^2$ for $3b^2$ in the last. The last step is again just simple algebra to factor $9b^2-6ab+a^2$ as $(3b-a)^2$.
The $3(a-b)$ actually does in a sense come out of thin air at this point. More accurately, it comes from looking ahead to what we want to do: we want to be able to say that
$$\sqrt3=\frac{3b-a}{a-b}\;,\tag{1}$$
because we can show that the denominator $a-b$ is positive and smaller than $b$ and in that way get a contradiction. $(1)$ is true if and only if $3(a-b)^2=(3b-a)^2$, so we prove the latter. And if you think that it’s not exactly obvious that $(1)$ should hold and that it would therefore be useful to prove that $3(a-b)^2=(3b-a)^2$, you’re right.