Your approach seems fine.
Alternatively,...
In each seat, the probability the person sitting there shakes the hand to the left is $$\frac{x}{x+y}\frac{x-1}{x+y-1} + \frac{y}{x+y}\frac{y-1}{x+y-1}=\frac{x(x-1)+y(y-1)}{(x+y)(x+y-1)}$$
So if $X_i$ is the number of handshakes to the left from seat $i,$ we have that $X_i=0\text{ or }1.$ And $X=X_1+\cdots+X_{x+y+1}$ is the total number of handshakes to the left from any seat, which is the total number of seats.
But $E(X)=\sum E(X_i),$ and $$E(X_i)=P(X_i=1)=\frac{x(x-1)+y(y-1)}{(x+y)(x+y-1)}$$
So: $$E(X)=(x+y)E(X_1)=\frac{x(x-1)+y(y-1)}{x+y-1}$$