If you first simplify your expression for $x$, you get
$\displaystyle x=\frac{(R-r)(\frac{Rh}{R-r}-y)}{h}=R-\frac{R-r}{h}y,\;\;$ and then
$\displaystyle V=\pi\int_{0}^{h}(R^2-\frac{2R(R-r)}{h}y+\frac{(R-r)^2}{h^2}y^2) \;dy$
$\;\;\;\displaystyle=\pi\left[R^2h-\frac{R(R-r)}{h}\cdot h^{2}+\frac{(R-r)^2}{h^2}\cdot\frac{h^3}{3}\right]$
$\;\;\;\displaystyle=\pi\left[R^2h-R(R-r)h+\frac{1}{3}(R-r)^2h\right]$
$\;\;\;\displaystyle=\frac{\pi}{3}[3R^2h-3R(R-r)h+(R-r)^{2}h]$
$\;\;\;\displaystyle=\frac{\pi}{3}[R^2+Rr+r^2]h$.