Instead of the Hamiltonian, I can give you the Lagrangian.
In the existence of friction, it is well known to physicists that the following Lagrangian gives the correct equation of motion.
$$L[x,\dot{x},t] = e^{\lambda t/m} \bigg(\frac{1}{2}m\dot{x}^2 - U(x)\bigg)$$
$$\Rightarrow m\ddot{x} +\lambda \dot{x} +U'(x)=0 $$
In your case, $m=1$ and $U(x)=-\frac{1}{2}x^2+\frac{1}{4}x^4$.
You can easily figure out the Hamiltonian.
Otherwise, I would start from defining the first order differential equations in the following way,
\begin{align} e^{\lambda t}\dot{x} &= y \\\ e^{-\lambda t} \dot{y} &= x-x^3 \end{align}