It doesn't matter.
Suppose $M$ is a 3-manifold with torus boundary and $N = S^1 \times D^2$ is the solid torus. Then if $\varphi,\psi: \partial M \to \partial N$ are diffeomorphisms, if $\psi \circ \varphi^{-1}$ extends to a diffeomorphism of $N$, then $M \cup_\varphi N \cong M \cup_{\psi} N$.
Now if $\varphi$ is an orientation-preserving (or reversing) diffeomorphism, and $r$ is the diffeomorphism represented by the matrix $$\begin{bmatrix} 1 & 0 \\\ 0 & -1 \end{bmatrix}$$
then $\varphi \circ r$ is an orientation-reserving (or preserving) diffeomorphism, and $M_\varphi N \cong M_{\varphi r} N$, because $r$ extends over $N$.
So you get the same manifolds no matter whether you demand some orientation restriction or not.