**Hint**. Your integral is **convergent**.
Potential problems are as $\| x\| \to +\infty$ and as $\coth (\| x\|) \to +\infty$ ($\| x\| \to 0$).
* As $\| x\| \to +\infty$, we have $$ -\| x\|^2- a \| x\| \coth (\| x\|) -\| x\| =-\| x\|^2+O(\| x\|) \sim -\| x\|^2 $$ and your integral is convergent since, for some $M>0$, it behaves as $$ \int_{\mathbb R^3 \:\cap\: \| x\|\geq M} e^{-\| x\|^2}dx <\int_{\mathbb R^3} e^{-\| x\|^2}dx <+\infty.$$
* As $\| x\| \to 0$, we have $$ -\| x\|^2- a \| x\| \coth (\| x\|) -\| x\| =-a+O(\| x\|) $$ and your integral is convergent since, for some $\epsilon>0$, it behaves as $$ \int_{\mathbb R^3 \:\cap\: \| x\|\leq \epsilon} e^{-a}dx=\int_{\| x\|\leq \epsilon} e^{-a}dx <+\infty.$$