Minimize dot product with constrain using Lagrange multiplier theorem Question is Minimize $||x||^2$ subject to $x^TQx=1$, ($x \in R^n$, $Q$ is positive definite matrix) First order necessary condition for Lagrange is $2x-\lambda(2Qx)=0$ It implies $(I-\lambda Q)x=0$. Since x is not zero by constrain, $(I-\lambda Q)=0$. But $\lambda $ cannot be $Q^-1$ because $\lambda$ is scalar. How do I solve this problem?

You want $(I-\lambda Q)x$ to vanish for some nonzero $x$. It is equivalent to say that $\lambda^{-1}$ is an eigenvalue of $Q$ with associated eigenvector $x$.

Now $x^T Q x = \lambda^{-1} x^T x$. Thus $x^T x = \lambda x^T Q x = \lambda$. So to minimize $x^T x$ subject to this constraint, simply choose it to be the eigenvector with the largest eigenvalue, subject to your normalization condition.

Note by the way that your problem is essentially equivalent to maximizing $x^T Q x$ subject to $x^T x = 1$, which is well known to be solved in this exact way. It is not quite the same (the scaling is different), but still, the connection can be made.