is a perfectly natural way of saying "What's your specialty?", but this question would make sense only to professional scholars, physicians and such. You would expect answers like "algebraic topology", "plasma cosmology", "pediatric hematology", etc.
To ask one's major to college students, works, but it's better to ask , because usually refers to true experts. To ask about someone's job in general, see the answer of @IgorSkochinsky.