Allergies are often caused by the immune system reacting to a part of a specific protein, therefore it is in fact a specific protein in the food that is causing the allergic reaction to occur.
Milk allergies are caused by casein and whey (Source: Mayo Clinic)
Oat allergies can be caused by avenin (Source: EJCI)
Soy and egg allergies are caused by their respective albumin proteins. Additionally, egg lysozyme can also cause allergy. (Source: Cleveland Clinic, Food Allergy)