This plant seems to be Carissa carandas. With the thorns placements besides leaves, I was sure that this plant belonged to genus _Carissa_. Taking advice from Bryan Hanson, I searched for the plants in the same genus in my locality(Hyderabad, India). The white flowers and their rosy stalk made it easier for me to search. I found out that this is a common plant in country side and is used in many dishes. Thanks Bryan for pointing me in the correct direction.