The definition in your first paragraph doesn't match your understanding in the second. If osmotic pressure is high in "A" relative to "B", you would _have to apply a physical pressure to "A" to prevent solvent moving from B to A_. If there is no such pressure applied, then solvent does move from B to A. The osmotic pressure and physical pressure are separate and opposite forces.
I prefer to think of osmotic pressure as sort of a "vacuum" that "pulls" solvent towards it (of course it isn't really a vacuum so don't take this analogy too far...). The definition still works given this form of thinking: you'd have to apply as much external pressure to equal the "vacuum" in order to have no movement of solute.