The endolymph inside the ear, like any kind of matter, has mass and so is subject to inertia. This means any change in speed and direction will be resisted. The ear and semicircular canals themselves are fairly solidly connected to the rest of the body and so accelerate smoothly. But since endolymph is fluid, it cannot be brought up to speed as fast, so it lags behind in moving. So, as the ear moves forward, the endolymph stays behind and so in relation to the ear, flows backward.
This is like what you experience when a car or airplane accelerates: you "flow" backwards into your seat as the vehicle surges forward.