Synapses are held together by adhesion molecules, like cadherins and neuroligins/neurexins, so they aren't just loose adjacent membrane, they are securely anchored in place.
The adhesion molecules serve both a structural and regulatory role, being important for synapse formation and plasticity.
Missler, M., Südhof, T. C., & Biederer, T. (2012). Synaptic cell adhesion. Cold Spring Harbor perspectives in biology, 4(4), a005694.