It is related to Eustachian tube. It links the back of throat and the middle ear and allows air pressure to equalize in the middle ear. When you yawn air pressure goes up in this and it bends the ear drum and causes impair hearing (notice, just impair and not stop). Yawning also helps to open Eustachian tube.