That is the joke, but a related fact is that systematizing and simplifying the sign (such as eliminating it, or making it independent of $p$ and $q$) was part of the motivation for using other forms of the symbol instead of Legendre's. For example, $(p,q)=(q,p)$ for the Hilbert symbol.