The real $n \times n$ matrices $M_n(\mathbb{R})$ form a normed vector space. (Take the operator norm or view a matrix as a vector in $\mathbb{R}^{n^2}$ and take the usual norm there.) For finite-dimensional spaces all norms are equivalent and so they induce the same topology. Thus an open set in $M_n(\mathbb{R})$ is a union of open balls with respect to the metric $d(A,B) = \|A-B\|$, where $\| \cdot \|$ is any norm. The topology on $O(p,q) \subset M_n(\mathbb{R})$ ($p+q = n$) is just the subspace topology.