Quite simply, because chromosome pairing is sequence specific. Holliday Junctions, which are the functional structures of a cross-over, occur through a process called "strand invasion," during which a region of one chromosome physically base-pairs with that of another. Thus one locus cannot pair with a random locus, as there is generally insufficient sequence complimentarity between two random regions to form a functional Holliday Junction. One interesting consequence of this mechanism is gene duplication and deletion in repetitious regions of the chromosome. For example, genes with large repeated regions, such as the gene responsible for Huntington's disease, can expand and contract during homologous recombination due to strand invasion occurring at non-equivalent, but still homologous, sites. Wikipedia does a nice job going over homologous recombination. I also recommend looking over the relevant sections in Molecular Biology of the Cell, available on the PubMed Bookshelf.