The numbers come from the numbering of the carbon atom of the ribose in the backbone of the DNA. The ribose is numbered as shown in the figure below:
!enter image description here
In the nucleotides the phosphate groups are attached to the 5' carbon atom and new nucleotide will be added at the 3' carbon atom. See the image below:
!enter image description here
First you have a OH-group at the 3' position which is then "replaced" by the new nucleotide.