**First Number** = Nominal Outer Diameter of tire, including tread.
**Second Number / Letter** = Nominal width of tire at the start of the tread.
Tire size is determined by the outside diameter of the tire, as regards the 26 inch or 700c number. The 2.125 inch number is the nominal width of the tire at the point where the tread starts.
These are nominal measurements, though, and there is no universal standard applied consistently other than bead diameter.
700c x 23 tires from Schwalbe will measure 28mm in width, and about 5 mm extra in diameter. From Conti, its about 24mm and 1-2mm extra. There is no stadard for point of measurement. It is left to the manufacturer to decide, even before the marketing department gets involved.