A mature human sperm has some mitochondria, a nucleus, an acrosome, and a flagellar apparatus. That's it. Because of the lack of other organelles in the sperm, they have to be kept in a kind of 'cold storage' so that they are viable for at least some time (usually a couple of days). The ideal temperature would be 1.8 °C to 2.5 °C below normal body temperature, i.e. 37.6 °C or 98 °F. If a man takes frequent hot water baths, his sperm count becomes low as the mature sperms get dessicated quickly.
The reason that women don't require this strict temperature regulation is that the mature ovum has the complete cellular machinery at its disposal- mitochondria, nucleus, golgi bodies, ER, the works. So it can remain viable for a much longer time (about seven days after ovulation) and has higher temperature tolerance than sperms.