When it comes to mammals this cannot really be evaluated without taking the Megafaunal mass extinction into account. This mainly refers to the extinction of large mammals from the Pleistocene Megafauna, which affected different parts of the world differently. More species were extinct in the Americas, Australia and northern Eurasia, and fewer species in Africa and southern Asia. One of the reasons for the differential extinction is thought to be the shorter evolutionary history together with humans in the areas that were more affected (since over-hunting is though to be one of the main explanations, see e.g Sandom er al. 2014).
If you only look at currently extant animals, you ignore the Wooly mammoth, the Woolly rhinoceros, Ground sloths and many other large species inhabiting higher latitudes, and this will give a very biased picture when searching for evolutionary reasons for the differences in body size between continents.