when the earth was hotter, animals were able to allocate calories towards growth instead of body heat
there were giant animals everywhere, and not a human in sight
You may be correct, at least partially, but the evidence suggests otherwise. Take the case of the hyena, a predator presently restricted to Arabia and sub-Sahara Africa. Wikipedia says this about ice-age hyenas, creatures very much large than present day hyenas:
The cave hyena (Crocuta crocuta spelaea), also known as the Ice Age spotted hyena, was a paleosubspecies of spotted hyena which ranged from the Iberian Peninsula to eastern Siberia. It is one of the best known mammals of the Ice Age and is well represented in many European bone caves. The cave hyena was a highly specialised animal, with its progressive and regressive features being more developed than in its modern African relative. It preyed on large mammals (primarily wild horses, steppe bison and woolly rhinoceros), and was responsible for the accumulation of hundreds of large Pleistocene mammal bones in areas including horizontal caves, sinkholes, mud pits and muddy areas along rivers.
The cause of the cave hyena's extinction is not fully understood, though it could have been due to a combination of factors, including climate change and competition with other predators.
Beavers during the Ice Age were vastly larger (250 pounds) than present day beavers. Ancient New Zealand penguins are estimated at 340 pounds. Ice Age Mammoths are bigger than present day elephants. The dominant creatures of New Zealand and Madagascar were huge birds (called Moa and Elephant Birds) but they're gone too, mostly because of human interaction. Then there are the slow moving Saber Tooth Tigers and Ground Sloths who perished because of humans. See the Slate article