Although animals can estimate numerical quantities, true counting and arithmetic abilities are unique to humans and are inextricably linked to symbolic competence. However, our unprecedented numerical skills are deeply rooted in our neuronal heritage as primates and vertebrates. In this article, I argue that numerical competence in humans is the result of three neural constraints. First, I propose that the neuronal mechanisms of quantity estimation are part of our evolutionary heritage and can be witnessed across primate and vertebrate phylogeny. Second, I suggest that a basic understanding of number, what numerical quantity means, is innately wired into the brain and gives rise to an intuitive number sense, or number instinct. Third and finally, I argue that symbolic counting and arithmetic in humans is rooted in an evolutionarily and ontogenetically primeval neural system for non-symbolic number representations. These three neural constraints jointly determine the basic processing of number concepts in the human mind.