The concept of time reveals the human's capacity for abstract thought. But time is not the only abstract domain, nor the only one we understand through metaphor. Spatial metaphors are very common, structuring how we think about kinship, politics, and power. ("She has the upper hand.")
The particular metaphors we use, however, rely on our culture, not biological evolution.