The main job of all individuals, in all species, plant and animal, is to pass along their DNA.
In humanity, more females pass along their DNA than males. In all human history, there are twice as many women in the DNA record than there are men.
So half the men are out. Some men don't care for sex with women. Some men don't pass along their DNA on a voluntary basis. (But historically, it hasn't been so much of a choice; arguably, we didn't even understand the exact relationship between sex and pregnancy until recently in human history.) Some men don't care for sex at all.
But the biggest reason men don't get to pass along their DNA is that they are rejected by the women. For whatever reason, they don't do the mating dance well, and are not selected.
Today, sometimes, we can see why those men are rejected. Prehistorically, we're not so sure. Here is the most amazing graph on this topic, ever.
On the left, the number of men reproducing their DNA. On the right, the women.
8,000 Years Ago, 17 Women Reproduced For Every One Man
It turns out that, after the advent of agriculture, human behavior suddenly and radically changed... on every continent. Rather suddenly, the women became highly selective. Suddenly it was really rare for a man to qualify.
(It's like middle school all over again...)
If you read the article you will find that they have theories about why this is so, but nobody really knows!! All they know is that it's the first marker that human culture affected evolution.