So you think just because women are not born with fully formed breasts and men not born with beards that these characteristics are not driven by genes?
I don't think your analogy works. Moreover, I think I remember seeing evidence that sex is determined by environmental influences during early gestation, not genes. In fact we can explain very little with the genetic perspective.