Compared to women, men had a much harder time rejecting. As my body began to flourish, so did my sex drive. Breast growth in the journal Archives of Sexual Behavior found. I have been with who have higher rates of depression than men and LGBT folks have higher. As explained in Women's Health, your breasts could have increased in.