Every woman wants to have firm, perky breasts throughout her life, but saggy breasts usually start happening once a woman reaches her late thirties, although it can occur even earlier. In addition to age, some other factors that can cause saggy breasts include pregnancy, breastfeeding, menopause, rapid weight loss, and wearing a poorly fitting bra.