Sex differences in immunity are well-documented, though mechanisms underpinning these differences remain ill-defined. Here, in a human-only ex vivo study, we demonstrate that postpubertal cisgender females have higher levels of CD19+CD27+IgD- class-switched memory B cells compared with age-matched cisgender males. This increase is only observed after puberty and before menopause, suggesting a strong influence for sex hormones. Accordingly, B cells express high levels of estrogen receptor 2 (ESR2), and class-switch-regulating genes are enriched for ESR2-binding sites. In a gender-diverse cohort, blockade of natal estrogen in transgender males (XX karyotype) reduced class-switched memory B cell frequency, while gender-affirming estradiol treatment in transgender females (XY karyotype) did not increase these levels. In postmenopausal cis-females, class-switched memory B cells were increased in those taking hormone replacement therapy (HRT) compared with those who were not. These data demonstrate that sex hormones and chromosomes work in tandem to impact immune responses, with estrogen only influencing the frequency of class-switched memory B cells in individuals with an XX chromosomal background.