BACKGROUND: Human leukocyte antigen class II (HLA-II) genes are highly polymorphic affecting the specificity of human antibody responses, as presentation of processed antigen peptides by HLA-II on B cells is essential for T helper cell dependent affinity maturation and class switching. The combination of high-throughput immunoassays and genome-wide association studies has recently revealed strong associations between HLA-II variants and antibody responses against specific antigens. However, factors underlying these associations remain incompletely understood. METHODS: Here, we have leveraged paired data sets of SNP arrays and functional antibody epitope repertoires against 344,000 peptide antigens in 1940 individuals to mine for key determinants linking genetics and antibody specificity. RESULTS: We show that secreted proteins and antigens presented in small modules (i.e., viruses) are significantly more frequently associated with HLA-II alleles, than membrane bound or intracellular proteins. This data suggests a model in which antibody responses against separate antigen units composed of single or few proteins dominate HLA-II associations. In contrast, the presence of manifold intracellular or membrane proteins (peptides of which could be bound by different HLA-II alleles) on bacterial cells dilutes potential associations to antibody specificities. CONCLUSIONS: Hence, genetic associations to antibody specificities are shaped by antigen intrinsic properties. Given the prominent role of HLA-II alleles in infection, autoimmune diseases, allergies, and cancer, our work provides a theoretical framework to study antigen/HLA-II risk factors in these disease settings and will fuel the design of improved immunogenetics screens.