Abstract
In proteins, the optimal residue at any position is determined by its structural, evolutionary, and functional contexts-much like how a word may be inferred from its context in language. We trained masked label prediction models to learn representations of amino acid residues in different contexts. We focus questions on evolution and structural flexibility and whether and how contextual encodings derived through pretraining and fine-tuning may improve representations for specialized contexts. Sequences sampled from our learned representations fold into template structure and reflect sequence variations seen in related proteins. For flexible proteins, sampled sequences traverse the full conformational space of the native sequence, suggesting that plasticity is encoded in the template structure. For protein-protein interfaces, generated sequences replicate wild-type binding energies across diverse interfaces and binding strengths in silico. For the antibody-antigen interface, fine-tuning recapitulate conserved sequence patterns, while pretraining on general contexts improves sequence recovery for the hypervariable H3 loop. A record of this paper's transparent peer review process is included in the supplemental information.