Neutralizing antibodies strongly correlate with protection for COVID-19 vaccines, but the corresponding memory B cells that form to protect against future infection are relatively understudied. Here we examine the effect of prior SARS-CoV-2 infection on the magnitude and phenotype of the memory B cell response to single dose Johnson and Johnson (Ad26.COV2.S) vaccination in South African health care workers. Participants were either naïve to SARS-CoV-2 or had been infected before vaccination. SARS-CoV-2-specific memory B-cells expand in response to Ad26.COV2.S and are maintained for the study duration (84 days) in all individuals. However, prior infection is associated with a greater frequency of these cells, a significant reduction in expression of the germinal center chemokine receptor CXCR5, and increased class switching. These B cell features correlated with neutralization and antibody-dependent cytotoxicity (ADCC) activity, and with the frequency of SARS-CoV-2 specific circulating T follicular helper cells (cTfh). Vaccination-induced effective neutralization of the D614G variant in both infected and naïve participants but boosted neutralizing antibodies against the Beta and Omicron variants only in participants with prior infection. In addition, the SARS-CoV-2 specific CD8+ T cell response correlated with increased memory B cell expression of the lung-homing receptor CXCR3, which was sustained in the previously infected group. Finally, although vaccination achieved equivalent B cell activation regardless of infection history, it was negatively impacted by age. These data show that phenotyping the response to vaccination can provide insight into the impact of prior infection on memory B cell homing, CSM, cTfh, and neutralization activity. These data can provide early signals to inform studies of vaccine boosting, durability, and co-morbidities.