Deep mutational scanning (DMS) has emerged as a powerful approach for evaluating the effects of mutations on binding or function. Here, we developed a DMS by phage display protocol to define the specificity determinants of short linear motifs (SLiMs) binding to peptide-binding domains. We first designed a benchmarking DMS library to evaluate the performance of the approach on well-known ligands for 11 different peptide-binding domains, including the talin-1 PTB domain, the G3BP1 NTF2 domain, and the MDM2 SWIB domain. Comparison with a set of reference motifs from the eukaryotic linear motif (ELM) database confirmed that the DMS by phage display analysis correctly identifies known motif binding determinants and provides novel insights into specificity determinants, including defining a non-canonical talin-1 PTB binding motif with a putative extended conformation. A second DMS library was designed, aiming to provide information on the binding determinants for 19 SLiM-based interactions between human and SARS-CoV-2 proteins. The analysis confirmed the affinity determining residues of viral peptides binding to host proteins and refined the consensus motifs in human peptides binding to five domains from SARS-CoV-2 proteins, including the non-structural protein (NSP) 9. The DMS analysis further pinpointed mutations that increased the affinity of ligands for NSP3 and NSP9. An affinity-improved cell-permeable NSP9-binding peptide was found to exert stronger antiviral effects than the wild-type peptide. Our study demonstrates that DMS by phage display can efficiently be multiplexed and applied to refine binding determinants and shows how the results can guide peptide-engineering efforts.