Unlike sequencing-based methods, which require cell lysis, optical pooled genetic screens enable investigation of spatial phenotypes, including cell morphology, protein subcellular localization, cell-cell interactions and tissue organization, in response to targeted CRISPR perturbations. Here we report a multimodal optical pooled CRISPR screening method, which we call CRISPRmap. CRISPRmap combines in situ CRISPR guide-identifying barcode readout with multiplexed immunofluorescence and RNA detection. Barcodes are detected and read out through combinatorial hybridization of DNA oligos, enhancing barcode detection efficiency. CRISPRmap enables in situ barcode readout in cell types and contexts that were elusive to conventional optical pooled screening, including cultured primary cells, embryonic stem cells, induced pluripotent stem cells, derived neurons and in vivo cells in a tissue context. We conducted a screen in a breast cancer cell line of the effects of DNA damage repair gene variants on cellular responses to commonly used cancer therapies, and we show that optical phenotyping pinpoints likely pathogenic patient-derived mutations that were previously classified as variants of unknown clinical significance.