The pan-immune-inflammation value reflects the systemic inflammatory response, and tumor-infiltrating lymphocytes indicate a local immune response in rectal cancer. However, the association between systemic inflammatory response, as indicated by the pan-immune-inflammation value, and local immune responses in rectal cancer remains unclear. This study analyzed 915 treatment-naïve rectal cancer patients from the Peking Union Medical College Hospital and PLA General Hospital (PLAGH) cohorts who underwent radical surgery to investigate the relationship between the pan-immune-inflammation value and immune responses. Lower pan-immune-inflammation value was significantly associated with improved disease-free survival and cancer-specific survival. Multivariate Cox regression models identified the pan-immune-inflammation value as an independent prognostic factor. In the PLAGH cohort, patients with low pan-immune-inflammation values had higher immune cell levels, activated immune pathways, and increased expression of immune checkpoint genes according to RNA sequencing. Hematoxylin and eosin staining and immunohistochemical analysis revealed that lower pan-immune-inflammation value was associated with higher tumor-infiltrating lymphocyte density, more mature tertiary lymphoid structures, increased CD8+ T cells, and elevated human lymphocyte antigen class I expression. Conversely, patients with high pan-immune-inflammation values exhibited pathways linked to tumor progression, such as angiogenesis, epithelial-mesenchymal transition, hypoxia, KRAS signaling, and TGF-ß signaling. Among patients receiving anti-PD-1 therapy, responders had low pre- and post-treatment pan-immune-inflammation values. The pan-immune-inflammation value is a reliable marker associated with distinct immune microenvironment characteristics and can effectively predict disease-free survival, cancer-specific survival, and response to immunotherapy.