Abstract Background Tissue-resident memory T (Trm) cells are associated with cytotoxicity not only in viral infection and autoimmune disease pathologies but also in many cancers. Tumour-infiltrating CD103⁺ Trm cells predominantly comprise CD8 T cells that express cytotoxic activation and immune checkpoint molecules called exhausted markers. This study aimed to investigate the role of Trm in colorectal cancer (CRC) and characterise the cancer-specific Trm. Methods Immunochemical staining with anti-CD8 and anti-CD103 antibodies for resected CRC tissues was used to identify the tumour-infiltrating Trm cells. The Kaplan–Meier estimator was used to evaluate the prognostic significance. Cells immune to CRC were targeted for single-cell RNA-seq analysis to characterise cancer-specific Trm cells in CRC. Results The number of CD103⁺/CD8⁺ tumour-infiltrating lymphocytes (TILs) was a favourable prognostic and predictive factor of the overall survival and recurrence-free survival in patients with CRC. Single-cell RNA-seq analysis of 17,257 CRC-infiltrating immune cells revealed a more increased zinc finger protein 683 (ZNF683) expression in cancer Trm cells than in noncancer Trm cells and in high-infiltrating Trm cells than low-infiltrating Trm in cancer, with an upregulated T-cell receptor (TCR)- and interferon-γ (IFN-γ) signalling-related gene expression in ZNF683⁺ Trm cells. Conclusions The number of CD103⁺/CD8⁺ TILs is a prognostic predictive factor in CRC. In addition, we identified the ZNF683 expression as one of the candidate markers of cancer-specific Trm cells. IFN-γ and TCR signalling and ZNF683 expression are involved in Trm cell activation in tumours and are promising targets for cancer immunity regulation.