Background. The tumor immune microenvironment exerts a pivotal influence in tumor initiation and progression. The aim of this study was to analyze the immune context of sporadic and familial adenomatous polyposis (FAP) lesions along the colorectal adenoma–carcinoma sequence (ACS). Methods. We analyzed immune cell counts (CD3+, CD4+, CD8+, Foxp3+, and CD57+), tumor mutation burden (TMB), MHC-I expression and PD-L1 expression of 59 FAP and 74 sporadic colorectal lesions, encompassing adenomas with low-grade dysplasia (LGD) (30 FAP; 30 sporadic), adenomas with high-grade dysplasia (22 FAP; 30 sporadic), and invasive adenocarcinomas (7 FAP; 14 sporadic). Results. The sporadic colorectal ACS was characterized by (1) a stepwise decrease in immune cell counts, (2) an increase in TMB and MHC-I expression, and (3) a lower PD-L1 expression. In FAP lesions, we observed the same patterns, except for an increase in TMB along the ACS. FAP LGD lesions harbored lower Foxp3+ T cell counts than sporadic LGD lesions. A decrease in PD-L1 expression occurred earlier in FAP lesions compared to sporadic ones. Conclusions. The colorectal ACS is characterized by a progressive loss of adaptive immune infiltrate and by the establishment of a progressively immune cold microenvironment. These changes do not appear to be related with the loss of immunogenicity of tumor cells, or to the onset of an immunosuppressive tumor microenvironment.