Prolonged patient survival after surgical resection, is associated with a higher cytotoxic and memory T cell density within colorectal cancers (CRC). High endothelial venules (HEVs) are specialized blood vessels present in secondary lymphoid organs (SLO) that allow ingress of naïve and central memory T cells from the blood. It has been proposed that HEVs in tumors might serve as a similar route of entry for lymphocytes into the tumor and result in an improved prognosis. The present study aimed to characterize HEVs and their microenvironment in resected tumors from colorectal cancer patients (n = 62). We observed HEVs in association with lymphoid aggregates in 49 out of 62 patients. However, these HEV(+) lymphoid aggregates were largely at the invasive margin of the tumor and although there was an association with lymphocytes and HEVs at the invasive margin (p = 0.002) there was only a very weak association with tumor infiltrating lymphocytes. Indeed, lymphoid aggregates were associated with more advanced disease (Dukes' stage C) and did not indicate a favorable prognosis.