Background and Aims: Perihilar cholangiocarcinoma (pCCA) is a hepatobiliary malignancy, with a dismal prognosis. Nerve fiber density (NFD)—a novel prognostic biomarker—describes the density of small nerve fibers without cancer invasion and is categorized into high numbers and low numbers of small nerve fibers (high vs low NFD). NFD is different than perineural invasion (PNI), defined as nerve fiber trunks invaded by cancer cells. Here, we aim to explore differences in immune cell populations and survival between high and low NFD patients. Approach and Results: We applied multiplex immunofluorescence (mIF) on 47 pCCA patients and investigated immune cell composition in the tumor microenvironment (TME) of high and low NFD. Group comparison and oncological outcome analysis was performed. CD8+PD-1 expression was higher in the high NFD than in the low NFD group (12.24 × 10−6 vs. 1.38 × 10−6 positive cells by overall cell count, p = 0.017). High CD8+PD-1 expression was further identified as an independent predictor of overall (OS; Hazard ratio (HR) = 0.41; p = 0.031) and recurrence-free survival (RFS; HR = 0.40; p = 0.039). Correspondingly, the median OS was 83 months (95% confidence interval (CI): 18–48) in patients with high CD8+PD-1+ expression compared to 19 months (95% CI: 5–93) in patients with low CD8+PD-1+ expression (p = 0.018 log rank). Furthermore, RFS was significantly lower in patients with low CD8+PD-1+ expression (14 months (95% CI: 6–22)) compared to patients with high CD8+PD-1+ expression (83 months (95% CI: 17–149), p = 0.018 log rank). Conclusions: PD-1+ T-cells correlate with high NFD as a prognostic biomarker and predict good survival; the biological pathway needs to be investigated.