To date, the impact of the tumor microenvironment on the prognosis of patients with classic Hodgkin lymphoma (cHL) during anti-PD-1 therapy has been studied insufficiently. This retrospective study included 61 primary samples of lymph nodes from patients who had relapsed/refractory (r/r) cHL and were treated with nivolumab. Repeated samples were obtained in 15 patients at relapse or disease progression after immunotherapy. Median follow-up was 55 (13–63) months. The best overall response rate and progression-free survival (PFS) were analyzed depending on the expression of CD68, CD163, PD-1, LAG-3, TIM-3, CTLA-4, TIGIT, CD163/c-maf in the tumor microenvironment in primary and sequential biopsies. The combination of CD163/c-maf antibodies was used for the identification of M2 macrophages (M2). A low number of macrophages in primary samples was associated with inferior PFS during nivolumab treatment (for CD163-positive cells p = 0.0086; for CD68-positive cells p = 0.037), while a low number of M2 with higher PFS (p = 0.014). Complete response was associated with a lower level of M2 (p = 0.011). In sequential samples (before and after nivolumab therapy) an increase in PD-1 (p = 0.011) and LAG-3 (p = 0.0045) and a depletion of CD68 (p = 0.057) and CD163 (p = 0.0049)-positive cells were observed. The study expands understanding of the cHL microenvironment structure and dynamics during nivolumab therapy in patients with r/r cHL.