The tumor immune microenvironment (TIME) plays an important role in penile squamous cell carcinoma (peSCC) pathogenesis. Here, the immunophenotype of the TIME in peSCC was determined by integrating the expression patterns of immune checkpoints (programmed cell death-1 (PD-1)/programmed cell death ligand-1 (PD-L1), cytotoxic T lymphocyte antigen 4 (CTLA-4), and Siglec-15) and the components of tumor-infiltrating lymphocytes, including CD8+ or Granzyme B+ T cells, FOXP3+ regulatory T cells, and CD68+ or CD206+ macrophages, in 178 patients. A high density of Granzyme B, FOXP3, CD68, CD206, PD-1, and CTLA-4 was associated with better disease-specific survival (DSS). The patients with diffuse PD-L1 tumor cell expression had worse prognoses than those with marginal or negative PD-L1 expression. Four immunophenotypes were identified by unsupervised clustering analysis, based on certain immune markers, which were associated with DSS and lymph node metastasis (LNM) in peSCC. There was no significant relationship between the immunophenotypes and high-risk human papillomavirus (hrHPV) infection. However, the hrHPV–positive peSCC exhibited a higher density of stromal Granzyme B and intratumoral PD-1 than the hrHPV–negative tumors (p = 0.049 and 0.002, respectively). In conclusion, the immunophenotypes of peSCC were of great value in predicting LNM and prognosis, and may provide support for clinical stratification management and immunotherapy intervention.