Abstract Single-cell sequencing opens a new era for the investigation of tumor immune microenvironments (TIME). However, at single-cell resolution, a pan-cancer analysis that addresses the identity and diversity of TIMEs is lacking. Here, we first built a pan-cancer single-cell reference of TIMEs with refined subcell types and recognized new cell type–specific transcription factors. We then presented a pan-cancer view of the common features of the TIME and compared the variation of each immune cell type across patients and tumor types in the aspects of abundance, cell states, and cell communications. We found that the abundance and the cell states of dysfunctional T cells were most variable, whereas those of regulatory T cells were relatively stable. A subset of tumor-associated macrophages (TAM), PLTP+C1QC+ TAMs, may regulate the abundance of dysfunctional T cells through cytokine/chemokine signaling. The ligand–receptor communication network of TIMEs was tumor-type specific and dominated by the tumor-enriched immune cells. We additionally developed the single-cell TIME (scTIME) portal (http://scTIME.sklehabc.com) with the scTIME-specific analysis modules and a unified cell annotation. In addition to the immune cell compositions and correlation analysis using refined cell type classifications, the portal also provides cell–cell interaction and cell type–specific gene signature analysis. Our single-cell pan-cancer analysis and scTIME portal will provide more insights into the features of TIMEs, as well as the molecular and cellular mechanisms underlying immunotherapies.