Human cytomegalovirus (HCMV) causes a lifelong infection through establishment of latency. Although reactivation from latency can cause life-threatening disease, our molecular understanding of HCMV latency is incomplete. Here we use single cell RNA-seq analysis to characterize latency in monocytes and hematopoietic stem and progenitor cells (HSPCs). In monocytes, we identify host cell surface markers that enable enrichment of latent cells harboring higher viral transcript levels, which can reactivate more efficiently, and are characterized by reduced intrinsic immune response that is important for viral gene expression. Significantly, in latent HSPCs, viral transcripts could be detected only in monocyte progenitors and were also associated with reduced immune-response. Overall, our work indicates that regardless of the developmental stage in which HCMV infects, HCMV drives hematopoietic cells towards a weaker immune-responsive monocyte state and that this anergic-like state is crucial for the virus ability to express its transcripts and to eventually reactivate.