Murid γ-herpesvirus-4 (MuHV-4) promotes polyclonal B cell activation and establishes latency in memory B cells via unclear mechanisms. We aimed at exploring whether B cell receptor specificity plays a role in B cell susceptibility to viral latency and how this is related to B cell activation. We first observed that MuHV-4-specific B cells represent a minority of the latent population, and to better understand the influence of the virus on non-MuHV-4 specific B cells we used the SWHEL mouse model, which produce hen egg lysozyme (HEL)-specific B cells. By tracking HEL+ and HEL− B cells, we showed that in vivo latency was restricted to HEL− B cells while the two populations were equally sensitive to the virus in vitro. Moreover, MuHV-4 induced two waves of B cell activation. While the first wave was characterized by a general B cell activation, as shown by HEL+ and HEL− B cells expansion and upregulation of CD69 expression, the second wave was restricted to the HEL− population, which acquired germinal center (GC) and plasma cell phenotypes. Antigenic stimulation of HEL+ B cells led to the development of HEL+ GC B cells where latent infection remained undetectable, indicating that MuHV-4 does not benefit from acute B cell responses to establish latency in non-virus specific B cells but relies on other mechanisms of the humoral response. These data support a model in which the establishment of latency in B cells by γ-herpesviruses is not stochastic in terms of BCR specificity and is tightly linked to the formation of GCs.