Extracellular vesicles (EVs) comprise a broad range of secreted cell-derived membrane vesicles. Beyond their more well-characterized role in cell communication, in recent years, EVs have also been shown to play important roles during infection. Viruses can hijack the biogenesis of exosomes (which are small EVs) to promote viral spreading. Additionally, these exosomes are also important mediators in inflammation and immune responses during both bacterial and viral infections. This review summarizes these mechanisms while also describing the impact of bacterial EVs in regulating immune responses. Finally, the review also focuses on the potential and challenges of using EVs, in particular, to tackle infectious diseases.