Introduction: Immunosenescence and inflammaging have been implicated in the pathophysiology of frailty. Torquetenovirus (TTV), a single-stranded DNA anellovirus, the major component of the human blood virome, shows an increased replication rate with advancing age. An elevated TTV viremia has been associated with an impaired immune function and an increased risk of mortality in the older population. The objective of this study was to analyze the relation between TTV viremia, physical frailty, and cognitive impairment. Methods: TTV viremia was measured in 1,131 nonfrail, 45 physically frail, and 113 cognitively impaired older adults recruited in the MARK-AGE study (overall mean age 64.7 ± 5.9 years), and then the results were checked in two other independent cohorts from Spain and Portugal, including 126 frail, 252 prefrail, and 141 nonfrail individuals (overall mean age: 77.5 ± 8.3 years). Results: TTV viremia ≥4log was associated with physical frailty (OR: 4.69; 95% CI: 2.06–10.67, p < 0.0001) and cognitive impairment (OR: 3.49, 95% CI: 2.14–5.69, p < 0.0001) in the MARK-AGE population. The association between TTV DNA load and frailty status was confirmed in the Spanish cohort, while a slight association with cognitive impairment was observed (OR: 1.33; 95% CI: 1.000–1.773), only in the unadjusted model. No association between TTV load and frailty or cognitive impairment was found in the Portuguese sample, although a negative association between TTV viremia and MMSE score was observed in Spanish and Portuguese females. Conclusions: These findings demonstrate an association between TTV viremia and physical frailty, while the association with cognitive impairment was observed only in the younger population from the MARK-AGE study. Further research is necessary to clarify TTV’s clinical relevance in the onset and progression of frailty and cognitive decline in older individuals.