(1) Background: Tooth loss is an important component of the global burden of oral disease, greatly reducing the quality of life of those affected. Tooth loss can also affect diet and subsequent incidences of lifestyle diseases, such as hypertension and metabolic syndromes. This study aimed to evaluate the oral health-related quality of life (OHRQoL) score using the oral impacts on daily performance (OIDP) index in relation to tooth loss patterns among adults. (2) Methods: From 2014 to 2016, a cross-sectional study was conducted on adults living in Bellville South, Cape Town, South Africa. The OHRQoL measure was used to evaluate the impact of tooth loss. (3) Results: A total of 1615 participants were included, and 143 (8.85%) had at least one impact (OIDP > 0). Males were less likely to experience at least one impact compared to the females, OR=0.6, 95% C.I.: 0.385 to 0.942, p = 0.026. Those participants who did not seek dental help due to financial constraints were 6.54 (4.49 to 9.54) times more likely to experience at least one impact, p < 0.001. (4) Conclusions: Tooth loss did not impact the OHRQoL of these subjects. There was no difference in the reported odds for participants experiencing at least one oral impact with the loss of their four anterior teeth, the loss of their posterior occlusal pairs, or the loss of their other teeth.