(1) Background: The athlete’s heart may develop permanent vessel enlargement. The purpose of our study was to define normal values for coronary artery dimensions of endurance athletes by coronary computed tomography angiography (CTA). (2) Methods: Ninety-eight individuals (56.2 ± 11 years) were included into this retrospective matched case-controlled-study. Endurance athletes had regular training volumes of ≥1 h per unit, ≥3–7 times per week (either cycling, running or mountain-endurance). Athletes were matched for age and gender with sedentary controls using propensity score. Quantitative CTA analysis included coronary vessel dimensions (two diameters and area) of the LM, LAD, CX and RCA for all AHA-16-segments. (3) Results: Proximal LAD area and diameter (p = 0.019); proximal/mid CX (diameter and area; p = 0.026 and p = 0.018/p = 0.008 and p = 0.009); mid RCA diameter and area; and proximal RCA diameter were significantly larger in endurance athletes (p < 0.05). The left main area (p = 0.708) and diameter (p = 0.809) as well as the mid LAD and distal segments were not different. We present the histograms and data for normal values ±1 and ± 2 SD. (4) Conclusions: Endurance athletes have larger proximal LAD, proximal/mid CX and RCA vessel dimensions, while LM and distal segments are similar. Hence, dilated coronary arteries in endurance athletes (“Athlete’s arteries”) have to be distinguished from diffuse ectatic segments developing during Kawasaki disease or multisystemic inflammation syndrome after COVID-19.