We investigate electron transport through clean open quantum dots (quantum billiards). We present a semiclassical theory that allows to accurately reproduce quantum transport calculations. Quantitative agreement is reached for individual energy and magnetic field dependent elements of the scattering matrix. Two key ingredients are essential: (i) inclusion of pseudo-paths which have the topology of linked classical paths resulting from diffraction in addition to classical paths and (ii) a high-level approximation to diffractive scattering. Within this framework of the pseudo-path semiclassical approximation (PSCA), typical shortcomings of semiclassical theories such as violation of the anti-correlation between reflection and transmission and the overestimation of conductance fluctuations are overcome. Beyond its predictive capabilities the PSCA provides deeper insights into the quantum-to-classical crossover. ; Comment: 20 pages, 19 figures