High precision cosmological distance measurements towards individual objects such as time delay gravitational lenses or type Ia supernovae are affected by weak lensing perturbations by galaxies and groups along the line of sight. In time delay gravitational lenses, "external convergence," kappa, can dominate the uncertainty in the inferred distances and hence cosmological parameters. In this paper we attempt to reconstruct kappa, due to line of sight structure, using a simple halo model. We use mock catalogues from the Millennium Simulation, and calibrate and compare our reconstructed P(kappa) to ray-traced kappa "truth" values; taking into account realistic observational uncertainties. We find that the reconstruction of kappa provides an improvement in precision of ~50% over galaxy number counts. We find that the lowest-kappa lines of sight have the best constrained P(kappa). In anticipation of large future samples of lenses, we find that selecting the third of the systems with the highest precision kappa estimates gives a sample of unbiased time delay distance measurements with just ~1% uncertainty due to line of sight external convergence effects. Photometric data are sufficient to pre-select the best-constrained lines of sight, and can be done before investment in light-curve monitoring. Conversely, we show that selecting lines of sight with high external shear could, with the reconstruction model presented, induce biases of up to 1% in time delay distance. We find that a major potential source of systematic error is uncertainty in the high mass end of the stellar mass-halo mass relation; this could introduce ~2% biases on the time-delay distance if completely ignored. We suggest areas for the improvement of this general analysis framework (including more sophisticated treatment of high mass structures) that should allow yet more accurate cosmological inferences to be made. ; Comment: 15 pages. Accepted for publication in MNRAS. Line of sight reconstruction code available at https://github.com/drphilmarshall/Pangloss