Off-axis holography has successfully revealed the built-in potential in highly doped pn-junctions, making holographic phase retrieval in transmission electron microscopy (TEM) a prospective tool for imaging the electrostatic potential in a semiconductor sample or device. The effect of electron-hole pair generation during electron illumination and the presence of defect rich amorphized sample surfaces in a TEM sample on the electrostatic potential is investigated for the test case of an electrically active grain boundary and a pn-junction in silicon by numerical simulation. In the case of the grain boundary, the pair generation in the electron beam leads to significantly increased recombination currents into trap state defects. As a result, the trapped charge in the defects is decreased and the potential drop around the electrically active defect is considerably reduced, for electron-hole pair generation rates typically present in a TEM experiment even below the detection limit of holography. In the case of pn-junctions distorted potential maps are predicted for dopant densities smaller than 10(17) cm(-3).