Plasmonics offers the opportunity of tailoring the interaction of light with single quantum emitters. However, the strong field localization of plasmons requires spatial fabrication accuracy far beyond what is required for other nanophotonic technologies. Furthermore, this accuracy has to be achieved across different fabrication processes to combine quantum emitters and plasmonics. We demonstrate a solution to this critical problem by controlled positioning of plasmonic nanoantennas with an accuracy of 11 nm next to single self-assembled GaAs semiconductor quantum dots, whose position can be determined with nanometer precision. These dots do not suffer from blinking or bleaching or from random orientation of the transition dipole moment as colloidal nanocrystals do. Our method introduces flexible fabrication of arbitrary nanostructures coupled to single-photon sources in a controllable and scalable fashion.