We present a cryogenically compatible method for permanently connecting and coupling a single mode fiber to a single mode of a micropillar cavity with embedded quantum dots (QDs). Efficient coupling of up to 40% was measured which requires a 300 nm positioning accuracy that remains preserved during the fiber attachment procedure and during cool-down to 4 K. Fiber coupling, as opposed to conventional free space coupling, makes it possible to connect many such QD-cavity systems within the same cryostat which can interact through an external optical network, facilitating the implementation of hybrid photon/confined-electron schemes for quantum communication and information processing. (C) 2010 American Institute of Physics. [doi: 10.1063/1.3493187]