A unifying model based on enhanced elctron-hole recombination rate generated by surface plasmon (SP) waves of Au nanoparticles (NPs), electrons transferred from the CdSe quantum dots (QDs) to the Au NPs, as well as the photoexcitation wavelength is proposed to explain the observed optical enhancement and quenching from Au and CdSe nanocomposites. In our system, the photoluminescence-enhancement ratio can be manipulated to ~130, the largest value ever reported. Our experimental results clarify the ambiguity in controlling the light emission enhancement of semiconductor nanocrystals which are coupled with the SP waves of metal NPs.