Coherent control based on a feedback-controlled self-learning loop was applied to enhance the ratio of two-photon (2P) fluorescence from a fluorescent label enhanced green fluorescence protein and three-photon (3P) fluorescence from the essential amino acid L-Tryptophan. The two biosamples were mixed in a buffer solution contained in a quartz cuvette and exposed to near-infrared laser pulses from a femtosecond (fs) oscillator. However, the enhancement of the 2P/3P fluorescence ratio was always accompanied by a significant loss of the valuable 2P fluorescence. To achieve a trade-off between the 2P/3P fluorescence ratio and the 2P fluorescence intensity, we then engineered the cost function in the self-learning algorithm. The optimal pulse shape obtained by use of the engineered cost function could be useful for 2P fluorescence imaging of living cells with reduced phototoxicity, because DNA and protein can be directly damaged by 3P absorption of fs laser according to an excitation band ~270nm.