High-quality single crystals with nominal composition M0.05Fe0.95Te0.8Se0.2 (M = Cr, Mn, Co, Ni, Cu, and Zn) have been grown, through which the doping effect on magnetism and superconductivity is studied. Elementary analysis reveals that Cu, Co, and Ni, with smaller ionic radii for valence state 2+, can substitute effectively for Fe with doping levels near 5%. In contrast, the solid solution of Cr, Mn, and Zn in the host system is low. Magnetic and electronic investigations show that the substitution of Co, Ni, or Cu for Fe leads to the formation of spin-glass state and suppression of superconductivity. The superconductivity is partly suppressed by Co doping, while completely destroyed by Ni and Cu doping. Compared with Cu- and Ni-doped samples, the Co-doped sample has the smallest lattice constant, indicating that the superconductivity might be also modulated by the changes of microstructure.