We report on the photo-thermal activation of dielectric loaded plasmonic switches comprised of gold nanoparticle-doped polymer deposited onto a gold film. The plasmonic switches rely on a multi-mode interferometer design and are fabricated by electron beam lithography applied to a positive resin doped with gold nanoparticles at a volume ratio of 0.52%. A cross-bar switching is obtained at telecom wavelengths by pumping the devices with a visible beam having a frequency within the localized surface plasmon resonance band of the embedded nanoparticles. By comparing the switching performances of doped and undoped devices, we show that for the modest doping level we consider, the power needed to activate the doped switches is reduced by a factor 2.5 compared to undoped devices. The minimization of activation power is attributed to enhanced light-heat conversion and optimized spatial heat generation for doped devices and not to a change of the thermo-optic coefficient of the doped polymer.