We demonstrate the strategies and principles for the performance improvement of layered semiconductor based photodetectors using multilayer indium selenide (InSe) as the model materials. It is discovered that multiple reflection interference at the interfaces in phototransistor device leads to thickness-dependent photo-response, which provide a guideline to improve the performance of layered semiconductor based phototransistors. The responsivity and detectivity of InSe nanosheets phototransistor can be adjustable using applied gate voltage. Our InSe nanosheets phototransistor exhibits ultrahigh responsivity and detectivity. An ultrahigh external photo-responsivity of ~104 AW-1 can be achieved during broad spectra ranging from UV to near infrared wavelength from our InSe nanosheets photodetectors. The detectivity of multilayer InSe devices is ~ 1012 ~ 1013Jones, which surpasses that of currently-exploited InGaAs photodetectors (1011 ~ 1012 Jones). This research shows that multilayer InSe nanosheets are promising materials for high performance photodetectors.