In this work, high-responsivity graphene photodetectors operating in the middle-wavelength infrared (MWIR) spectral band were fabricated by taking advantage of the photo-gating effect. Graphene-based field effect transistors were fabricated on indium antimonide (InSb) substrates. The InSb generated photo-carriers in response to incident IR light modulated the graphene channel gate voltage and induced a large photocurrent. These graphene-based photodetectors exhibited a clear photoresponse during irradiation with 4.6 μm MWIR laser light and an ultrahigh responsivity of 33.8 A/W was achieved at 50 K due to the photo-gating effect. These devices were found to maintain an MWIR photoresponse up to 150 K. Our graphene-based photodetector design is expected to contribute to the development of high-performance MWIR image sensors.