Prompt photon production is known as a powerful tool for testing perturbative QCD predictions and also the validity of parton densities in the nucleon and nuclei especially of the gluon. In this work, we have performed a detailed study on this subject focusing on the isolated prompt photon production in $ p $-Pb collisions at forward rapidity at the LHC. The impact of input nuclear modifications obtained from different global analyses by various groups on several quantities has been investigated to estimate the order of magnitude of the difference between their predictions. We have also studied in detail the theoretical uncertainties in the results due to various sources. We found that there is a remarkable difference between the predictions from the nCTEQ15 and other groups in all ranges of photon transverse momentum $ p_\textrm{T}^γ $. Their differences become more explicit in the calculation of the nuclear modification ratio and also the yield asymmetry between the forward and backward rapidities rather than single differential cross sections. We emphasize that future measurements with ALICE will be very useful not only for decreasing the uncertainty of the gluon nuclear modification, but also to accurately determine its central values, especially in the shadowing region.