We study the temperature dependence as well as anisotropy of optical conductivity ($σ_1$) in the pseudocubic single crystal Pr$_{0.5}$Ca$_{1.5}$MnO$_{4}$ using spectrocopic ellipsometry. Three transition temperatures are observed and can be linked to charge-orbital ($T_{\rm CO/OO}$ $∼$ 320 K), two-dimensional-antiferromagnetic (2D-AFM) ($∼$ 200 K), and three-dimensional AFM ($T_{\rm N} ∼$ 125 K) orderings. Below $T_{\rm CO/OO}$, $σ_1$ shows a charge ordering peak ($∼$0.8 eV) with a significant blue shift as the temperature decreases. Calculations based on a model that incorporates a static Jahn-Teller distortion and assumes the existence of a local charge imbalance between two different sublattices support this assignment and explain the blue shift. This view is further supported by the partial spectral weight analysis showing the onset of optical anisotropy at $T_{\rm CO/OO}$ in the charge-ordering region ($0.5-2.5$ eV). Interestingly, in the charge-transfer region ($2.5-4$ eV), the spectral weight shows anomalies around the $T_{\rm 2D-AFM}$ that we attribute to the role of oxygen-$p$ orbitals in stabilizing the CE-type magnetic ordering. Our result shows the importance of spin, charge, and lattice degrees of freedom in this layered manganite. ; Comment: 6 pages, 3 figures