Band gap tuning of hybrid metal-halide perovskites by halide substitution holds promise for tailored light absorption in tandem solar cells and emission in light-emitting diodes. However, the impact of halide substitution on the crystal structure and the fundamental mechanism of photo-induced halide segregation remain open questions. Here, using a combination of temperature-dependent X-ray diffraction and calorimetry measurements, we report the emergence of a disorder- and frustration-driven orientational glass for a wide range of compositions in CH3NH3Pb(Cl x Br1-x )3. Using temperature-dependent photoluminescence measurements, we find a correlation between halide segregation under illumination and local strains from the orientational glass. We observe no glassy behavior in CsPb(Cl x Br1-x )3, highlighting the importance of the A-site cation for the structure and optoelectronic properties. Using first-principles calculations, we identify the local preferential alignment of the organic cations as the glass formation mechanism. Our findings rationalize the superior photostability of mixed-cation metal-halide perovskites and provide guidelines for further stabilization strategies.