The effect of P2O5 and/or Al2O3 addition in Er-doped borosilicate glasses on the physical, thermal, optical, and luminescence properties is investigated. The changes in these glass properties are related to the glass structure modifications induced by the addition of P2O5 and/or Al2O3, which were probed by FTIR, 11B MAS NMR and X-ray photoelectron spectroscopies. Variations of the polymerization degree of the silicate tetrahedra and modifications in the [3]B/[4]B ratio are explained by a charge compensation mechanism due to the formation of AlO4, PO4 groups and the formation of AlOP linkages in the glass network. From the absorption and luminescence properties of the Er3+ ions at 980 nm and 1530 nm, declustering is suspected for the highest P2O5 concentrations while for the highest Al2O3 concentrations no declustering is observed.