Polarization-insensitive modulation, i.e., overcoming the limit of conventional modulators operating under only a single-polarization state, is desirable for high-capacity on-chip optical interconnects. Here, we propose a hybrid graphene-silicon-based polarization-insensitive electro-absorption modulator (EAM) with high-modulation efficiency and ultra-broad bandwidth. The hybrid graphene-silicon waveguide is formed by leveraging multi-deposited and multi-transferred methods to enable light interaction with graphene layers in its intense field distribution region instead of the commonly used weak cladding region, thus resulting in enhanced light–graphene interaction. By optimizing the dimensions of all hybrid graphene-silicon waveguide layers, polarization-insensitive modulation is achieved with a modulation efficiency (ME) of ~1.11 dB/µm for both polarizations (ME discrepancy < 0.006 dB/µm), which outperforms that of previous reports. Based on this excellent modulation performance, we designed a hybrid graphene-silicon-based EAM with a length of only 20 µm. The modulation depth (MD) and insertion loss obtained were higher than 22 dB and lower than 0.23 dB at 1.55 µm, respectively, for both polarizations. Meanwhile, its allowable bandwidth can exceed 300 nm by keeping MD more than 20 dB and MD discrepancy less than 2 dB, simultaneously, and its electrical properties were also analyzed. Therefore, the proposed device can be applied in on-chip optical interconnects.