The ongoing and future experiments will measure the B-mode from different sky coverage and frequency bands, with the potential to reveal non-trivial features in polarization map. In this work we study the TT, TB, EB and BB correlations associated with the B-mode polarization of CMB map in models of charged anisotropic inflation. The model contains a chaotic-type large field complex inflaton which is charged under the $U(1)$ gauge field. We calculate the statistical anisotropies generated in the power spectra of the curvature perturbation, the tensor perturbation and their cross-correlation. It is shown that the asymmetry in tensor power spectrum is a very sensitive probe of the gauge coupling. While the level of statistical anisotropy in temperature power spectrum can be small and satisfy the observational bounds, the interactions from the gauge coupling can induce large directional dependence in tensor modes. This will leave interesting anisotropic fingerprints in various correlations involving the B-mode polarization such as the TB cross-correlation which may be detected in upcoming Planck polarization data. In addition, the TT correlation receives an anisotropic contribution from the tensor sector which naturally decays after $l \gtrsim 100$. We expect that the mechanism of using tensor sector to induce asymmetry at low $l$ to be generic which can also be applied to address other low $l$ CMB anomalies. ; Comment: 27 pages, 10 figures; v2: references added; minor improvements in discussion; v3: version published on JCAP