Abstract We investigate general relativistic magnetohydrodynamic simulations to determine the physical origin of the twisty patterns of linear polarization seen in spatially resolved black hole images and explain their morphological dependence on black hole spin. By characterizing the observed emission with a simple analytic ring model, we find that the twisty morphology is determined by the magnetic field structure in the emitting region. Moreover, the dependence of this twisty pattern on spin can be attributed to changes in the magnetic field geometry that occur due to the frame dragging. By studying an analytic ring model, we find that the roles of Doppler boosting and lensing are subdominant. Faraday rotation may cause a systematic shift in the linear polarization pattern, but we find that its impact is subdominant for models with strong magnetic fields and modest ion-to-electron temperature ratios. Models with weaker magnetic fields are much more strongly affected by Faraday rotation and have more complicated emission geometries than can be captured by a ring model. However, these models are currently disfavoured by the recent EHT observations of M87*. Our results suggest that linear polarization maps can provide a probe of the underlying magnetic field structure around a black hole, which may then be usable to indirectly infer black hole spins. The generality of these results should be tested with alternative codes, initial conditions, and plasma physics prescriptions.