Even after realignment there is residual movement-related variance present in fMRI time-series, causing loss of sensitivity and, potentially, also specificity. One cause is the differential deformation of the sampling matrix, by field inhomogeneities, at different object positions, i.e., a movement-by-inhomogeneity interaction. This has been addressed previously by using empirical field measurements. In the present paper we suggest a forward model of how data is affected by an inhomogeneous field at different object positions. From this model we derive a method to solve the inverse problem of estimating the field inhomogeneities and their derivatives with respect to object position, directly from the EPI data and estimated realignment parameters. The field is modeled as a linear combination of cosine basis fields, which facilitates a fast way of implementing the necessary matrix operations. Simulations suggest that the solution is tractable and that the fields are estimable given the deformed images and knowledge of the relative positions at which they have been acquired. An experiment on a subject performing voluntary movements in the scanner yielded plausible estimates of the deformation fields and their application to "unwarp" the time series significantly reduced movement-related variance.