We present the first Bayesian method for tomographic decomposition of the plane-of-sky orientation of the magnetic field with the use of stellar polarimetry and distance. This standalone tomographic inversion method presents an important step forward in reconstructing the magnetized interstellar medium (ISM) in three dimensions within dusty regions. We develop a model in which the polarization signal from the magnetized and dusty ISM is described by thin layers at various distances, a working assumption which should be satisfied in small-angular circular apertures. Our modeling makes it possible to infer the mean polarization (amplitude and orientation) induced by individual dusty clouds and to account for the turbulence-induced scatter in a generic way. We present a likelihood function that explicitly accounts for uncertainties in polarization and parallax. We develop a framework for reconstructing the magnetized ISM through the maximization of the log-likelihood using a nested sampling method. We test our Bayesian inversion method on mock data, representative of the high Galactic latitude sky, taking into account realistic uncertainties from Gaia and as expected for the optical polarization survey PASIPHAE according to the currently planned observing strategy. We demonstrate that our method is effective at recovering the cloud properties as soon as the polarization induced by a cloud to its background stars is higher than ~0.1% for the adopted survey exposure time and level of systematic uncertainty. The larger the induced polarization is, the better the method’s performance, and the lower the number of required stars. Our method makes it possible to recover not only the mean polarization properties but also to characterize the intrinsic scatter, thus creating new ways to characterize ISM turbulence and the magnetic field strength. Finally, we apply our method to an existing data set of starlight polarization with known line-of-sight decomposition, demonstrating agreement with previous results and an improved quantification of uncertainties in cloud properties.