Context. The low-metallicity interstellar medium (ISM) is profoundly different from that of normal systems, being clumpy with low dust abundance and little CO-traced molecular gas. Yet many dwarf galaxies in the nearby universe are actively forming stars. As the complex ISM phases are spatially mixed with each other, detailed modeling is needed to understand the gas emission and subsequent composition and structure of the ISM. Aims. Our goal is to describe the multi-phase ISM of the infrared bright low-metallicity galaxy Haro 11, dissecting the photoionised and photodissociated gas components. Methods. We present observations of the mid-infrared and far-infrared fine-structure cooling lines obtained with the Spitzer/IRS and Herschel/PACS spectrometers. We use the spectral synthesis code Cloudy to methodically model the ionised and neutral gas from which these lines originate. Results. We find that the mid-and far-infrared lines account for similar to 1% of the total infrared luminosity L-TIR, acting as major coolants of the gas. Haro 11 is undergoing a phase of intense star formation, as traced by the brightest line, [O III] 88 mu m, with L-[O III]/L-TIR similar to 0.3%, and high ratios of [Ne III]/[Ne II] and [S IV]/[S III]. Due to their different origins, the observed lines require a multi-phase modeling comprising: a compact HII region, dense fragmented photodissociation regions (PDRs), a diffuse extended low-ionisation/neutral gas which has a volume filling factor of at least 90%, and porous warm dust in proximity to the stellar source. For a more realistic picture of the ISM of Haro 11 we would need to model the clumpy source and gas structures. We combine these 4 model components to explain the emission of 17 spectral lines, investigate the global energy balance of the galaxy through its spectral energy distribution, and establish a phase mass inventory. While the ionic emission lines of Haro 11 essentially originate from the dense H II region component, a diffuse low-ionisation gas is needed to explain the [Ne II], [N II], and [C II] line intensities. The [O III] 88 mu m line intensity is not fully reproduced by our model, hinting towards the possible presence of yet another low-density high-ionisation medium. The [O I] emission is consistent with a dense PDR of low covering factor, and we find no evidence for an X-ray dominated component. The PDR component accounts for only 10% of the [C II] emission. Magnetic fields, known to be strong in star-forming regions, may dominate the pressure in the PDR. For example, for field strengths of the order of 100 mu G, up to 50% of the [C II] emission may come from the PDR.