The far-infrared conductivity of single-wall carbon-nanotube ensembles is dominated by a broad absorption peak around 4 THz whose origin is still debated. We observe an overall depletion of this peak when the nanotubes are excited by a short visible laser pulse. This finding excludes optical absorption due to a particle-plasmon resonance and instead shows that interband transitions in tubes with an energy gap of approximately 10 meV dominate the far-infrared conductivity. A simple model based on an ensemble of two-level systems naturally explains the weak temperature dependence of the far-infrared conductivity by the tube-to-tube variation of the chemical potential.