Many works on speech processing have dealt with auto-regressive (AR) models for spectral envelope and formant frequency estima­ tion, mostly focusing on the estimation of the AR parameters. How­ ever, it is also interesting to be able to directly estimate the formant frequencies, or equivalently the poles of the AR filter. To tackle this issue, we propose in this paper to decompose the signal onto several bases, one for each formant, taking advantage of recent works on nonnegative matrix factorization (NMF) for the estimation stage, fur­ ther refined by sparsity and smoothness penalties. The results are en­ couraging, and the proposed system provides formant tracks which seem robust enough to be used in different applications such as pho­ netic analysis, emotion detection or as visual cue for computer-aided pronunciation training applications. The model can also be extended to deal with multiple-speaker signals.