Specific T cells are crucial to control SARS-CoV-2 infection, avoid reinfection and confer protection after vaccination. We have studied patients with severe or moderate COVID-19 pneumonia, compared to patients who recovered from a severe or moderate infection that had occurred about 4 months before the analyses. In all these subjects, we assessed the polyfunctionality of virus-specific CD4+ and CD8+ T cells by quantifying cytokine production after in vitro stimulation with different SARS-CoV-2 peptide pools covering different proteins (M, N and S). In particular, we quantified the percentage of CD4+ and CD8+ T cells simultaneously producing interferon-γ, tumor necrosis factor, interleukin (IL)-2, IL-17, granzyme B, and expressing CD107a. Recovered patients who experienced a severe disease display high proportions of antigen-specific CD4+ T cells producing Th1 and Th17 cytokines and are characterized by polyfunctional SARS-CoV-2-specific CD4+ T cells. A similar profile was found in patients experiencing a moderate form of COVID-19 pneumonia. No main differences in polyfunctionality were observed among the CD8+ T cell compartments, even if the proportion of responding cells was higher during the infection. The identification of those functional cell subsets that might influence protection can thus help in better understanding the complexity of immune response to SARS-CoV-2.