Tree nut and/or peanut allergy impairs patients’ quality of life, but data on the impact of age and the type of nut or peanut on the quality of life are lacking. To evaluate the impact at different ages, age-appropriate survey questionnaires accompanied by FAQLQ and FAIM were distributed to patients with suspected tree nut and/or peanut allergy who presented at the allergy departments of three hospitals in Athens. Out of 200 questionnaires distributed, 106 met the inclusion criteria (46 children, 26 teenagers, 34 adults). The median score of each age group for FAQLQ was 4.6 (3.3–5.1), 4.7 (3.9–5.5), and 3.9 (3.2–5.1) and for FAIM was 3.7 (3.0–4.0), 3.4 (2.8–4.0), and 3.2 (2.7–4.1), respectively. FAQLQ and FAIM scores were correlated with the reported probability of using the rescue anaphylaxis set upon reaction (15.4%, p = 0.04 and 17.8%, p = 0.02, respectively) and pistachio allergy (FAQLQ: 4.8 vs. 4.0, p = 0.04; FAIM: 3.5 vs. 3.2, p = 0.03). Patients with additional food allergies reported worse FAQLQ scores (4.6 vs. 3.8, p = 0.05). Worse FAIM scores were associated with younger age (−18.2%, p = 0.01) and the number of life-threatening allergic reactions (25.3%, p < 0.001). The overall impact of tree nut and/or peanut allergy on patients’ quality of life is moderate but differs with age, the type of nut, the use of adrenaline, and the number of previous reactions. The aspects of life affected and contributed factors also vary across age groups.