In Taiwan, the age-standardized incidence of EC, especially esophageal squamous cell carcinoma (ESCC), has increased substantially during the past thirty years. We described the incidence trends of EC from 1985–2019 by an average annual percentage change (AAPC) and age-period-cohort model by using Taiwan Cancer Registry data. Age-period-cohort modeling was used to estimate the period and cohort effects of ESCC and esophageal adenocarcinoma (EAC). The Spearman’s correlation coefficient was used to analyze the correlation between age-adjusted incidence rates of EC and the prevalence of risk factors from national surveys. The results showed the incidence rate of ESCC in men (AAPC = 4.2, 95% CI = 3.1–5.4, p < 0.001) increased prominently from 1985–1989 to 2015–2019 while that of EAC in men (AAPC = 1.2, 95% CI = 0.9–1.5, p < 0.001) and ESCC in women (AAPC = 1.7, 95% CI = 1.4–2.1, p < 0.001) increased to a lesser degree. Increased period effects were observed in ESCC in men, ESCC in women, and EAC in men. High correlations were found between the risk factors and the increased birth-cohort effects of ESCC (p < 0.05). To conclude, the incidence of ESCC in both sex and EAC in men increased with statistical significance in recent decades. The increased prevalence of risk factors from approximately 1970–1995 could explain the increased cohort effects of ESCC.