Background Dietary diversity can play an important role in providing essential nutrients for both mother and fetus during pregnancy. This study aimed to assess the factors associated with dietary diversity during pregnancy in the western hill region of Nepal. Methods A cross-sectional study of 327 pregnant women was conducted in an urban municipality of Baglung district in the western hill region of Nepal. A semi-structured questionnaire was used to collect information on household demographic and socioeconomic status, food taboos, household food security status, nutrition-related knowledge in pregnancy, and women’s empowerment. Women consuming ≥5 of 10 food groups in the past 24 hours were defined as consuming a diverse diet using the Minimum Dietary Diversity Score for Women (MDD-W) tool. Bivariate and multivariate logistic regression was used to estimate crude odds ratio (cOR) and adjusted odds ratios (aOR) and 95% confidence intervals (CIs) to understand factors associated with dietary diversity. Results Almost 45% (95% CI: 39.6–50.4) of the participants did not consume a diverse diet and the mean dietary diversity score was 4.76 ± 1.23. Multivariable analysis revealed that women with greater empowerment (aOR = 4.3, 95% CI: 1.9–9.9), from wealthier households (aOR = 5.1, 95% CI: 2.7–9.3), joint families (aOR = 2.7, 95% CI: 1.4–5.1), employment (aOR = 2.2, 95% CI: 1.2–4.1), and had adequate nutrition knowledge (aOR: 1.9, 95% CI 1.1–3.4) had higher odds of dietary diversity. Conclusion Along with socioeconomic status, women’s empowerment and nutrition knowledge were modifiable risk factors that should be considered as targets for programs to improve women’s health during pregnancy.