Introduction Animal source foods contain quality nutrients, immunity, and behavioral outcome and are important for growth, and development. However, evidence on the level of animal source food consumption frequency and associated factors among pregnant women in Ethiopia, particularly rural residents are limited. Therefore, this study aimed to assess the consumption frequency of animal source food and to identify associated factors among pregnant women in the Haramaya district. Methods A community-based cross-sectional study was conducted among 448 pregnant women. Data were collected through face-to-face interviews by trained research assistants, using a validated frequency questionnaire. Consumption of animal food sources was assessed by counting the frequency of each food from animal sources that pregnant women ate over a seven-day reference period. The highest tertile for animal source food consumption was considered as the high frequency of animal source food consumption; whereas the two lower tertiles were taken as the low frequency of animal source food consumption. A binary logistic regression model was used to investigate the association of the independent variables with the animal source food consumption. An adjusted odds ratio with a 95% confidence interval was reported to show an association using a p-value <0.05. Results The high frequency of animal source food consumption among the study participants was 24.78% (95% CI = 21%-29%). High animal source food consumption was more likely higher among respondents who were literate (AOR = 1.80; 95% CI = 1.048–3.095), and those who owned milk cows (ARO = 1.70; 95% CI = 1.003–2.863). However, respondent who reported chewing khat (AOR = 0.51; 95% CI = 0.313–0.805) (AOR = 0.56; 95% CI = 0.349–0.903), were less likely experienced animal source food consumption. Conclusion We found low animal source food consumption among pregnant women in this predominantly rural setting. Women’s educational level and milk cow ownership were positively associated with animal source food consumption. Additionally, a lower frequency of animal source food consumption was observed among women who reported chewing khat. Therefore, nutrition policy programs and interventions aimed at encouraging maternal nutritional guidance and counseling are recommended.