Elucidation of complex phenotypes for mechanistic insights presents a significant challenge in systems biology. We report a strategy to automatically infer mechanistic models of cell fate differentiation based on live-imaging data. We use cell lineage tracing and combinations of tissue-specific marker expression to assay progenitor cell fate and detect fate changes upon genetic perturbation. Based on the cellular phenotypes, we further construct a model for how fate differentiation progresses in progenitor cells and predict cell-specific gene modules and cell-to-cell signaling events that regulate the series of fate choices. We validate our approach in C. elegans embryogenesis by perturbing 20 genes in over 300 embryos. The result not only recapitulates current knowledge but also provides insights into gene function and regulated fate choice, including an unexpected self-renewal. Our study provides a powerful approach for automated and quantitative interpretation of complex in vivo information.