Current imaging technology provides an experimental platform in which complex developmental processes can be observed at cellular resolution over an extended time frame. New computational tools are essential to achieve a systems-level understanding of this high-content information. We have devised a structured approach to systematically analyze complex in vivo phenotypes at cellular resolution, which divides the task into a panel of statistical measurements of each cell in terms of cell differentiation, proliferation and morphogenesis, followed by their spatial and temporal organization in groups and the cohesion within the whole specimen. We demonstrate the approach to C. elegans embryogenesis with in toto imaging and automated cell lineage tracing. We define statistical distributions of the wild-type developmental behaviors at single-cell resolution based on over 50 embryos, cumulating in over 4000 distinct, developmentally based measurements per embryo. These methods enable statistical quantification of abnormalities in mutant or RNAi-treated embryos and a rigorous comparison of embryos by testing each measurement for the probability that it would occur in a wild-type embryo. We demonstrate the power of this structured approach by uncovering quantitative properties including subtle phenotypes in both wild-type and perturbed embryos, transient behaviors that lead to new insights into gene function and a previously undetected source of developmental noise and its subsequent correction.