Dynamical phase transitions can occur in isolated quantum systems that are brought out of equilibrium by sudden parameter changes. We discuss the characterization of such dynamical phase transitions based on the statistics of produced excitations. We consider both the O(N) model in the large N limit and a spin model with long range interactions and show that the dynamical criticality of their prethermal steady-states manifests most dramatically not in the average number of excitations but in their higher moments. We argue that the growth of defect fluctuations carries unique signatures of the dynamical criticality, irrespective of the precise details of the model. Our theoretical results should be relevant to quantum quench experiments with ultracold bosonic atoms in optical lattices. ; Comment: 10 pages; 10 figures