In a number of domains involving decision making under high time pressure, the appearance of an unexpected critical event can startle operators causing errors that may potentially lead to a chain reaction of unfavorable events. Simulating such situations in a laboratory requires overcoming several challenges tied to human responses to unexpected and rare critical events. We discuss cases of experimental studies that have incorporated simulations, critical events, interruptions, predictive displays and ecological displays involving naïve and expert participants. We spell out guidelines and caveats pertaining to training, expertise, temporal variables, critical event frequencies, scenario design, mediator and moderator variables to name a few. The guidelines and caveats may inform researchers in the design of generalizable and reproducible experiments and in the analysis of data.