Very little success has been reported in the literature in developing diagnostic systems trained on simulated data that can accurately describe the real situation. Furthermore, no studies attempting automated structural health monitoring (SHM) system performance qualification are available. A diagnostic algorithm based on an artificial neural network and trained with finite element simulated strains has been verified during repeated fatigue crack growth tests on metallic helicopter fuselage panels. Strain measures from a network of fiber Bragg gratings are provided as input to the diagnostic system, allowing fatigue crack damage identification. Anomaly detection performances have been evaluated with reference to the recent Aerospace Recommended Practice (ARP-6461) and the Recommended Practice for a Demonstration of Nondestructive Evaluation Reliability on Aircraft Production Parts, providing a SHM system qualification in terms of minimum detectable crack length, based on reliability-confidence curves. In particular, repeated fatigue crack growth tests on metallic aerospace panels allowed the estimation of the probability of detection as a function of crack length. Furthermore, a numerical model of the monitored structure has been used for the generation of virtual specimens, thus enabling the model-assisted probability of detection assessment.