The present work aimed at the problems of less negative samples and more positive samples in rail fastener fault diagnosis and low detection accuracy of heavy manual patrol inspection tasks. Exploiting the capacity of a Convolution Neural Network (CNN) to process unbalanced data to solve tedious and inefficient manual processing, a fault diagnosis method based on a Generative Adversarial Network (GAN) and a Residual Network (ResNet) was developed. First, GAN was used to track the distribution of rail fastener failure data. To study the noise distribution, the mapping relationship between image data was established. Additional real fault samples were then generated to balance and extend the existing data sets, and these data sets were used as input to ResNet for recognition and detection training. Finally, the average accuracy of multiple experiments was used as the evaluation index. The experimental results revealed that the fault diagnosis of rail fastener based on GAN and ResNet could improve the fault detection accuracy in the case of a serious shortage of fault data.