Worldwide, railways are among the safest transportation services. Nevertheless, every year some serious accidents are reported. A noticeable portion of these accidents are a result of defective wheels, bearings, or brakes. Train wheels are subjected to different types of damage due to their interaction with the brakes and the track and they are required to be periodically inspected to ensure they meet all the safety criteria for proper operation. If the wheel damage remains undetected, it can worsen and result in overheating and severe damage to the wheel and track. There are a variety of wheel damages, classified in different groups based on the type and severity of the defect. The most usual cause of damage is severe braking, which applies directly to the wheel and results in local heating of the wheel. This can stop the wheel from rotating while the train is still moving, producing a defect called a “flat spot” or “hot spot”. Flat-spotted wheels are a serious concern for the railroad industry. Depending on the level of wheel defect, different solutions should be taken. This paper will focus on automatically detecting flat-spotted wheels from thermal imagery using computer vision methods and introduces an algorithm to detect hot bearings. We first extract and segment both the wheel and bearing regions from the whole image, then we introduce a fuzzy inference to categorize the level of wheel damage. The whole process is done automatically and without any need for time consuming and costly manned inspection. Based on the severity of the defect, it can be decided which solution should be taken.