A fundamental understanding of superconducting radio frequency Nb cavity processing is necessary to achieve the desired improvement in their performance, which is needed for further upgrades of modern particle accelerators. To recognize the physical processes behind the losses in the accelerator modules, it is required to address not only the observed improvements but also the degradation occurring after different surface treatments. Here, we report on microscopic and spectroscopic studies of several cutouts from an extremely well performing cavity, which showed a systematic degradation after modified surface treatments and annealing conditions. Our results suggest that an abundance of low-angle grain boundaries surrounding the small sized grains can be related to the local superconductivity breakdown at high accelerating field gradients. Losses due to grain boundary segregated carbides are discussed to being most dominant and to leading to an anomalous Q-degradation of the whole cavity starting at low fields.