Tinnitus is an auditory phantom perception in the ears or head in the absence of a corresponding external stimulus. There is currently no effective treatment available that reliably reduces tinnitus. Educational counseling is a treatment approach that aims to educate patients and inform them about possible coping strategies. For this feasibility study, we implemented educational material and self-help advice in a smartphone app. Participants used the educational smartphone app unsupervised during their daily routine over a period of four months. Comparing the tinnitus outcome measures before and after smartphone-guided treatment, we measured changes in tinnitus-related distress, but not in tinnitus loudness. Improvements on the Tinnitus Severity numeric rating scale reached an effect size of 0.408, while the improvements on the Tinnitus Handicap Inventory (THI) were much smaller with an effect size of 0.168. An analysis of user behavior showed that frequent and intensive use of the app is a crucial factor for treatment success: participants that used the app more often and interacted with the app intensively reported a stronger improvement in the tinnitus. Between study allocation and final assessment, 26 of 52 participants dropped out of the study. Reasons for the dropouts and lessons for future studies are discussed in this paper.