Abstract Background This study evaluated pre-defined aspects of content validity of the 18-item NCCN FACT-Ovarian Symptom Index (NFOSI-18) and its Disease-Related Symptoms-Physical (DRS-P) subscale, as clinical trial outcome tools for patients with advanced ovarian cancer. Methods Twenty-one women (mean age 59.5 years) diagnosed with advanced ovarian cancer completed the NFOSI-18 and participated in a cognitive interview to explore: (1) whether ‘pain’ and ‘cramps’ are considered redundant; (2) whether ‘fatigue’ and ‘lack of energy’ are overlapping concepts; (3) whether patients consider severity when responding to the item “I am bothered by constipation;” and (4) factors considered when responding to the item “I am sleeping well.” Interviews were audio-recorded, transcribed, and analyzed qualitatively. Results Pain was associated with discomfort, hurt, and life interference; ‘cramps’ was associated with pain, muscle tightening, and menstrual or digestive issues. Most (81%) considered the items “I have pain” and “I have cramps in my stomach area” to be more different than similar. Participants associated ‘fatigue’ with intense tiredness and ‘lack of energy’ with motivation and capability to complete daily activities. Item comparisons revealed a majority (65%) considered the items to be more different than similar. When responding to “I am bothered by constipation,” patients indicated constipation severity was related to bother. Finally, patients considered disease, treatment, and other factors when responding to “I am sleeping well.” Conclusions Findings support content validity of the NFOSI-18 and its DRS-P as originally constructed. We propose an alternative scoring option that excludes the item “I am sleeping well” from the DRS-P when used as a symptom-focused index for clinical research in a regulatory context.