Bell peppers are one of the most important species consumed and cultivated in Spain. Peppers are a source of carotenoids and phenolic compounds widely associated with biological activities such as antimicrobial, antiseptic, anticancer, counterirritant, cardioprotective, appetite stimulator, antioxidant, and immunomodulator. However, undersized and damaged fruits are usually wasted. Thus, in order to evaluate the phenolic content, a Box–Behnken design has been carried out to optimize the extraction from Capsicum annuum yellow pepper by ultrasound-assisted extraction (UAE). The independent factors were time (min), ethanol/water (% v/v) and solvent/sample ratio (v/w). The model was validated by ANOVA and confirmed. Furthermore, the whole pepper and the pepper without peduncles and seeds were extracted using optimal conditions and characterized by HPLC-ESI-TOF-MS. Moreover, their antioxidant activities, measured by three different methods (DPPH, ABTS, and FRAP), carotenoid composition, assessed by HPLC-MS, and chlorophyll content, assessed by a spectrophotometric method, were compared. A total of 38 polar compounds were found of which seven have been identified in pepper fruit extracts for the first time. According to the results, whole pepper (WP) samples presented higher content in phenolic acids; meanwhile, the edible portion (EP) was higher in flavonoids. No differences were found in the antioxidant activity except for the FRAP assay where the WP sample showed higher radical scavenging activity. EP samples showed the highest content of carotenoids and WP ones in chlorophylls.