The impervious surface area (ISA) is a key indicator of urbanization, which brings out serious adverse environmental and ecological consequences. The ISA is often estimated from remotely sensed data via spectral mixture analysis (SMA). However, accurate extraction of ISA using SMA is compromised by two major factors, endmember spectral variability and plant phenology. This study developed a novel approach that incorporates phenology with Fisher transformation into a conventional linear spectral mixture analysis (PF-LSMA) to address these challenges. Four endmembers, high albedo, low albedo, evergreen vegetation, and seasonally exposed soil (H-L-EV-SS) were identified for PF-LSMA, considering the phenological characteristic of Shanghai. Our study demonstrated that the PF-LSMA effectively reduced the within-endmember spectral signature variation and accounted for the endmember phenology effects, and thus well-discriminated impervious surface from seasonally exposed soil, enhancing the accuracy of ISA extraction. The ISA fraction map produced by PF-LSMA (RMSE = 0.1112) outperforms the single-date image Fisher transformed unmixing method (F-LSMA) (RMSE = 0.1327) and the other existing major global ISA products. The PF-LSMA was implemented on the Google Earth Engine platform and thus can be easily adapted to extract ISA in other places with similar climate conditions.