Context. Reference-star differential imaging (RDI) is a promising technique in high-contrast imaging that is thought to be more sensitive to exoplanets and disks than angular differential imaging (ADI) at short angular separations (i.e., <0.3’). However, it is unknown whether the performance of RDI on ground-based instruments can be improved by using all the archival data to optimize the subtraction of stellar contributions. Aims. We characterize the performance of RDI on SPHERE/IRDIS data in direct imaging of exoplanets and disks. Methods. We made use of all the archival data in H23 obtained by SPHERE/IRDIS in the past 5 yr to build a master reference library and perform RDI. To avoid biases caused by limited test targets under specific conditions, 32 targets were selected to obtain the average performances of RDI under different conditions, and we compared the performances with those of ADI. Results. In the point-source detection, RDI can outperform ADI at small angular separations (<0.4’) if the observing conditions are around the median conditions of our master reference library. On average, RDI has a gain of ~0.8 mag over ADI at 0.15’ separation for observations under median conditions. We demonstrate that including more reference targets in the master reference library can indeed help to improve the performance of RDI. In disk imaging, RDI can reveal more disk features and provide a more robust recovery of the disk morphology. We resolve 33 disks in total intensity (19 planet-forming disks and 14 debris disks), and 4 of them can only be detected with RDI. Two disks are resolved in scattered light for the first time. Three disks are detected in total intensity for the first time. Conclusions. RDI is a promising imaging technique for ground-based instruments such as SPHERE. The master reference library we built in this work can be easily implemented into legacy or future SPHERE surveys to perform RDI, achieving better performance than that of ADI. To obtain optimal RDI gains over ADI, we recommend future observations be carried out under seeing conditions of 0.6’–0.8’.