Widefield quantum microscopy based on nitrogen-vacancy (NV) centers in diamond has emerged as a powerful technique for quantitative mapping of magnetic fields with a sub-micrometer resolution. However, the accuracy of the technique has not been characterized in detail so far. Here, we show that optical aberrations in the imaging system may cause large systematic errors in the measured quantity beyond trivial blurring. We introduce a simple theoretical framework to model these effects, which extends the concept of a point spread function to the domain of spectral imaging. Using this model, the magnetic field imaging of test magnetic samples is simulated under various scenarios, and the resulting errors are quantified. We then apply the model to previously published data, show that apparent magnetic anomalies can be explained by the presence of optical aberrations, and demonstrate a post-processing technique to retrieve the source quantity with improved accuracy. This work presents a guide to predict and mitigate aberration induced artifacts in quantitative NV-based widefield imaging and in spectral imaging more generally.