With its height-resolved measurements and near global coverage, the CALIOP lidar onboard the CALIPSO satellite offers a new capability for aerosol retrievals in cloudy skies. Validation of these retrievals is difficult, however, as independent, collocated and co-temporal datasets are generally not available. In this paper, we evaluate CALIOP aerosol products above opaque water clouds by applying multiple retrieval techniques to CALIOP Level 1 profile data and comparing the results. This approach allows us to both characterize the accuracy of the CALIOP above-cloud aerosol optical depth (AOD) and develop an error budget that quantifies the relative contributions of different error sources. We focus on two geographical regions: the African dust transport pathway over the tropical North Atlantic and the African smoke transport pathway over the southeastern Atlantic. Six years of CALIOP observations (2007–2012) from the Northern Hemisphere summer and early fall are analyzed. The analysis is limited to cases where aerosol layers are located above opaque water clouds so that a constrained retrieval technique can be used to directly retrieve 532 nm aerosol optical depth and lidar ratio. For the moderately dense Sahara dust layers detected in the CALIOP data used in this study, the mean/median value of the lidar ratios derived from a constrained opaque water cloud (OWC) technique is 45.1/44.4 ± 8.8 sr, which is somewhat larger than the value of 40 ± 20 sr used in the CALIOP level 2 (L2) data products. Comparisons of CALIOP L2 AOD with the OWC-retrieved AOD reveal that for nighttime conditions the L2 AOD in the dust region is underestimated on average by ∼26% (0.184 vs. 0.248). Examination of the error sources indicates that errors in the L2 dust AOD are primarily due to use of a lidar ratio which is somewhat too small and to misestimates of dust layer base heights. The mean/median lidar ratio retrieved for smoke is 69.4/70.4 ± 16.2 sr, which is consistent with the modeled value of 70 ± 28 sr used in the CALIOP L2 retrieval. Smoke AOD is found to be underestimated, on average, by ∼39% (0.191 vs. 0.311). The primary cause of AOD differences in the smoke transport region is the tendency of the CALIOP layer detection scheme to prematurely assign layer base altitudes and thus underestimate the geometric thickness of smoke layers.