Noise annotation lines are a promising technique to visualize thematic uncertainty in maps. However, their potential has not yet been evaluated in user studies. In two experiments, we assessed the usability of this technique with respect to a different number of uncertainty levels as well as the influence of two design aspects of noise annotation lines: the grain and the width of the noise grid. We conducted a web-based study utilizing a qualitative comparison of 2 areas in 150 different maps. We recruited participants from Amazon Mechanical Turk with the majority being nonexperts with respect to the use of maps. Our findings suggest that for qualitative comparisons of “constant uncertainty” (i.e., constant uncertainty per area) in thematic maps, noise annotation lines can be used for up to six uncertainty levels. During comparison of four, six, and eight levels, the different designs of the technique yielded significantly different accuracies. We propose to use the “large noise width, coarse grain” design that was most successful. For “mixed uncertainty” (i.e., uncertainty is not constant per area) we observed a significant decrease in accuracy, but for four levels the technique can still be recommended. This article is a follow-up to our conference paper reporting on preliminary results of the first of the two experiments.