Cancellation tests have traditionally been used to measure visuospatial neglect. Recently, such tests have also been viewed as potential measures of executive function. Mark and colleagues (2004; Disorganized search on cancellation is not a consequence of neglect. Neurology, 63, 78-84) developed three objective measures of organization based on reconstructing target-marking pathways from one form of a cancellation test: mean inter-target distance, path intersection rate, and a quantification of overall path uniformity (i.e., predominantly radial versus horizontal cancellation progress) that they termed "best r". However, the validity of these measures with respect to a subjective judgment of organization was not assessed. In the present study, 50 observers rated the overall organization of 50 reconstructed pathways from stroke patients and non-brain-injured adults to allow evaluating the convergent validity of the three proposed executive organization measures. Inter-target distance (r = .86), intersection rate (r = .87), and best r (r = -.69) were all found to be highly correlated with observer ratings. These results indicate good convergent validity for all three executive organization measures on a form of the cancellation test.