Methods. We conducted a prospective, double-blinded, paired-measurement validation study of 174 non-naive adult vaccinees and their physician evaluators. Subjects provided paired, blinded, independent assessments of take 7–9 days after vaccination. Results. Overall, vaccinees and evaluators agreed on 157 of 174 (90.2%) take assessments. Sensitivity of the tool was 99.1%, and specificity was 75%. The positive predictive value of self-assessment was 87.2% and the negative predictive value was 98%. Specificity of the tool and measures of agreement were significantly modified by age, education, and occupation. When adjusted for the expected take rate among a population including naive vaccinees, positive predictive value and overall agreement increased significantly. Conclusions. Self-assessment may be a feasible option for evaluation of take in the event of mass smallpox vaccination. The predictive values and overall agreement of the tool are satisfactory, and can be expected to increase when used in a largely naive population.