Polarization-sensitive optical coherence tomography (PS-OCT) may facilitate in vivo measurement of airway smooth muscle (ASM). We present a quantitative validation correlating absolute ASM area from PS-OCT to directly matched histological cross sections using human tissue. A major confounder for ASM quantification was observed and resolved: fibrous perichondrium surrounding hyaline cartilage in human airways presents a PS-OCT signature similar to ASM for birefringence and optic axis orientation. Findings impact the development of automated methods for ASM segmentation.