One hundred primary ketamine users and 100 controls were recruited in Hong Kong between December 2009 and December 2011. Cognitive assessment included general intelligence, working, verbal, and visual memory, and executive functions. A Univariate General Linear Model was used to compare cognitive performance between the male and female ketamine users and controls. The female users appeared to have a higher risk of visual memory impairment than their male counterparts. Further studies are warranted to clarify the mechanism of the sex-specific effect of ketamine on cognitive functions.