The morphology of the central sulcus (CS), at the level of the hand primary motor cortex, has been shown to be related to hand preference and skill. Differences in the cerebral functional organisation of left and right-handers have been described, notably with respect to hemispheric specialisation, which might cause the neural substrate of hand dominance or skill to differ between the two groups. Here, we further explored the relationship between the anatomical variability of the central sulcus and hand skill in two groups of young male subjects differing by handedness (n = 56 right-handers and n = 55 left-handers). Grey matter volume (GMV) in the upper region of the central sulcus was estimated with Voxel Based Morphometry, using a probabilistic region of interest approach, while hand motor skill was measured with the finger tapping test. No significant anatomical differences could be evidenced between the two hand preference groups, a rightward hemispheric asymmetry being observed in both samples. However, multiple regression analyses showed that, in the right-handed group, the maximum tapping rate of the right hand correlated positively with the left central sulcus GMV, but negatively with the right. Similar analyses showed that, in left-handers, the maximum tapping rate of the non-dominant right hand was strongly correlated with the GMV of the ipsilateral CS but not significantly with that of the contralateral CS. These results may be due to differences in the organisation of motor systems between these two groups, possibly concerning a left hemispheric specialisation for fast repetitive movements in right-handers, which would be different in left-handers.