This paper discusses a novel approach to an EEG (electroencephalogram)-based driver distraction classification by using brain connectivity estimators as features. Ten healthy volunteers with more than one year of driving experience and an average age of 24.3 participated in a virtual reality environment with two conditions, a simple math problem-solving task and a lane-keeping task to mimic the distracted driving task and a non-distracted driving task, respectively. Independent component analysis (ICA) was conducted on the selected epochs of six selected components relevant to the frontal, central, parietal, occipital, left motor, and right motor areas. Granger–Geweke causality (GGC), directed transfer function (DTF), partial directed coherence (PDC), and generalized partial directed coherence (GPDC) brain connectivity estimators were used to calculate the connectivity matrixes. These connectivity matrixes were used as features to train the support vector machine (SVM) with the radial basis function (RBF) and classify the distracted and non-distracted driving tasks. GGC, DTF, PDC, and GPDC connectivity estimators yielded the classification accuracies of 82.27%, 70.02%, 86.19%, and 80.95%, respectively. Further analysis of the PDC connectivity estimator was conducted to determine the best window to differentiate between the distracted and non-distracted driving tasks. This study suggests that the PDC connectivity estimator can yield better classification accuracy for driver distractions.