We developed an algorithm to detect premature atrial contraction (PAC) and premature ventricular contraction (PVC) using photoplethysmographic (PPG) data acquired from a smartwatch. Our PAC/PVC detection algorithm is composed of a sequence of algorithms that are combined to discriminate various arrhythmias. A novel vector resemblance method is used to enhance the PAC/PVC detection results of the Poincaré plot method. The new PAC/PVC detection algorithm with our automated motion and noise artifact detection approach yielded a sensitivity of 86% for atrial fibrillation (AF) subjects while the overall sensitivity was 67% when normal sinus rhythm (NSR) subjects were also included. The specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy values for the combined data consisting of both NSR and AF subjects were 97%, 81%, 94% and 92%, respectively, for PAC/PVC detection combined with our automated motion and noise artifact detection approach. Moreover, when AF detection was compared with and without PAC/PVC, the sensitivity and specificity increased from 94.55% to 98.18% and from 95.75% to 97.90%, respectively. For additional independent testing data, we used two datasets: a smartwatch PPG dataset that was collected in our ongoing clinical study, and a pulse oximetry PPG dataset from the Medical Information Mart for Intensive Care III database. The PAC/PVC classification results of the independent testing on these two other datasets are all above 92% for sensitivity, specificity, PPV, NPV, and accuracy. The proposed combined approach to detect PAC and PVC can ultimately lead to better accuracy in AF detection. This is one of the first studies involving detection of PAC and PVC using PPG recordings from a smartwatch. The proposed method can potentially be of clinical importance as this enhanced capability can lead to fewer false positive detections of AF, especially for those NSR subjects with frequent episodes of PAC/PVC.