Background Although primary intraventricular hemorrhage is frequently due to trauma or vascular lesions, the etiology of idiopathic primary intraventricular hemorrhage (IP-IVH) is not defined. Aims Herein, we test the hypothesis that cerebral small vessel diseases (cSVD) including hypertensive cSVD (HTN-cSVD) and cerebral amyloid angiopathy are associated with IP-IVH. Methods Brain magnetic resonance imaging from consecutive patients (January 2011 to September 2019) with non-traumatic intracerebral hemorrhage from a single referral center were reviewed for the presence of HTN-cSVD (defined by strictly deep or mixed-location intracerebral hemorrhage/cerebral microbleeds) and cerebral amyloid angiopathy (applying modified Boston criteria). Results Forty-six (4%) out of 1276 patients were identified as having IP-IVH. Among these, the mean age was 74.4 ± 12.2 years and 18 (39%) were females. Forty (87%) had hypertension, and the mean initial blood pressure was 169.2 ± 40.4/88.8 ± 22.2 mmHg. Of the 35 (76%) patients who received a brain magnetic resonance imaging, two (6%) fulfilled the modified Boston criteria for possible cerebral amyloid angiopathy and 10 (29%) for probable cerebral amyloid angiopathy. Probable cerebral amyloid angiopathy was found at a similar frequency when comparing IP-IVH patients to the remaining patients with primary intraparenchymal hemorrhage (P-IPH) (27%, p = 0.85). Furthermore, imaging evidence for HTN-cSVD was found in 8 (24%) patients with IP-IVH compared to 209 (28%, p = 0.52) patients with P-IPH. Conclusions Among IP-IVH patients, cerebral amyloid angiopathy was found in approximately one-third of patients, whereas HTN-cSVD was detected in 23%—both similar rates to P-IPH patients. Our results suggest that both cSVD subtypes may be associated with IP-IVH.