We sought to develop a quantitative tool to quickly determine relative differences in MRI volumes both within and between large MR imaging cohorts (such as available in The Cancer Imaging Archive (TCIA)), in order to help determine the generalizability of radiomics and machine learning schemes to unseen datasets. The tool is intended to help quantify presence of (a) site- or scanner-specific variations in image resolution, field-of-view, or image contrast, or (b) imaging artifacts such as noise, motion, inhomogeneity, ringing, or aliasing; which can adversely affect relative image quality between data cohorts. We present MRQy, a new open-source quality control tool to (a) interrogate MRI cohorts for site- or equipment-based differences, and (b) quantify the impact of MRI artifacts on relative image quality; to help determine how to correct for these variations prior to model development. MRQy extracts a series of quality measures (e.g. noise ratios, variation metrics, entropy and energy criteria) and MR image metadata (e.g. voxel resolution, image dimensions) for subsequent interrogation via a specialized HTML5 based front-end designed for real-time filtering and trend visualization. MRQy was used to evaluate (a) n=133 brain MRIs from TCIA (7 sites), and (b) n=104 rectal MRIs (3 local sites). MRQy measures revealed significant site-specific variations in both cohorts, indicating potential batch effects. Marked differences in specific MRQy measures were also able to identify outlier MRI datasets that needed to be corrected for common MR imaging artifacts. MRQy is designed to be a standalone, unsupervised tool that can be efficiently run on a standard desktop computer. It has been made freely accessible at \url{http://github.com/ccipd/MRQy} for wider community use and feedback.