Inorganic nanoparticles have shown promising potentials as novel biomedical imaging agents with high sensitivity, high spatial and temporal resolution. To translate the laboratory innovations into clinic applications, their potential toxicities are highly concerned and have to be evaluated comprehensively both in vitro and in vivo before their clinical applications. In this review, we first summarized the in vivo and in vitro toxicities of the representative inorganic nanoparticles used in biomedical imagings. Then we further discuss the origin of nanotoxicity of inorganic nanomaterials, including ROS generation and oxidative stress, chemical instability, chemical composition, the surface modification, dissolution of nanoparticles to release excess free ions of metals, metal redox state, and left-over chemicals from synthesis, etc. We intend to provide the readers a better understanding of the toxicology aspects of inorganic nanomaterials and knowledge for achieving optimized designs of safer inorganic nanomaterials for clinical applications.