This is a review article in which we will introduce, in a unifying fashion and with more intermediate steps in some difficult calculations, two infinite-dimensional Lie algebras of quantum matrix models, one for the open string sector and the other for the closed string sector. Physical observables of quantum matrix models in the large N limit can be expressed as elements of these Lie algebras. We will see that both algebras arise as quotient algebras of a larger Lie algebra. We will also discuss some properties of these Lie algebras not published elsewhere yet, and briefly review their relationship with well-known algebras like the Cuntz algebra, the Witt algebra and the Virasoro algebra. We will also review how the Yang–Mills theory, various low energy effective models of string theory, quantum gravity, string-bit models, and the quantum spin chain models can be formulated as quantum matrix models. Studying these algebras thus help us understand the common symmetry of these physical systems.