Abstract Magnetic topological insulators, such as MnBi₂Te₄ have attracted great attention recently due to their application to the quantum anomalous Hall (QAH) effect. However, the magnetic quantum spin Hall (QSH) effect in two-dimensional (2D) materials has not yet been reported. Here based on first-principle calculations we find that Ti₂Te₂O, a van der Waals layered compound, can cherish both the QAH and QSH states, depending on the magnetic order in its single layer. If the single layer was in a chessboard antiferromagnetic (FM) state, it is a QSH insulator which carries two counterpropagating helical edge states. The spin–orbit-couplings induced bulk band gap can approach as large as 0.31 eV. On the other hand, if the monolayer becomes FM, exchange interactions would push one pair of bands away from the Fermi energy and leave only one chiral edge state remaining, which turns the compound into a Chern insulator (precisely, it is semimetallic with a topologically direct band gap). Both magnetic orders explicitly break the time reversal symmetry and split the energy bands of different spin orientations. To our knowledge, Ti₂Te₂O is the first compound that predicted to possess both intrinsic QSH and QAH effects. Our works provide new possibilities to reach a controllable phase transition between two topological nontrivial phases through magnetism tailoring.