Ca3Mn2O7 and Ca3Ti2O7 have been proposed as the prototypical hybrid improper ferroelectrics (HIFs), and a significant magnetoelectric (ME) coupling in magnetic Ca3Mn2O7 is, in fact, reported theoretically and experimentally. Although the switchability of polarization is confirmed in Ca3Ti2O7 and other non-magnetic HIFs, there is no report of switchable polarization in the isostructural Ca3Mn2O7. We constructed the phase diagram of Ca3Mn2-xTixO7 through our systematic study of a series of single crystalline Ca3Mn2-xTixO7 (x = 0, 0.1, 1, 1.5, and 2). Using transmission electron microscopy, we have unveiled the unique domain structure of Ca3Mn2O7: the high-density 90° stacking of a- and b-domains along the c-axis due to the phase transition through an intermediate Acca phase and the in-plane irregular wavy ferroelastic twin domains. The interrelation between domain structures and physical properties is unprecedented: the stacking along the c-axis prevents the switching of polarization and causes the irregular in-plane ferroelastic domain pattern. In addition, we have determined the magnetic phase diagram and found complex magnetism of Ca3Mn2O7 with isotropic canted moments. These results lead to negligible observable ME coupling in Ca3Mn2O7 and guide us to explore multiferroics with large ME coupling.