Context. The remnants of core-collapse supernovae (SNe) are probes of the physical processes associated with their parent SNe. Aims. Here we aim to explore to which extent the remnant keeps memory of the asymmetries that develop stochastically in the neutrino-heating layer due to hydrodynamic instabilities (e.g., convective overturn and the standing accretion shock instability; SASI) during the first second after core bounce. Methods. We coupled a three-dimensional (3D) hydrodynamic model of a neutrino-driven SN explosion, which has the potential to reproduce the observed morphology of the Cassiopeia A (Cas A) remnant, with 3D (magneto)-hydrodynamic simulations of the remnant formation. The simulations cover ≈2000 yr of expansion and include all physical processes relevant to describe the complexities in the SN evolution and the subsequent interaction of the stellar debris with the wind of the progenitor star. Results. The interaction of large-scale asymmetries left from the earliest phases of the explosion with the reverse shock produces, at the age of ≈350 yr, an ejecta structure and a remnant morphology which are remarkably similar to those observed in Cas A. Small-scale structures in the large-scale Fe-rich plumes that were created during the initial stages of the SN, combined with hydrodynamic instabilities that develop after the passage of the reverse shock, naturally produce a pattern of ring- and crown-like structures of shocked ejecta. The consequence is a spatial inversion of the ejecta layers with Si-rich ejecta being physically interior to Fe-rich ejecta. The full-fledged remnant shows voids and cavities in the innermost unshocked ejecta, which are physically connected with ring-like features of shocked ejecta in the main shell in most cases, resulting from the expansion of Fe-rich plumes and their inflation due to the decay of radioactive species. The asymmetric distributions of ⁴⁴Ti and ⁵⁶Fe, which are mostly concentrated in the northern hemisphere, and pointing opposite to the kick velocity of the neutron star, as well as their abundance ratio are both compatible with those inferred from high-energy observations of Chandra and NuSTAR. Finally, the simulations show that the fingerprints of the SN can still be visible ≈2000 yr after the explosion. Conclusions. The main asymmetries and features observed in the ejecta distribution of Cas A can be explained by the interaction of the reverse shock with the initial large-scale asymmetries that developed from stochastic processes (e.g., convective overturn and SASI activity) that originate during the first seconds of the SN blast.