The high-k-based MOS-like capacitors are a promising approach for the domain of non-volatile memory devices, which currently is limited by SiO2 technology and cannot face the rapid downsizing of the electronic device trend. In this paper, we prepare MOS-like trilayer memory structures based on high-k ZrO2 by magnetron sputtering, with a 5% and a 10% concentrations of Zr in the Zr–ZrO2 floating gate layer. For crystallization of the memory structure, rapid thermal annealing at different temperatures between 500 °C and 700 °C was performed. Additionally, Al electrodes were deposited in a top-down configuration. High-resolution transmission electron microscopy reveals that ZrO2 has a polycrystalline–columnar crystallization and a tetragonal crystalline structure, which was confirmed by X-ray diffraction measurements. It is shown that the tetragonal phase is stabilized during the crystallization by the fast diffusion of oxygen atoms. The capacitance–voltage characteristics show that the widest memory window (ΔV = 2.23 V) was obtained for samples with 10% Zr annealed at 700 °C for 4 min. The charge retention characteristics show a capacitance decrease of 36% after 10 years.