Optical encryption has attracted wide attention for its remarkable characteristics. Inspired by the development of double random phase encoding, many researchers have developed a number of optical encryption systems for practical applications. It has also been found that computer-generated hologram (CGH) is highly promising for optical encryption, and the CGH-based optical encryption possesses remarkable advantages of simplicity and high feasibility for practical implementations. An input image, i.e., plaintext, can be iteratively or non-iteratively encoded into one or several phase-only masks via phase retrieval algorithms. Without security keys, it is impossible for unauthorized receivers to correctly extract the input image from ciphertext. However, cryptoanalysis of CGH-based optical encryption systems has not been effectively carried out before, and it is also concerned whether CGH-based optical encryption is sufficiently secure for practical applications. In this paper, learning-based attack is proposed to demonstrate the vulnerability of CGH-based optical security system without the direct retrieval of optical encryption keys for the first time to our knowledge. Many pairs of the extracted CGH patterns and their corresponding input images (i.e., ciphertext-plaintext pairs) are used to train a designed learning model. After training, it is straightforward to directly retrieve unknown plaintexts from the given ciphertexts (i.e., phase-only masks) by using the trained learning model without subsidiary conditions. Moreover, the proposed learning-based attacks are also feasible and effective for the cryptoanalysis of CGH-based optical security systems with multiple cascaded phase-only masks. The proposed learning-based attacking method paves the way for the cryptoanalysis of CGH-based optical encryption.