The generation of speckle patterns via random matrices, statistical definitions, or apertures may not always result in optimal outcomes. Issues such as correlation fluctuations in low ensemble numbers and diffraction in long-distance propagation can arise. Instead of improving results of specific applications, our solution is catching deep correlations of patterns with the framework, Speckle-Net, which is fundamental and universally applicable to various systems. We demonstrate this in computational ghost imaging (CGI) and structured illumination microscopy (SIM). In CGI with extremely low ensemble number, it customizes correlation width and minimizes correlation fluctuations in illuminating patterns to achieve higher-quality images. It also creates non-Rayleigh nondiffracting speckle patterns only through a phase mask modulation, which overcomes the power loss in the traditional ring-aperture method. Our approach provides new insights into the nontrivial speckle patterns and has great potential for a variety of applications including dynamic SIM, X-ray and photo-acoustic imaging, and disorder physics.