Multilayer diffractive optical neural networks (DONNs) can perform machine learning (ML) tasks at the speed of light with low energy consumption. Decreasing the number of diffractive layers can reduce inevitable material and diffraction losses to improve system performance, and incorporating compact devices can reduce the system footprint. However, current analytical DONN models cannot accurately describe such physical systems. Here we show the ever-ignored effects of interlayer reflection and interpixel interaction on the deployment performance of DONNs through full-wave electromagnetic simulations and terahertz (THz) experiments. We demonstrate that the drop of handwritten digit classification accuracy due to reflection is negligible with conventional low-index THz polymer materials, while it can be substantial with high-index materials. We further show that one- and few-layer DONN systems can achieve high classification accuracy, but there is a trade-off between accuracy and model-system matching rate because of the fast-varying spatial distribution of optical responses in diffractive masks. Deep DONNs can break down such a trade-off because of reduced mask spatial complexity. Our results suggest that new accurate and trainable DONN models are needed to advance the development and deployment of compact DONN systems for sophisticated ML tasks.