Lung cancer is one of the most common causes of cancer deaths in the modern world. Screening of lung nodules is essential for early recognition to facilitate treatment that improves the rate of patient rehabilitation. An increase in accuracy during lung cancer detection is vital for sustaining the rate of patient persistence, even though several research works have been conducted in this research domain. Moreover, the classical system fails to segment cancer cells of different sizes accurately and with excellent reliability. This paper proposes a sooty tern optimization algorithm-based deep learning (DL) model for diagnosing non-small cell lung cancer (NSCLC) tumours with increased accuracy. We discuss various algorithms for diagnosing models that adopt the Otsu segmentation method to perfectly isolate the lung nodules. Then, the sooty tern optimization algorithm (SHOA) is adopted for partitioning the cancer nodules by defining the best characteristics, which aids in improving diagnostic accuracy. It further utilizes a local binary pattern (LBP) for determining appropriate feature retrieval from the lung nodules. In addition, it adopts CNN and GRU-based classifiers for identifying whether the lung nodules are malignant or non-malignant depending on the features retrieved during the diagnosing process. The experimental results of this SHOA-optimized DNN model achieved an accuracy of 98.32%, better than the baseline schemes used for comparison.