Spatial transcriptomics (ST) technology, providing spatially resolved transcriptional profiles, facilitates advanced understanding of key biological processes related to health and disease. Sequencing-based ST technologies provide whole-transcriptome profiles but are limited by the non–single cell–level resolution. Lack of knowledge in the number of cells or cell type composition at each spot can lead to invalid downstream analysis, which is a critical issue recognized in ST data analysis. Methods developed, however, tend to underuse histological images, which conceptually provide important and complementary information including anatomical structure and distribution of cells. To fill in the gaps, we present POLARIS, a versatile ST analysis method that can perform cell type deconvolution, identify anatomical or functional layer-wise differentially expressed (LDE) genes, and enable cell composition inference from histology images. Applied to four tissues, POLARIS demonstrates high deconvolution accuracy, accurately predicts cell composition solely from images, and identifies LDE genes that are biologically relevant and meaningful.