Abstract People’s observations of climate change and its impacts, mediated by cultures and capacities, shape adaptive responses. Adaptation is critical in regions of rainfed smallholder agriculture where changing rainfall patterns have disproportionate impacts on livelihoods, yet scientific climate data to inform responses are often sparse. Despite calls for better integration of local knowledge into adaptation frameworks, there is a lack of empirical evidence linking both smallholder climate observations and scientific data to on-farm outcomes. We combine smallholder observations of past seasonal rainfall timing with satellite-based rainfall estimates in Uganda to explore whether farmers’ ability to track climate patterns is associated with higher crop yields. We show that high-fidelity tracking, or alignment of farmer recall with recent rainfall patterns, predicts higher yields in the present year, suggesting that farmers may translate their cumulative record of environmental knowledge into productive on-farm decisions, such as crop selection and timing of planting. However, tracking of less-recent rainfall (i.e., 1–2 decades in the past) does not predict higher yields in the present, while climate data indicate significant trends over this period toward warmer and wetter seasons. Our findings demonstrate the value of smallholder knowledge systems in filling information gaps in climate science while suggesting ways to improve adaptive capacity to climate change.