High-Resolution Computed Tomography (HRCT) plays a central role in diagnosing Idiopathic Pulmonary Fibrosis (IPF) while its role in monitoring disease progression is not clearly defined. Given the variable clinical course of the disease, we evaluated whether HRCT abnormalities predict disease behavior and correlate with functional decline in untreated IPF patients. Forty-nine patients (with HRCT1) were functionally categorized as rapid or slow progressors. Twenty-one had a second HRCT2. Thirteen patients underwent lung transplantation and pathology was quantified. HRCT Alveolar (AS) and Interstitial Scores (IS) were assessed and correlated with Forced Vital Capacity (FVC) decline between HRCT1 and HRCT2. At baseline, AS was greater in rapids than in slows, while IS was similar in the two groups. In the 21 subjects with HRCT2, IS increased over time in both slows and rapids, while AS increased only in rapids. The IS change from HRCT1 to HRCT2 normalized per month correlated with FVC decline/month in the whole population, but the change in AS did not. In the 13 patients with pathology, the number of total lymphocytes was higher in rapids than in slows and correlated with AS. Quantitative estimation of HRCTs AS and IS reflects the distinct clinical and pathological behavior of slow and rapid decliners. Furthermore, AS, which reflects the immune/inflammatory infiltrate in lung tissue, could be a useful tool to differentiate rapid from slow progressors at presentation.