Agricultural land abandonment and the consequent natural forest regrowth in areas once cultivated has significantly transformed Alpine ecosystems and landscapes. In this paper, we estimated the loss of agricultural areas due to reafforestation in the Belluno Province (Eastern Italian Alps) and integrated socioeconomic indicators with topographical features to evaluate the drivers of this change. Land use mapping, obtained from photo-interpretation of aerial photographs and technical maps, was used to quantify changes in forest and agricultural areas (crops, meadows and pastures) during the period of 1980–2000. On average, the forest surface expanded by 21%, while the agricultural areas decreased by 40%. This loss increased in areas with steeper slope, which confirms that the areas that would be first abandoned are those that are less productive and more difficult to manage. The reafforestation rate was particularly strong in the north of the province, where the livestock sector experienced a dramatic decline. In light of this situation and using multiple regression models with a large set of socioeconomic and agricultural indicators, we found that the loss of agricultural areas in 69 municipalities was primarily counterbalanced by the maintenance of livestock farming. The loss of steeper agricultural areas was counteracted by traditional extensive systems; however, it was not counteracted by modern intensive systems, which has important implications because steeper areas have higher landscape and biodiversity values. In addition, tourism development had a positive, although smaller, effect on the maintenance of agricultural areas, while industry development had a negative effect. The results of this study suggest that efforts are needed to maintain a territorial network of traditional extensive farms to avoid further landscape deterioration in Alpine areas. Likewise, certain choices toward the economic development of local communities might have an impact on land conservation.