Recent advances in biophotonics have enabled in-vivo, in-situ histopathology for routine clinical applications. The non-invasive nature of these optical 'biopsy' techniques, however, entails the difficulty of identifying previously visited biopsy locations, particularly for surveillance examinations. This paper presents a novel region-matching approach for narrow-band endoscopy to facilitate retargeting the optical biopsy sites. The task of matching sparse affine covariant image regions is modelled in a Markov Random Field (MRF) framework. The proposed model incorporates appearance based region similarities as well as spatial correlations of neighbouring regions. In particular, a geometric constraint that is robust to deviations in relative positioning of the detected regions is introduced. In the proposed model, the appearance and geometric constraints are evaluated in the same space (photometry), allowing for their seamless integration into the MRF objective function. The performance of the method as compared to the existing state-of-the-art is evaluated with both in-vivo and simulation datasets with varying levels of visual complexities.