Fluorescence endoscopy is a novel imaging technique that offers a non-invasive means to diagnose and stage cancers without the need to conduct biopsies of suspected lesions. Work carried out by our research group showed that fluorescence diagnosis can be further enhanced by incorporating a ratio diagnostic algorithm in the system. Currently images captured using these imaging techniques have to be processed and analyzed off-line, adding a delay to the diagnosis process. We aim to develop a real-time image processing and analysis system to be used with fluorescence endoscopy for early diagnosis and staging of oral and bladder cancers. Fast capturing of suspicious features implied by the fluorescence images provides a means for efficient communication, and an optimized and focused in vivo imaging process. The ultimate aim is to provide an accurate, sensitive and non-invasive real-time cancer diagnosis and staging system that can be used in an outpatient clinical setting. In this paper, we describe the framework of such an imaging system, as well as the initial algorithm development and implementation with the field-programmable gate arrays.