Omnidirectional cameras are widely used for robotic applications in structured environments. However, because of the distorted field of view (FOV), it is hard to describe the primitive features extracted from them robustly. In this paper, we tackle the problem by using Histogram of Gradient (HoG) statistics for the regions of interest (ROI) in the neighborhood of major vertical lines extracted from the panoramic image. As a validation, we compare the proposed algorithm with state-of-the-art based on two widely used data-sets, leading to evidently better performance. We also introduce a scene reconstruction scenario using the proposed descriptor based on 1D Trifocal Tensor framework. The comparative results show the competence of the descriptor.