Single image depth estimation, which aims at estimating 3-D depth from a single image, is a challenging task in computer vision since a single image does not provide any depth cue itself. Machine learning-based methods transfer depth from a pool of images with available depth maps to query image in parametric and non-parametric manners. However, these methods generally involve processing a large dataset, therefore are rather time-consuming. This paper proposes to speed up the whole implementation in a hierarchical way. First, feature extraction based methods are utilized to evaluate image similarities. Then, clustering methods are performed on the image dataset to partition the dataset into several groups. Finally, instead of searching the whole dataset, the query image only compares with each cluster's representative image and regards the most similar group as the final training dataset. Experiments show that the proposed method achieves significant speed up while keeping similar depth estimation performance compared with the state-of-the-art method.