Automatic ship detection in complicated maritime background is a challenging task in the field of optical remote sensing image interpretation and analysis. In this paper, we propose a novel and reliable ship detection framework based on a visual saliency model, which can efficiently detect multiple targets of different scales in complex scenes with sea clutter, clouds, wake and islands interferences. Firstly, we present a reliable background prior extraction method adaptive for the random locations of targets by computing boundary probability and then generate a saliency map based on the background prior. Secondly, we compute the prior probability of salient foreground regions and propose a weighting function to constrain false foreground clutter, gaining the foreground-based prediction map. Thirdly, we integrate the two prediction maps and improve the details of the integrated map by a guided filter function and a wake adjustment function, obtaining the fine selection of candidate regions. Afterwards, a classification is further performed to reduce false alarms and produce the final ship detection results. Qualitative and quantitative evaluations on two public available datasets demonstrate the robustness and efficiency of the proposed method against four advanced baseline methods.