Whole heart segmentation of 3D ultrasound (US), also referred to as echocardiography or simply echo, is useful in cardiac functional analysis to achieve quantitative diagnostic information of the heart. However, characteristics of US imaging such as limited field-of-view, artifacts and inconsistent intensity distribution makes automated approaches a challenge. In this paper, we present a framework for automatic whole heart segmentation from 3D echo. This work is motivated by the new technology of compounding 3D echo from 2D matrix array transducers. We propose to use the registration-based segmentation framework and adopt a new similarity measure combining local phase, intensity information and local geometry for registration. The experimental results demonstrated the proposed method had achieved an accuracy of 6.4% volume difference against the gold standard for the left ventricle segmentation and an average accuracy of 14% for segmentation of all four chambers and myocardium.