Accurate localization, identification and segmentation of vertebrae is an important task in medical and biological image analysis. The prevailing approach to solve such a task is to first generate pixel-independent features for each vertebra, e.g. via a random forest predic-tor, which are then fed into an MRF-based objective to infer the optimal MAP solution of a constellation model. We abandon this static, two-stage approach and mix feature generation with model-based inference in a new, more flexible, way. We evaluate our method on two data sets with different objectives. The first is semantic segmentation of a 21-part body plan of zebrafish embryos in microscopy images, and the second is localization and identification of vertebrae in benchmark human CT.