In this paper, a probabilistic approach to combining spatial context with visual and co-occurrence information for semantic image analysis is presented. Overall, the examined image is segmented and subsequently an initial classification of the resulting image regions to semantic concepts is performed based solely on visual information. Then, a Genetic Algorithm (GA) is introduced for deciding on the optimal semantic image interpretation, realizing image analysis as a global optimization problem. The fundamental novelty of this work is that the GA incorporates in its evolutionary procedure a set of Bayesian Networks (BNs), which probabilistically learn the impact of the available spatial, visual and co-occurrence information on the final outcome for every possible pair of semantic concepts. Experimental results on two publicly available datasets demonstrate the efficiency of the proposed approach.