This paper describes an approach to image retrieval based on the underlying semantics of images. To extract these semantics a hierarchical, probabilistic approach is proposed. The labels that are extracted in this case are man-made, natural, inside and outside. The hierarchical framework combines class likelihood probability estimates across a number of levels to form a posterior estimate of the probability of class membership. Unlike previous work in this field, the proposed algorithm can determine probabilities at any point in the scene and only a small number of images are required to train the system. To illustrate the potential of such an approach a prototype image retrieval system has been developed, initial results from this system are given in this paper.