We present an approach for localization and semantic mapping in ambiguous scenarios by incrementally maintaining a hybrid belief over continuous states and discrete classification and data association variables. Unlike existing incremental approaches, we explicitly maintain data association components over time, which allows us to deal with perceptual aliasing. Crucially, we utilize a viewpoint-dependent classifier model over rich classifier outputs and leverage the coupling between poses and semantic measurements both for disambiguating data association and in pose estimation. We demonstrate in simulation that incorporating semantic measurements with a viewpoint-dependent classifier model enhances disambiguation of both data association and localization over usage of only geometric measurements or viewpoint independent models, further contributing to the tractability of the approach in practice, and providing better estimates.

We propose an algorithm for robust visual classification of an object of interest observed from multiple views using a black-box Bayesian classifier which provides a measure of uncertainty, in the presence of significant ambiguity and classifier noise, and of localization error. The fusion of classifier outputs takes into account viewpoint dependency and spatial correlation among observations, as well as pose uncertainty when these observations are taken and a measure of confidence provided by the classifier itself. Our experiments confirm an improvement in robustness over state-of-the-art.