Open World Object Detection (OWOD) addresses realistic scenarios where unseen object classes emerge, enabling detectors trained on known classes to detect unknown objects and incrementally incorporate the knowledge they provide. While existing OWOD methods primarily focus on detecting unknown objects, they often overlook the rich semantic relationships between detected objects, which are essential for scene understanding and applications in open-world environments (e.g., open-world tracking and novel class discovery). In this paper, we extend the OWOD framework to jointly detect unknown objects and learn semantically rich instance embeddings, enabling the detector to capture fine-grained semantic relationships between instances. To this end, we propose modules that leverage the rich and generalizable knowledge of Vision Foundation Models (VFMs) and can be integrated into open-world object detectors. First, the Unknown Box Refine Module uses semantic masks from the Segment Anything Model to accurately localize unknown objects. The Embedding Transfer Module then distills instance-wise semantic similarities from VFM features to the detector’s embeddings via a relaxed contrastive loss, enabling the detector to learn a semantically meaningful and generalizable instance feature. Extensive experiments show that our method significantly improves both unknown object detection and instance embedding quality, while also enhancing performance in downstream tasks such as open-world tracking.

Despite the success of deep learning-based object detection methods in recent years, it is still challenging to make the object detector reliable in adverse weather conditions such as rain and snow. For the robust performance of object detectors, unsupervised domain adaptation has been utilized to adapt the detection network trained on clear weather images to adverse weather images. While previous methods do not explicitly address weather corruption during adaptation, the domain gap between clear and adverse weather can be decomposed into two factors with distinct characteristics: a style gap and a weather gap. In this paper, we present an unsupervised domain adaptation framework for object detection that can more effectively adapt to real-world environments with adverse weather conditions by addressing these two gaps separately. Our method resolves the style gap by concentrating on style-related information of high-level features using an attention module. Using self-supervised contrastive learning, our framework then reduces the weather gap and acquires instance features that are robust to weather corruption. Extensive experiments demonstrate that our method outperforms other methods for object detection in adverse weather conditions.

Robotic mapping with Bayesian Kernel Inference (BKI) has shown promise in creating semantic maps by effectively leveraging local spatial information. However, existing semantic mapping methods face challenges in constructing reliable maps in unstructured outdoor scenarios due to unreliable semantic predictions. To address this issue, we propose an evidential semantic mapping, which can enhance reliability in perceptually challenging off-road environments. We integrate Evidential Deep Learning into the semantic segmentation network to obtain the uncertainty estimate of semantic prediction. Subsequently, this semantic uncertainty is incorporated into an uncertainty-aware BKI, tailored to prioritize more confident semantic predictions when accumulating semantic information. By adaptively handling semantic uncertainties, the proposed framework constructs robust representations of the surroundings even in previously unseen environments. Comprehensive experiments across various off-road datasets demonstrate that our framework enhances accuracy and robustness, consistently outperforming existing methods in scenes with high perceptual uncertainties.

Estimating the traversability of rough terrain is a critical task for an outdoor mobile robot. While classifying structured environment can be learned from large number of training data, it is an extremely difficult task to learn and estimate traversability of unstructured rough terrain. Moreover, in many cases information from a single sensor may not be sufficient for estimating traversability reliably in the absence of artificial landmarks such as lane markings or curbs. Our approach estimates traversability of the terrain and build a 2D probabilistic grid map online using 3D-LIDAR and camera. The combination of LIDAR and camera is favoured in many robotic application because they provide complementary information. Our approach assumes the data captured by these two sensors are independent and build separate traversability maps, each with information captured from one sensor. Traversability estimation with vision sensor autonomously collects training data and update classifier without human intervention as the vehicle traverse the terrain. Traversability estimation with 3D-LIDAR measures the slopes of the ground to predict the traversability. Two independently built probabilistic maps are fused using Bayes' rule to improve the detection performance. This is in contrast with other methods in which each sensor performs different tasks. We have implemented the algorithm on a UGV(Unmanned Ground Vehicle) and tested our approach on a rough terrain to evaluate the detection performance.

In this paper we propose a hybrid vision-based SLAM and moving objects tracking (vSLAMMOT) approach. This approach tightly combines two key methods: a superpixel-based segmentation to detect moving objects and a Rao-Blackwellized Particle Filter to estimate a stereo-vision-based SLAM posterior. Most successful methods perform vision-based SLAM (vSLAM) and track moving objects independently. However, we pose both vSLAM and moving object tracking as a single correlated problem to leverage the performance. Our approach estimates the relative camera motion using the previous tracking result, and then detects moving objects from the estimated camera motion recursively. Moving superpixels are detected by a Markov Random Field (MRF) model which uses spatial and temporal information of the moving objects. We demonstrate the performance of the proposed approach for vSLAMMOT using both synthetic and real datasets and compare the performance with other methods.

In this paper, we propose a Markov Chain Monte Carlo (MCMC) sampling method with the data-driven proposal distribution for six-degree-of-freedom (6-DoF) SLAM. Recently, visual odometry priors have been widely used as the process model in the SLAM formulation to improve the SLAM performance. However, modeling the uncertainties of incremental motions estimated by visual odometry is especially difficult under challenging conditions, such as erratic motion. For a particle-based model representation, it can represent the uncertainty of the camera motion well under erratic motion compared to the constant velocity model or a Gaussian noise model, but the manner of representing the proposal distribution and sampling the particles is extremely important, as we can maintain only a limited number of particles in the high-dimensional state space. Hence, we propose an effective sampling approach by exploiting MCMC sampling and the data-driven proposal distribution to propagate the particles. We demonstrate the performance of the proposed approach for 6-DoF SLAM using both synthetic and real datasets and compare the performance with those of other sampling methods.

One fundamental problem of object tracking is the convergence of estimates to local maxima not corresponding to target objects. To mitigate this problem, constructing a good posterior distribution of the target state is important. In this letter, we propose a robust tracking approach by building a new posterior distribution model from multiple independent estimates of a target state. For each candidate of the target state, we compute a confidence score based on its spatial consistency with other estimates and photometric similarities with target models. Our posterior distribution model reflects tracking uncertainties well and adaptively defines the search region for the next frame. We validate the robustness of our approach on a number of challenging datasets.

We present a method for simultaneously locking on to a ground target and estimating the position of an unmanned aerial vehicle (UAV) under countermeasure (CM) conditions, where sensors are prevented from successfully tracking a target. Owing to the limited payload and power of the UAVs, we employ a monocular camera and a global positioning system (GPS) to carry out vision-based simultaneous localization and mapping (SLAM) using both an unscented Kalman filter and a Kalman filter. Since this approach estimates the state of the UAV and the location of the target, we can estimate the position of the target in the image, even in the presence of CMs. Our experiments show that the proposed method successfully locks on to the target and estimates the state of the UAV.

Various image restoration methods have been studied for removing space-variant motion blur such as iterative and POCS (projection on to convex sets) method. However, the computational complexity of the methods, such as regularized iteration and POCS method, is so high that they can hardly be implemented in real-time. We address a method to reduce the computational complexity by selecting the region to be restored. The primary application area of the proposed method is a surveillance system which requires accurate object extraction, identification and tracking functions. To remove motion blur, we propose a new spatially adaptive regularized iterative image restoration algorithm. Experimental results show the the proposed algorithm can efficiently remove space-variant motion blur with significantly reduced computational overhead.