Since a building's floorplan remains consistent over time and is inherently robust to changes in visual appearance, visual Floorplan Localization (FLoc) has received increasing attention from researchers. However, as a compact and minimalist representation of the building's layout, floorplans contain many repetitive structures (e.g., hallways and corners), thus easily result in ambiguous localization. Existing methods either pin their hopes on matching 2D structural cues in floorplans or rely on 3D geometry-constrained visual pre-trainings, ignoring the richer contextual information provided by visual images. In this paper, we suggest using broader visual scene context to empower FLoc algorithms with scene layout priors to eliminate localization uncertainty. In particular, we propose an unsupervised learning technique with clustering constraints to pre-train a room discriminator on self-collected unlabeled room images. Such a discriminator can empirically extract the hidden room type of the observed image and distinguish it from other room types. By injecting the scene context information summarized by the discriminator into an FLoc algorithm, the room style knowledge is effectively exploited to guide definite visual FLoc. We conducted sufficient comparative studies on two standard visual Floc benchmarks. Our experiments show that our approach outperforms state-of-the-art methods and achieves significant improvements in robustness and accuracy.

Object Target Search (OTS) tasks require robots to navigate to objects specified by semantic labels, (e.g., find a fire extinguisher). Many existing OTS methods strongly rely on semantic co-occurrence relations among objects to search and localize the object targets in a closed set. However, simplistic domestic or chaotic rescue scenes often fail to provide rich semantics, which in some cases even require the robot to find previously unseen object instances. In addition, most of the existing OTS methods adopt a fixed Field of View (FoV) setting relative to the robot base. When the limited FoV only allows the observation of incomplete objects, this can result in incorrect categorical information and lead to wrong navigation actions. To address the above issues, we propose an open-vocabulary object-seeking method named ChatSeek based on the Large Language Model (LLM)-informed Object Belief Field (OBF) and vehicle-arm cooperation. In particular, our method prompts LLM to generate target objects’ affordance and geometric-part attributes to enhance object localization. By projecting CLIP-based object recognition likelihoods into 3D reconstructions, the OBF is updated to maintain the robot’s cognition of the surrounding scene. During OTS, the robot achieves a flexible vision by moving the vehicle-mounted robotic arm intentionally to translate and rotate the robot’s FoV to look around or even inspect hidden corners. Sufficient comparative and ablation studies demonstrate that our method can significantly improve OTS performance. Furthermore, real-world experiments show that our approach can find novel objects without requiring semantic priors.

Today, robots are finding more and more applications in areas such as industrial and agricultural production, environmental exploration, and disaster relief. To be effective in these roles, robots must be able to autonomously navigate complex and unstructured environments. Manipulative robotic arms can navigate through confined spaces and have the ability to explore diverse scenes. Therefore, the exploration of unknown environments, facilitated by the synergy between robotic arms and wheeled chassis, provides a more efficient and accurate approach. We have designed a comprehensive arm-chassis collaboration system that uses an information gain-based utility function to support exploration. By coordinating the motion of the robotic arm with the mobility of the chassis, this system efficiently performs complex exploration tasks. It enables autonomous exploration of unknown environments with reduced time and energy consumption. The code is published at: https://github.com/Southyang/Arm-Chassis.