Aligning the decision-making process of deep learning models with that of experienced sonographers is essential for ultrasound-based reliable disease diagnosis. Although existing methods have made significant progress in this aspect, their alignments are primarily associational rather than causal, leading to pseudo-correlations between features and diagnostic results. Such a biased diagnosis blindly models the sonographer's diagnostic skills and attention to specific patterns, which we argue hardly produces an AI diagnoser that is comparable to human experts. To address this issue, we propose a causality-based diagnostic framework to align the model's diagnostic behaviors with those of experts. Specifically, by delving into both conspicuous and inconspicuous confounders within the ultrasound images, the back-door and front-door adjustment causal learning modules are proposed to promote unbiased learning by mitigating potential pseudo-correlations. In addition, we integrate causal inference into a well-designed dual-branch model with feature interaction bridges for compatibility with multimodal ultrasound inputs. To fully evaluate our method, we conduct comparative studies on different diseases and ultrasound modalities. In particular, we publish a carefully constructed multimodal ultrasound dataset for breast lesion diagnosis and segmentation. Sufficient comparative and ablation studies on this dataset emphasize that our method outperforms state-of-the-art methods.

Since a building's floorplan remains consistent over time and is inherently robust to changes in visual appearance, visual Floorplan Localization (FLoc) has received increasing attention from researchers. However, as a compact and minimalist representation of the building's layout, floorplans contain many repetitive structures (e.g., hallways and corners), thus easily result in ambiguous localization. Existing methods either pin their hopes on matching 2D structural cues in floorplans or rely on 3D geometry-constrained visual pre-trainings, ignoring the richer contextual information provided by visual images. In this paper, we suggest using broader visual scene context to empower FLoc algorithms with scene layout priors to eliminate localization uncertainty. In particular, we propose an unsupervised learning technique with clustering constraints to pre-train a room discriminator on self-collected unlabeled room images. Such a discriminator can empirically extract the hidden room type of the observed image and distinguish it from other room types. By injecting the scene context information summarized by the discriminator into an FLoc algorithm, the room style knowledge is effectively exploited to guide definite visual FLoc. We conducted sufficient comparative studies on two standard visual Floc benchmarks. Our experiments show that our approach outperforms state-of-the-art methods and achieves significant improvements in robustness and accuracy.

Object navigation (ObjcetNav), which enables an agent to seek any instance of an object category, has shown great advances. However, current agents are built upon occlusion-prone visual observations or compressed 2-D maps, which hinder their embodied perception of 3-D scene geometry. Furthermore, existing methods usually decouple ObjectNav into the exploration and exploitation subtasks, easily leading to ambiguous object localization and blind exploration. To address these issues, we first propose an embodied contrastive learning (ECL) method with geometric consistency (GC) and behavioral awareness (BA), which motivates agents to encode 3-D scene layouts and semantic cues actively. The BA is modeled by predicting navigational actions based on multiframe visual images, as behaviors causing differences between adjacent visual sensations are crucial for learning correlations among continuous visions. The GC is modeled by aligning the behavior-aware visual stimulus with 3-D semantic shapes through unsupervised contrastive learning. Then, based on the above ECL pretraining, a coarse-to-fine ObjectNav policy with explorer and discriminator cooperation is proposed, inspired by the treasure-hunting mindset. Concretely, the explorer is designed to adaptively switch the action spaces, thereby switching the global and local exploration thoughts according to the accumulated scene priors. The discriminator is designed to discriminate the target's authenticity using behavior-aware visual features and geometric invariance priors, which permits mimicking the human behavior of "approaching to confirm" when distinguishing objects from a distance. As expected, our ECL method performs well on object detection (ObjDet) and instance segmentation (InstSeg) tasks. Our ECL-enhanced ObjectNav strategy outperforms state-of-the-art (SOTA) methods on Matterport3D (MP3D), Gibson, and HM3D datasets.

Object Target Search (OTS) tasks require robots to navigate to objects specified by semantic labels, (e.g., find a fire extinguisher). Many existing OTS methods strongly rely on semantic co-occurrence relations among objects to search and localize the object targets in a closed set. However, simplistic domestic or chaotic rescue scenes often fail to provide rich semantics, which in some cases even require the robot to find previously unseen object instances. In addition, most of the existing OTS methods adopt a fixed Field of View (FoV) setting relative to the robot base. When the limited FoV only allows the observation of incomplete objects, this can result in incorrect categorical information and lead to wrong navigation actions. To address the above issues, we propose an open-vocabulary object-seeking method named ChatSeek based on the Large Language Model (LLM)-informed Object Belief Field (OBF) and vehicle-arm cooperation. In particular, our method prompts LLM to generate target objects’ affordance and geometric-part attributes to enhance object localization. By projecting CLIP-based object recognition likelihoods into 3D reconstructions, the OBF is updated to maintain the robot’s cognition of the surrounding scene. During OTS, the robot achieves a flexible vision by moving the vehicle-mounted robotic arm intentionally to translate and rotate the robot’s FoV to look around or even inspect hidden corners. Sufficient comparative and ablation studies demonstrate that our method can significantly improve OTS performance. Furthermore, real-world experiments show that our approach can find novel objects without requiring semantic priors.

Active 3D reconstruction has a wide range of applications, including augmented/virtual reality and various robotics tasks such as navigation, object recognition and manipulation, and scene perception. Most existing approaches rely on offline images or online streams captured by human-operated cameras, requiring significant manual effort and depending on the operator’s expertise. Although some methods have explored using robots to assist data collection by planning the Next-Best-View and then moving the camera to cover these viewpoints, they are often constrained by fixed camera perspectives and suboptimal robot control strategies. These limitations result in inaccurate camera positioning and inaccessible viewpoints, leading to ineffective data collection. In this paper, we explore the feasibility of using a vehicle-arm collaborative robot to tackle the challenges of active 3D reconstruction. Specifically, we decompose this task into two mutually iterative subtasks: the object-centric planning task, which focuses solely on the objects to generate candidate viewpoints, and the agent-centric interaction task, where the robot moves the camera to cover these viewpoints. To achieve this, we propose a collaborative control framework that integrates the motions of the base and arm, enabling efficient recovery of the 3D shape of objects. Sufficient comparative and ablation studies validate that our approach achieves finer surface reconstruction with reduced reconstruction time.

Since a building's floorplans are easily accessible, consistent over time, and inherently robust to changes in visual appearance, self-localization within the floorplan has attracted researchers' interest. However, since floorplans are minimalist representations of a building's structure, modal and geometric differences between visual perceptions and floorplans pose challenges to this task. While existing methods cleverly utilize 2D geometric features and pose filters to achieve promising performance, they fail to address the localization errors caused by frequent visual changes and view occlusions due to variously shaped 3D objects. To tackle these issues, this paper views the 2D Floorplan Localization (FLoc) problem from a higher dimension by injecting 3D geometric priors into the visual FLoc algorithm. For the 3D geometric prior modeling, we first model geometrically aware view invariance using multi-view constraints, i.e., leveraging imaging geometric principles to provide matching constraints between multiple images that see the same points. Then, we further model the view-scene aligned geometric priors, enhancing the cross-modal geometry-color correspondences by associating the scene's surface reconstruction with the RGB frames of the sequence. Both 3D priors are modeled through self-supervised contrastive learning, thus no additional geometric or semantic annotations are required. These 3D priors summarized in extensive realistic scenes bridge the modal gap while improving localization success without increasing the computational burden on the FLoc algorithm. Sufficient comparative studies demonstrate that our method significantly outperforms state-of-the-art methods and substantially boosts the FLoc accuracy.

Object Goal Navigation (ObjectNav) in novel environments relies on comprehensive scene understanding, including precise visual perception and accurate modeling of spatial-semantic regularities. However, excessive attention to the hand-crafted scene representation in prevailing approaches leads to the neglect of the negative influence of the perception bias hidden in the visual observations. The hand-crafted semantic distribution in domestic environments causes the spurious association bias, while the semantic conflict bias arises due to the dynamic perspective changes. Biased visual perception significantly limits the generalization of the navigation strategy. In this article, we propose the U nbiased E mbodied V isual R epresentation ( UEVR ), which overcomes the perception biases using causal inference and cross-modality alignment. Specifically, we establish reasonable assumptions about confounders for multi-object features through our proposed Unbiased Causal R-CNN framework and eliminate the spurious associations bias through B ackdoor I ntervention C ausal A djustment ( BICA ) module during navigation. To overcome the dynamic-view bias hidden in 2D image features, we propose to employ the cross-modality alignment mechanism with the Geometric Consistency ( GeoCon ) to encode 3D geometry prior into the 2D representations. Finally, we design a modular ObjectNav framework integrated with UEVR named Causal-ObjectNav , which consists of the corner-based scene exploration module and target object discrimination module. Extensive experiments on the MP3D and HM3D datasets demonstrate the superiority of the unbiased navigation model over existing ObjectNav methods.

Today, robots are finding more and more applications in areas such as industrial and agricultural production, environmental exploration, and disaster relief. To be effective in these roles, robots must be able to autonomously navigate complex and unstructured environments. Manipulative robotic arms can navigate through confined spaces and have the ability to explore diverse scenes. Therefore, the exploration of unknown environments, facilitated by the synergy between robotic arms and wheeled chassis, provides a more efficient and accurate approach. We have designed a comprehensive arm-chassis collaboration system that uses an information gain-based utility function to support exploration. By coordinating the motion of the robotic arm with the mobility of the chassis, this system efficiently performs complex exploration tasks. It enables autonomous exploration of unknown environments with reduced time and energy consumption. The code is published at: https://github.com/Southyang/Arm-Chassis.

Visual Semantic Navigation (VSN) aims at navigating a robot to a given target object in a previously unseen scene. To tackle this task, the robot must learn a nimble navigation policy by utilizing spatial patterns and semantic co-occurrence relations among objects in the scene. Prevailing approaches extract scene priors from the instant visual observations and solidify them in neural episodic memory to achieve flexible navigation. However, due to the oblivion and underuse of the scene priors, these methods are plagued by repeated exploration, effective-knowledge sparsity, and wrong decisions. To alleviate these issues, we propose a novel VSN policy, HSPNav, based on Hierarchical Scene Priors (HSP) and Deep Reinforcement Learning (DRL). The HSP contains two components, i.e., the egocentric semantic map-based Local Scene Priors (LSP) and the commonsense relational graph-based Global Scene Priors (GSP). Then, efficient semantic navigation is achieved by employing an immediate LSP to retrieve conducive contextual memories from the GSP. By utilizing the MP3D dataset, the experimental results in the Habitat simulator demonstrate that our HSP brings a significant boost over the baselines. Furthermore, we take an essential step from simulation to reality by bridging the gap from Habitat to ROS. The migration evaluations show that HSPNav can generalize to realistic settings well and achieve promising performance.

Social navigation is a key consideration for integrating robots into human environments. Concurrently, it imposes heightened requisites: tasks must not only be executed succesfully without collisions, but also adhere to principles encompassing comprehensibility, courtesy, social compliance, comprehension, foresight, and scenario compliance. In this paper, we present the incorporation of social norms as a guiding framework for robot navigation within social contexts. We adopt field theory to provide a formal elucidation of the social norms, using Physical-Informed Neural Network (PINN) to predict pedestrian movement under the influence of social norms, respectively, and using Reinforcement Learning (RL) for navigation. We use supervised learning to train the pedestrian velocity field prediction model and reinforcement learning to train the navigation policy. We conduct three parts of experiments: (1) analyzing the spatiotemporal characteristics of the velocity field in the walking pedestrians dataset; (2) evaluating the accuracy of the vector field prediction in the pedestrian dataset; (3) using Gazebo simulation and the PEDSIM library to evaluate the improvement of navigation performance under constraints of social norms. Experiments have confirmed that the pedestrian motion data set indeed satisfies the Gaussian divergence theorem and can be described by the concept of field. The performance of navigation strategies incorporating social rules has been improved to a certain extent.

Object Navigation (ObjcetNav), which enables an agent to seek any instance of an object category specified by a semantic label, has shown great advances. However, current agents are built upon occlusion-prone visual observations or compressed 2D semantic maps, which hinder their embodied perception of 3D scene geometry and easily lead to ambiguous object localization and blind exploration. To address these limitations, we present an Embodied Contrastive Learning (ECL) method with Geometric Consistency (GC) and Behavioral Awareness (BA), which motivates agents to actively encode 3D scene layouts and semantic cues. Driven by our embodied exploration strategy, BA is modeled by predicting navigational actions based on multi-frame visual images, as behaviors that cause differences between adjacent visual sensations are crucial for learning correlations among continuous visions. The GC is modeled as the alignment of behavior-aware visual stimulus with 3D semantic shapes by employing unsupervised contrastive learning. The aligned behavior-aware visual features and geometric invariance priors are injected into a modular ObjectNav framework to enhance object recognition and exploration capabilities. As expected, our ECL method performs well on object detection and instance segmentation tasks. Our ObjectNav strategy outperforms state-of-the-art methods on MP3D and Gibson datasets, showing the potential of our ECL in embodied navigation.

In recent years, since edge computing has improved the performance of transportation systems, research on edge-computing-enabled transportation systems has received widespread attention. However, most previous studies overlooked that task requests in transportation systems are unevenly distributed in time and space, which easily causes the overloading of edge servers, resulting in high response latency. To this end, we present a novel task offloading scheme based on graph neural network (GNN) and deep reinforcement learning (DRL) in edge-computing-enabled transportation systems (TransEdge). Specifically, we first propose an adaptive node placement algorithm to assign Internet of Things sensors to appropriate edge servers, thereby minimizing transmission latency. Then, an improved DRL scheme based on GNN is designed to capture the spatial features between sensors, aiming to improve the accuracy of task offloading decisions. Finally, we introduce a task forwarding strategy based on the greedy algorithm to achieve collaborative task offloading between different edge servers and overcome the system instability caused by a sudden surge in task requests. We conduct extensive experiments on two real-world traffic data sets. The results show that TransEdge reduces the response latency by at least 3.7% compared to four baselines while achieving a success rate of 99%.

Recently, the Industrial Internet of Things (IIoT) has shown great application value in environmental monitoring. However, it suffers from serious bottlenecks in energy and computing capability. To address them, researchers have made lots of effort. Nevertheless, they neglect either the edge–end collaboration or the impact of task queue backlog, resulting in low system revenue. To this end, we design a queue-aware computation offloading method based on DRL (QDRL). Specifically, we represent the long-term system operation as a multistage stochastic mixed-integer optimization problem (M-SMIP), which is further converted into a deterministic problem using Lyapunov optimization. Given that the resource allocation and computation offloading in this deterministic problem are strongly coupled and difficult to solve, we decompose this problem into two subproblems. Subsequently, a reinforcement learning scheme with actor–critic architecture is designed to solve these subproblems. The Actor module is designed based on a deep learning model and quantization strategy for generating computation offloading actions. The mathematical reasoning and learning-based methods are integrated as the Critic module for achieving resource allocation. Extensive simulation results show that the performance of QDRL surpasses four baselines and approaches the approximate optimal algorithm in terms of average task queue length, normalized real computation rate, and computation time.

The Object goal Navigation (ObjectNav) task requires an agent to navigate through a previously unknown domestic scenario using spatial and semantic contextual information, where the goal is specified by a semantic label (e.g., find a TV). Such a task is especially challenging as it requires formulating and understanding the complex co-occurrence relations among objects in diverse settings, which is critical for long-sequence navigational decision-making. Existing methods learn to either explicitly represent co-occurrence relationships as discrete semantic priors, or implicitly encode them from raw observations, thus can not benefit from the rich environmental semantics. In this work, we propose a novel Deep Reinforcement Learning (DRL) based ObjectNav strategy by actively imagining spatial and semantic clues outside the agent’s Field of View (FoV) and further mining Continuous Environmental Representations (CER) using self-supervised learning. Additionally, the illusion of spatial and semantic patterns allows the agent to perform Multi-Step Forward-Looking Planning (MSFLP) by considering the temporal evolution of egocentric local observations. Our approach is thoroughly evaluated and ablated in the visually realistic environments of the Matterport3D (MP3D) dataset. The experimental results reflect that our method combining CER and imagination-based MSFLP facilitates learning complicated semantic priors and navigation skills, thus achieving state-of-the-art performance on the ObjectNav task. In addition, adequate quantitative and qualitative analyses validate the excellent generalization ability and superiority of our method.

The autonomous exploration task we consider requires Unmanned Aerial Vehicles (UAVs) to actively navigate through unknown environments with the goal of fully perceiving and mapping the environments. Some existing exploration strategies suffer from rough cost budgets, ambiguous Information Gain (IG), and unnecessary backtracking exploration caused by Fragmented Regions (FRs). In our work, a hierarchical spatio-temporal-aware exploration framework is proposed to alleviate these problems. At the local exploration level, the Asymmetrical Traveling Salesman Problem (ATSP) is solved by comprehensively considering exploration time, IG, and heading consistency to avoid blindly exploring. Specifically, the exploration time is reasonably budgeted by fast marching in an artificial potential field. Meanwhile, a transformer-based map occupancy predictor is designed to assist in IG calculation by imagining spatial clues out of the Field of View (FoV), facilitating the prescient exploration. We verify that our local exploration is effective in alleviating the unnecessary back-and-forth movements caused by FRs and the interference of potential obstacle occlusion on the IG calculation. At the global exploration level, the classical Next Best View Points (NBVP) are generalized to Next Best Sub-Regions (NBSR) to choose informative sub-regions for further forward-looking exploration based on a well-designed utility function. Safe flight paths and dynamically feasible trajectories are reasonably generated throughout the exploration process by fast marching and B-spline curve optimization. Comparative simulations and benchmark tests demonstrate that our proposed exploration strategy is quite competitive in terms of exploration path length, total exploration time, and exploration ratio.