3D scene flow represents the dense per-point motion field in dynamic scenes, playing a crucial role in various downstream tasks, including motion segmentation, dynamic scene reconstruction, 4D content generation, etc. However, previous regression-based works commonly suffer from unreliable correlations caused by locally constrained search ranges and struggle with the absence of timely feedback regarding the flow estimation uncertainty during training. To address these challenges, we propose a novel uncertainty-aware network for scene flow estimation, termed DifFlow3D, based on the conditional probabilistic diffusion model. Hierarchical diffusion-based flow estimation blocks are designed to enhance the correlation robustness and resilience to challenging cases, e.g., dynamics, noisy inputs, repetitive patterns, etc. To mitigate the generation diversity, three key flow-related features are leveraged as conditions in our diffusion model. Furthermore, we develop an uncertainty estimation module within diffusion to assess the reliability of estimated scene flow dynamically. A Hidden State Denoising strategy (HSD) is also introduced to further boost the stability of the reverse denoising process. Extensive experiments conducted on four scene flow datasets, including both synthetic and real-world datasets (FlyingThings3D, KITTI 2015, Argoverse, and Waymo Open), demonstrate the superiority of our proposed DifFlow3D. Compared to prior state-of-the-art methods, DifFlow3D has 26.0%, 36.4%, 35.3%, and 17.7% EPE3D reduction respectively across four datasets. Only trained on the synthetic FlyingThings3D dataset, our method achieves an unprecedented millimeter-level accuracy (0.0070 m EPE3D) on the real-scene KITTI dataset, highlighting its exceptional generalization capability. Additionally, our diffusion-based refinement paradigm can be seamlessly integrated as a plug-and-play module into existing scene flow networks, significantly enhancing their estimation accuracy. We also introduce our pre-trained scene flow estimator as explicit motion priors into the novel dynamic LiDAR view synthesis task, which validates its great potential for improving the 4D LiDAR reconstruction performance.

Understanding human intentions and actions through egocentric videos is important on the path to embodied artificial intelligence. As a branch of egocentric vision techniques, hand trajectory prediction plays a vital role in comprehending human motion patterns, benefiting downstream tasks in extended reality and robot manipulation. However, capturing high-level human intentions consistent with reasonable temporal causality is challenging when only egocentric videos are available. This difficulty is exacerbated under camera egomotion interference and the absence of affordance labels to explicitly guide the optimization of hand waypoint distribution. In this work, we propose a novel hand trajectory prediction method dubbed MADiff, which forecasts future hand waypoints with diffusion models. The devised denoising operation in the latent space is achieved by our proposed motion-aware Mamba, where the camera wearer's egomotion is integrated to achieve motion-driven selective scan (MDSS). To discern the relationship between hands and scenarios without explicit affordance supervision, we leverage a foundation model that fuses visual and language features to capture high-level semantics from video clips. Comprehensive experiments conducted on five public datasets with the existing and our new evaluation metrics demonstrate that MADiff predicts comparably reasonable hand trajectories compared to the state-of-the-art baselines.

We propose SNI-SLAM++, a tightly-coupled semantic SLAM system utilizing neural implicit representation, that simultaneously performs accurate semantic mapping, high-quality surface reconstruction, and robust camera tracking. Our system tightly integrates visual appearance, geometry, and semantics through five key components: (i) We introduce hierarchical semantic representation to allow multi-level semantic comprehension for top-down structured semantic mapping of the scene. (ii) To fully utilize the correlation between multiple attributes of the environment, we integrate appearance, geometry and semantic features through cross-attention for feature collaboration. This strategy enables a more multifaceted understanding of the environment, thereby allowing SNI-SLAM++ to remain robust even when single attribute is defective. (iii) We design an internal fusion-based decoder to obtain semantic, RGB, and Truncated Signed Distance Field (TSDF) values from multi-level features for accurate decoding. (iv) We introduce a semantics-coupled tracking framework that tightly incorporates semantic constraints for camera pose estimation in neural implicit SLAM. This framework leverages the multi-view consistency of semantics to construct a pose graph and perform semantic loop closure optimization, enabling robust tracking. (v) We propose a feature loss to update the scene representation at the feature level. Compared with low-level losses such as RGB loss and depth loss, our feature loss is capable of guiding the network optimization on a higher level. Our SNI-SLAM++ demonstrates superior performance over all recent visual SLAM methods in terms of mapping and tracking accuracy on the datasets of Replica, ScanNet, TUM-RGBD, and ScanNet++, while also showing excellent capabilities in accurate semantic segmentation and 3D semantic mapping.

Physical human-robot interaction (pHRI) systems can combine the robot’s precise control ability with the human’s decision-making ability. However, the robot’s inability to fully understand human intentions may inevitably cause disagreements during the interaction process. This work develops a novel human-robot shared strategy to reduce human-robot disagreements and enhance the robot assistance level. We present a diffusion model to predict human intention using as inputs the past trajectories, the past human forces, the past robot forces and task environment information. Considering the uncertainty of human behavior and prediction model, we model the pHRI system as a Linear Quadratic Gaussian (LQG) stochastic system, incorporating the uncertainty into the cost function of the pHRI system. Finally, we propose an adaptive impedance control strategy based on human-robot disagreements to ensure that the robot can assist human movement to the greatest extent. Experiments are conducted on a Sawyer robot to verify the effectiveness of the proposed method. The results show that the proposed strategy enables the robot to adapt to the human based on human-robot disagreements and enhance the assistance level.

Recently, Shuttle-Based Storage and Retrieval Systems (SBS/RSs) have emerged as a cornerstone of modern robotic handling systems. However, motion conflicts between four-way shuttle vehicles are introduced frequently due to the sparsely connected roadmap topology. Furthermore, the different time and energy costs between linear and direction-changing motions bring additional difficulties for classic solvers when applied directly. To address these issues, we propose Dynamic Graph-Driven A* (DGA*) for cooperative path planning in SBS/RSs. By fully exploiting the structured topology of SBS/RS roadmaps, the algorithm extracts sector-level topological graphs enriched with spatiotemporal attributes, effectively resolving conflicts while simultaneously optimizing motion costs. In addition, Recursive Preemption and Avoidance (RPA) is introduced to ensure motion coordination in congestion-prone areas commonly seen in SBS/RSs, through a combination of independent avoidance and recursive avoidance mechanisms. We implement our approach in a real warehouse with more than 240 vehicles and achieve a 17% improvement in system throughput while reducing runtime.

Open-vocabulary grasping is essential for enabling robots to operate robustly in open and dynamic real-world environments. Current methods typically focus on predicting grasp poses from fused cross-modal features. The coupling of visual features and textual semantics in the feature space results in scene-level representations. However, such representations lack object-level discrimination, thereby limiting the model’s ability to recognize and generalize to novel targets. Moreover, existing positional encoding methods employed in cross-modal modeling lack the capacity to effectively capture spatial relationships between regions, which limits the model’s spatial reasoning performance. To tackle these challenges, we propose a method that combines Structured Semantic Disentanglement with Attention Diffusion (SDAD) for open-vocabulary robotic grasping. Specifically, to improve the semantic disentanglement in cross-modal features, a language-guided Disentanglement strategy that regularizes the feature space is proposed to disentangle target semantics features and conditional semantics features. To further improve spatial context modeling, we introduce an attention diffusion mechanism inspired by Fick’s law, which describes natural diffusion driven by concentration gradients, enabling attention to propagate smoothly from condition-feature-anchored regions across the scene. The target features are subsequently used to complete the matching and localization of the corresponding object regions. Extensive quantitative and qualitative experiments demonstrate that our method outperforms existing approaches in open-vocabulary grasping tasks. Moreover, in real-world scenarios, the proposed model achieves a grasp success rate of 84% on base categories and 66% on novel categories, outperforming the state-of-the-art method GLIPv2 by 7% and 5%, respectively.

3D Gaussian Splatting (3DGS) has emerged as a powerful scene representation, offering geometrically dense and photometrically accurate modeling capabilities that present a promising new paradigm for accurate targetless sensor calibration. Current 3DGS-based LiDAR-camera calibration methods usually highly rely on the joint optimization of the Gaussian model and extrinsics, and thus suffer two critical limitations. On the one hand, the global optimization is usually sensitive to the accumulated localization error of LiDAR. On the other hand, the inaccurate extrinsics may cause oscillations during the joint optimization. Specifically, there is a fundamental dilemma: accurate extrinsic calibration requires accurate scene models, while constructing accurate models itself depends on accurate extrinsics. This dilemma frequently triggers oscillatory optimization trajectories, significantly increasing vulnerability to premature convergence at suboptimal states. To address these challenges, we propose HiGS-Calib, a novel 3DGS calibration pipeline featuring the integration of our proposed Local-Consistent Photometric-Geometric (LCPG) error model and the hierarchical architecture. The LCPG error leverages the spatial consistency within local windows to quantify pose misalignment using only geometric attributes of the 3DGS model, bypassing color-pose reliance constraints. Besides, diverging from joint optimization paradigms, HiGS-Calib implements coarse-to-fine iterative optimization, decoupling scene modeling from extrinsic refinement and thereby achieving stable and accurate calibration. Extensive evaluation demonstrates the significantly improved calibration accuracy and stability of our HiGS-Calib over other state-of-the-art methods. To make our results reproducible, the source code has been released at https://github.com/IRMVLab/HiGS-Calib.

Vision-language navigation (VLN) requires an agent to follow visual observations and language instructions to navigate within a 3D environment. Most prior VLN approaches utilize recurrent units, topological graphs, or grid maps to represent the agent’s previously explored environment. However, these methods face two key limitations: (1) difficulty in accurately and adequately representing multi-level spatial information, and (2) a lack of robustness in semantic feature extraction against viewpoint variations. To overcome these challenges, we design a novel spatial-aware scene representation (SSR) as well as a viewpoint-robust feature extraction method. Specifically, for SSR, we first construct a multi-layer map that models the scene at different granularities, incorporating waypoints, objects, rooms, and floors. We then propose to extract spatial features from this multi-layer map based on a heterogeneous graph transformer and align them with the instructions, effectively guiding the agent’s planning process. As to viewpoint-robust feature extraction, we integrate 3D Gaussian splatting to capture the 3D geometry and visual texture of the environment, enabling novel view object observation and feature extraction. This improves observation coverage and enhances the viewpoint robustness of feature extraction. Extensive experiments demonstrated that our approach achieves state-of-the-art performance on two VLN benchmarks and shows strong performance in real-world scenarios. Source codes will be publicly available upon paper acceptance.

3-D hand pose estimation plays a crucial role in various applications, including virtual interaction, augmented reality, and human–robot collaboration. However, the complex articulation of hand joints and frequent self-occlusions make this task highly challenging. Existing methods based on convolutional or graph architectures are effective for local modeling, but often struggle to jointly capture global context and structured joint dependencies. To address this limitation, we proposeHandMLP, a novel framework that integrates multilayer perceptron (MLP)-based global modeling with graph attention for structured joint reasoning. Specifically, the MLP module enhances global semantic representation and nonlinear fitting capacity, while the graph attention module explicitly models local structural relationships and dynamic dependencies among hand joints. In addition, we introduce a voting-based initialization strategy to generate coarse pose hypotheses, which are progressively refined through collaborative reasoning, improving robustness under sparse or noisy point clouds. Extensive experiments on three benchmark datasets demonstrate thatHandMLPconsistently outperforms state-of-the-art methods, achieving superior accuracy and robustness in 3-D hand pose estimation.

With the rapid development of multimodal data fusion based on deep learning, semantic segmentation using the complementary information from the camera and LiDAR sensor can enhance the environmental perception of autonomous driving. However, most fusion methods overlook critical issues arising during the fusion process, including: first, insufficient fusion effectiveness in one-to-one scenarios due to a lack of precise feature alignment; second, low efficiency in the fusion process and a large computational burden on the model. To address these issues, we propose a trans-modal fusion method based on state space model and deformable transformer module. The proposed trans-modal fusion method makes use of the 2-D prior information of the image, such as texture and appearance information. By distilling knowledge from fused 2D–3D representations, it significantly enhances the feature extraction capability of the 3-D point cloud. First, we design a lightweight global gap adjustment attention module based on a linear state-space model, named global gap attention, to better bridge the information gap between point clouds and images. Then, to enhance the efficiency of trans-modal fusion, we adopt a local cross fusion module, which utilizes a many-to-many fusion approach to achieve more accurate feature alignment. Finally, to strengthen the channel information of point cloud features, we introduce an efficient channel attention during the feature extraction process. The experimental results show that the proposed method achieves a better segmentation accuracy compared with baseline. We show superiority of our method on two popular urban datasets SemanticKITTI and nuScenes (achieving mean Intersection over Union (mIoU) scores of 70.9% and 80.4%, respectively).

Generating dynamic scenes from images has gained increasing attention. Existing methods have two major limitations: 1) they can hardly handle sparse images which exhibit limited geometry constraints and insufficient motion; 2) they struggle to maintain spatial-temporal consistency when rendering multi-view videos. To address these limitations, we propose SCSV, a spatial-temporal consistent dynamic scene generation method from sparse views. Our method consists of two stages: scene reconstruction and scene expansion, both of which decouple background and foreground. In the scene reconstruction stage, we first interpolate a set of images between the input images based on a video generation model, followed by the optimization of the scene Gaussian from the interpolated and input images. To improve the spatial-temporal consistency of the reconstructed scene, we propose an uncertainty-aware Gaussian training approach, which introduces adaptive weights of images and pixels. In the scene expansion stage, for background, we render novel views and refine them with a geometry-aware diffusion process. These refined images are then used to incrementally add the Gaussians. As to foreground, we generate human motion according to previous motion, enabling temporal coherent generation of motion. To further enhance the physical plausibility, we integrate the expanded foreground into the background using a gravity-aware alignment. Experiments on NeuMan, Bonn, and EMDB datasets demonstrate that our SCSV achieves superior performance compared to state-of-the-art methods. The code will be released upon acceptance.

Monocular endoscope-based reconstruction of dynamic 3D surgical fields is beneficial for both intraoperative manual/robotic manipulation and post-operative surgical skills training. However, the natural characteristics of tissue deformations with instruments and blood obscuration bring great challenges to 3D scene awareness, especially under sparse viewpoints limited by laparoscopic movements. In this work, we propose Endo-4SRF, an effective Neural Radiance Field (NeRF)-based method that can reconstruct deformable tissues with instruments stealth by solely relying on monocular endoscopic image flows from a single viewpoint. Specifically, to enhance the 3D reconstruction accuracy under the deficiency of depth ground truth, we devised a dynamic Gaussian-based neural sampling strategy, leveraging the depth inherently obtained from NeRF and the conjunction information inferred by a prior learning-based depth estimation network. Besides, we integrated the Signed Distance Function (SDF) and resolved its singularity problem by furnishing additional geometric constraints for the neural radiance field, thereby achieving precise reconstruction of dynamic scenes devoid of depth ground truth supervision. Furthermore, adopting spherical harmonic functions for color fitting has significantly improved our model's computational efficiency and rendering quality. We extensively performed cross validation experiments to verify the performance using public and in-house datasets. Our quantitative and qualitative results demonstrate remarkable superiority over the state-of-the-art (SOTA) approaches concerning depth prediction accuracy, image rendering quality, model training efficiency, and 3D reconstruction outcomes.

Last-mile transport is costly and operationally fragile due to frequent layout changes, mixed traffic with pedestrians and other robots, and unreliable connectivity that makes precise maps hard to maintain. Classical multi-agent planners rely on globally consistent maps or communication. Local BEV pipelines depend on tightly registered perception stacks. End-to-end vision models can be brittle across scene changes. While 3D LiDAR offers geometry that transfers well, its high dimensionality and occlusions complicate real-time, multi-agent control without messaging. We propose an end-to-end mapless, communication-free navigator that consumes a single onboard 3D LiDAR scan and outputs discrete actions for decentralized execution. The approach introduces a dual-channel LiDAR projection that preserves near-field free space while aligning goal guidance. A graph-attention interaction head infers neighbors’ motion tendencies from local observations only, which guides different coordination strategies for pedestrians and vehicles. A structure-aware dense reward stabilizes learning around occlusion-prone layouts (e.g., long walls, U-shaped bays). Our simulations demonstrate strong generalizability and scalability across layouts and sizes. The module trained only in random scenes with 10 agents transfers to crowd and warehouse styles, which to clusters with over$10^{3}$agents. Real-world trials validate sim-to-real transfer without retraining. We deploy the model on transport vehicles and small robots under changing environments. The results indicate robust, scalable, and deployment-friendly navigation for last-mile operations.

Humans navigate unfamiliar environments using episodic simulation and episodic memory, which facilitate a deeper understanding of the complex relationships between environments and objects. Developing an imaginative memory system inspired by human mechanisms can enhance the navigation performance of embodied agents in unseen environments. However, existing Vision-and-Language Navigation (VLN) agents lack a memory mechanism of this kind. To address this, we propose a novel architecture that equips agents with a reality-imagination hybrid memory system. This system enables agents to maintain and expand their memory through both imaginative mechanisms and navigation actions. Additionally, we design tailored pre-training tasks to develop the agent's imaginative capabilities. Our agent can imagine high-fidelity RGB images for future scenes, achieving state-of-the-art results in a Success rate weighted by Path Length (SPL).

Large Language Models (LLMs) have demonstrated potential in Vision-and-Language Navigation (VLN) tasks, yet current applications face challenges. While LLMs excel in general conversation scenarios, they struggle with specialized navigation tasks, yielding suboptimal performance compared to specialized VLN models. We introduce FLAME (FLAMingo-Architected Embodied Agent), a novel Multimodal LLM-based agent and architecture designed for urban VLN tasks that efficiently handles multiple observations. Our approach implements a three-phase tuning technique for effective adaptation to navigation tasks, including single perception tuning for street view description, multiple perception tuning for route summarization, and end-to-end training on VLN datasets. The augmented datasets are synthesized automatically. Experimental results demonstrate FLAME's superiority over existing methods, surpassing state-of-the-art methods by a 7.3% increase in task completion on Touchdown dataset. This work showcases the potential of Multimodal LLMs (MLLMs) in complex navigation tasks, representing an advancement towards applications of MLLMs in the field of embodied intelligence.

Neural implicit scene representations have recently shown promising results in dense visual SLAM. However, existing implicit SLAM algorithms are constrained to single-agent scenarios, and fall difficulty in large indoor scenes and long sequences. Existing multi-agent SLAM frameworks cannot meet the constraints of communication bandwidth. To this end, we propose the first distributed multi-agent collaborative SLAM framework with distributed mapping and camera tracking, joint scene representation, intra-to-inter loop closure, and multi-submap fusion. Specifically, our proposed distributed neural mapping and tracking framework only needs peer-to-peer communication, which can greatly improve multi-agent cooperation and communication efficiency. A novel intra-to-inter loop closure method is designed to achieve local (single-agent) and global (multi-agent) consistency. Furthermore, to the best of our knowledge, there is no real-world dataset for NeRF-based/GS-based SLAM that provides both continuous-time trajectories groundtruth and high-accuracy 3D meshes groundtruth. To this end, we propose the first real-world indoor neural slam (INS) dataset covering both single-agent and multi-agent scenarios, ranging from small room to large-scale scenes, with high-accuracy ground truth for both 3D mesh and continuous-time camera trajectory. This dataset can advance the development of the community. Experiments on various datasets demonstrate the superiority of the proposed method in both mapping, tracking, and communication. The dataset and code will be open-source on https://github.com/dtc111111/MNESLAM.

Point cloud videos can faithfully capture real-world spatial geometries and temporal dynamics, which are essential for enabling intelligent agents to understand the dynamically changing world. However, designing an effective 4D backbone remains challenging, mainly due to the irregular and unordered distribution of points and temporal inconsistencies across frames. Also, recent transformer-based 4D backbones commonly suffer from large computational costs due to their quadratic complexity, particularly for long video sequences. To address these challenges, we propose a novel point cloud video understanding backbone purely based on the State Space Models (SSMs). Specifically, we first disentangle space and time in 4D video sequences and then establish the spatio-temporal correlation with the unified spatial-temporal Mamba blocks. The Intra-frame Spatial Mamba module is developed to encode locally similar geometric structures within a certain temporal stride. Subsequently, locally correlated tokens are delivered to the Inter-frame Temporal Mamba module, which integrates long-term point features across the entire video with linear complexity. Our proposed Mamba4D achieves competitive performance on the MSR-Action3D action recognition (+10.4% accuracy), HOI4D action segmentation (+0.7 F1 Score), and Synthia4D semantic segmentation (+0.19 mIoU) datasets. Mamba4D also has a significant efficiency improvement, especially for long video sequences, with 87.5% GPU memory reduction and × 5.36 speed-up. Codes are released at https://github.com/IRMVLab/Mamba4D.

Semantic correspondence made tremendous progress through the recent advancements of large vision models (LVM). While these LVMs have been shown to reliably capture local semantics, the same can currently not be said for capturing global geometric relationships between semantic object regions. This problem leads to unreliable performance for semantic correspondence between images with extreme view variation. In this work, we aim to leverage monocular depth estimates to capture these geometric relationships for more robust and data-efficient semantic correspondence. First, we introduce a simple but effective method to build 3D object-class representations from monocular depth estimates and LVM features using a sparsely annotated image correspondence dataset. Second, we formulate an alignment energy that can be minimized using gradient descent to obtain an alignment between the 3D object-class representation and the object-class instance in the input RGB-image. Our method achieves state-of-the-art matching accuracy in multiple categories on the challenging SPair-71k dataset, increasing the [email protected] score by more than 10 points on three categories and overall by 3.3 points from 85.6% to 88.9%. Additional resources and code are available at dub.sh/semalign3d.

In warehousing systems, to enhance efficiency amid surging demand volumes, much attention has been placed on how to reasonably allocate tasks of delivery to robots. However, the labor of robots is still inevitably wasted to some extent. In this paper, we propose a pre-scheduling enhanced warehousing framework aiming to foresee and act in advance, which consists of task flow prediction and hybrid task allocation. For task prediction, we design the spatio-temporal representations of the task flow and introduce a periodicity-decoupled mechanism tailored for the generation patterns of aggregated orders, and then further extract spatial features of task distribution with a novel combination of graph structures. In hybrid tasks allocation, we consider the known tasks and predicted future tasks simultaneously to optimize the task allocation. In addition, we consider factors such as predicted task uncertainty and sector-level efficiency to realize more balanced and rational allocations. We validate our task prediction model across datasets derived from factories, achieving SOTA performance. Furthermore, we implement our system in a real-world robotic warehouse, demonstrating more than 30% improvements in efficiency.

Efficient and high-fidelity reconstruction of deformable surgical scenes is a critical yet challenging task. Building on recent advancements in 3D Gaussian splatting, current methods have seen significant improvements in both reconstruction quality and rendering speed. However, two major limitations remain: (1) difficulty in handling irreversible dynamic changes, such as tissue shearing, which are common in surgical scenes; and (2) the lack of hierarchical modeling for surgical scene deformation, which reduces rendering speed. To address these challenges, we introduce EH-SurGS, an efficient and high-fidelity reconstruction algorithm for deformable surgical scenes. We propose a deformation modeling approach that incorporates the life cycle of 3D Gaussians, effectively capturing both regular and irreversible deformations, thus enhancing reconstruction quality. Additionally, we present an adaptive motion hierarchy strategy that distinguishes between static and deformable regions within the surgical scene. This strategy reduces the number of 3D Gaussians passing through the deformation field, thereby improving rendering speed. Extensive experiments on public datasets captured with static endoscopes demonstrate that our method surpasses existing state-of-the-art approaches in both reconstruction quality and rendering speed. Ablation studies further validate the effectiveness and necessity of our proposed components. We will open-source our code upon acceptance of the paper.

Dynamic scene reconstruction for autonomous driving enables vehicles to perceive and interpret complex scene changes more precisely. Dynamic Neural Radiance Fields (NeRFs) have recently shown promising capability in scene modeling. However, many existing methods rely heavily on accurate poses inputs and multi-sensor data, leading to increased system complexity. To address this, we propose FreeDriveRF, which reconstructs dynamic driving scenes using only sequential RGB images without requiring poses inputs. We innovatively decouple dynamic and static parts at the early sampling level using semantic supervision, mitigating image blurring and artifacts. To overcome the challenges posed by object motion and occlusion in monocular camera, we introduce a warped ray-guided dynamic object rendering consistency loss, utilizing optical flow to better constrain the dynamic modeling process. Additionally, we incorporate estimated dynamic flow to constrain the pose optimization process, improving the stability and accuracy of unbounded scene reconstruction. Extensive experiments conducted on the KITTI and Waymo datasets demonstrate the superior performance of our method in dynamic scene modeling for autonomous driving. Our implementation will be available at https://github.com/IRMVLab/FreeDriveRF.

Collecting data from simulated scenarios for training robotic skills provides a safer and more controllable alternative to real-world environments. However, it demands considerable effort, including the manual construction of simulation environments, the careful design of tasks, and the challenge of obtaining effective trajectories. These limitations hinder the efficiency of data collection from simulated scenarios. In this paper, we leverage the prior knowledge of Large Language Models (LLMs) and Large Multimodal Models (LMMs) to generate simulated scenarios and embodied tasks. We introduce a novel framework, ASCENT (Autonomous Skill learning toward Complex Embodied tasks with fouNdaTion models), designed to efficiently accomplish these tasks and generate trajectory data. ASCENT features a fully autonomous skill learning mechanism based on AI agent. During task training, the AI agent identifies suitable atomic skills from an atomic skill library to either directly complete the task or serve as an initial policy for further training. Newly acquired atomic skills are subsequently added to the library. To address training failures and enhance efficiency, the AI agent uses an LLM to automatically optimize the skill training process based on feedback received from simulations. Experimental results indicate that the number of training steps required for learning new tasks can be reduced by up to 65.9 %.

Sim-to-Real refers to the process of transferring policies learned in simulation to the real world, which is crucial for achieving practical robotics applications. However, recent Sim2real methods either rely on a large amount of augmented data or large learning models, which is inefficient for specific tasks. In recent years, with the emergence of radiance field reconstruction methods, especially 3D Gaussian splatting, it has become possible to construct realistic real-world scenes. To this end, we propose RL-GSBridge, a novel real-to-sim-to-real framework which incorporates 3D Gaussian Splatting into the conventional RL simulation pipeline, enabling zero-shot sim-to-real transfer for vision-based deep reinforcement learning. We introduce a mesh-based 3D GS method with soft binding constraints, enhancing the rendering quality of mesh models. Then utilizing a GS editing approach to synchronize the rendering with the physics simulator, RL-GSBridge could reflect the visual interactions of the physical robot accurately. Through a series of sim-to-real experiments, including grasping and pick-and-place tasks, we demonstrate that RL-GSBridge maintains a satisfactory success rate in real-world task completion during sim-to-real transfer. Furthermore, a series of rendering metrics and visualization results indicate that our proposed mesh-based 3D GS reduces artifacts in unstructured objects, demonstrating more realistic rendering performance.

Recent research on Simultaneous Localization and Mapping (SLAM) based on implicit representation has shown promising results in indoor environments. However, some challenges remain: the limited scene representation capability of implicit encoding, the uncertainty in the rendering process from implicit representations, and the disruption of consistency by dynamic objects. To address these challenges, we propose a dynamic visual SLAM system based on local-global fusion neural implicit representation, named DVN-SLAM. To improve the scene representation capability, we introduce a local-global fusion neural implicit representation that enables the construction of an implicit map while considering both global structure and local details. To tackle uncertainties arising from the rendering process, we design an information concentration loss for optimization, aiming to concentrate scene information on object surfaces. The proposed DVN-SLAM achieves competitive performance in localization and mapping across multiple datasets. More importantly, DVN-SLAM demonstrates robustness without semantic and optical flow prior in dynamic scenes, which sets it apart from other NeRF-based methods.

Place recognition is an important component for autonomous robot navigation. Many existing LiDAR-based place recognition methods encode the structural information of 3D LiDAR data into 2D image representations. However, most of these intermediates only exploit the projection in a single view, ignoring a great amount of useful information. In this paper, a compact fusion-view image representation of LiDAR point cloud is proposed to extract important structural information from different views. Our proposed method generates such fusion-view images using the corresponding geometric information among points highlighting the edges of objects. It then extracts texture features encoding the shapes and layouts of scene elements into global descriptors, where regional features are designed to adapt local discrepancies caused by seasonal changes. Extensive experiments on the Oxford RobotCar, NCLT, UTBM datasets and our cross-season dataset validate the proposed method and demonstrate its superior generalization performance under different LiDAR sensors and season shifts. Moreover, our proposed method can operate online with a single CPU, making it suitable for resource-limited real robot platforms.

LiDAR scene generation is critical for mitigating real-world LiDAR data collection costs and enhancing the robustness of downstream perception tasks in autonomous driving. However, existing methods commonly struggle to capture geometric realism and global topological consistency. Recent LiDAR Diffusion Models (LiDMs) predominantly embed LiDAR points into the latent space for improved generation efficiency, which limits their interpretable ability to model detailed geometric structures and preserve global topological consistency. To address these challenges, we propose TopoLiDM, a novel framework that integrates graph neural networks (GNNs) with diffusion models under topological regularization for high-fidelity LiDAR generation. Our approach first trains a topological-preserving VAE to extract latent graph representations by graph construction and multiple graph convolutional layers. Then we freeze the VAE and generate novel latent topological graphs through the latent diffusion models. We also introduce 0-dimensional persistent homology (PH) constraints, ensuring the generated LiDAR scenes adhere to real-world global topological structures. Extensive experiments on the KITTI-360 dataset demonstrate TopoLiDM’s superiority over state-of-the-art methods, achieving improvements of 22.6% lower Fréchet Range Image Distance (FRID) and 9.2% lower Minimum Matching Distance (MMD). Notably, our model also enables fast generation speed with an average inference time of 1.68 samples/s, showcasing its scalability for real-world applications. We will release the related codes at https://github.com/IRMVLab/TopoLiDM.

Predicting hand motion is critical for understanding human intentions and bridging the action space between human movements and robot manipulations. Existing hand trajectory prediction (HTP) methods forecast the future hand waypoints in 3D space conditioned on past egocentric observations. However, such models are only designed to accommodate 2D egocentric video inputs. There is a lack of awareness of multimodal environmental information from both 2D and 3D observations, hindering the further improvement of 3D HTP performance. In addition, these models overlook the synergy between hand movements and headset camera egomotion, either predicting hand trajectories in isolation or encoding egomotion only from past frames. To address these limitations, we propose novel diffusion models (MMTwin) for multimodal 3D hand trajectory prediction. MMTwin is designed to absorb multi-modal information as input encompassing 2D RGB images, 3D point clouds, past hand waypoints, and text prompt. Besides, two latent diffusion models, the egomotion diffusion and the HTP diffusion as twins, are integrated into MMTwin to predict camera egomotion and future hand trajectories concurrently. We propose a novel hybrid Mamba-Transformer module as the denoising model of the HTP diffusion to better fuse multimodal features. The experimental results on three publicly available datasets and our self-recorded data demonstrate that our proposed MMTwin can predict plausible future 3D hand trajectories compared to the state-of-the-art baselines, and generalizes well to unseen environments. The code and pretrained models will be released at https://github.com/IRMVLab/MMTwin.

Understanding how humans would behave during hand-object interaction (HOI) is vital for applications in service robot manipulation and extended reality. To achieve this, some recent works simultaneously forecast hand trajectories and object affordances on human egocentric videos. The joint prediction serves as a comprehensive representation of future HOI in 2D space, indicating potential human motion and motivation. However, the existing approaches mostly adopt the autoregressive paradigm, which lacks bidirectional constraints within the holistic future sequence, and accumulates errors along the time axis. Meanwhile, they overlook the effect of camera egomotion on first-person view predictions. To address these limitations, we propose a novel diffusion-based HOI prediction method, namely Diff-IP2D, to forecast future hand trajectories and object affordances with bidirectional constraints in an iterative non-autoregressive manner on egocentric videos. Motion features are further integrated into the conditional denoising process to enable Diff-IP2D aware of the camera wearer’s dynamics for more accurate interaction prediction. Extensive experiments demonstrate that Diff-IP2D significantly outperforms the state-of-the-art baselines on both the off-the-shelf and our newly proposed evaluation metrics. This highlights the efficacy of leveraging a generative paradigm for 2D HOI prediction. The code and the video have been released at https://github.com/IRMVLab/Diff-IP2D.

Forecasting how human hands would move around target objects on egocentric videos can provide prior knowledge to enhance the path planning capabilities of service robots and assistive wearable devices. During the hand-object interaction process, head movements always occur concurrently to provide observations for the interaction scene from different egocentric views. Although some prior works have successfully integrated head motion information into hand trajectory prediction (HTP), they basically overlook the multi-view consistency (MVC) inherent in headset camera egomotion. We argue that multi-view consistency reveals geometric and semantic relationships during hand-object interaction, and can be regarded as additional supervision signals for predicting more realistic hand trajectories. Therefore, in this work, we propose a novel learning scheme dubbed EER to improve diffusion-based 2D hand trajectory prediction methods, which involves exploiting the geometric consistency, enhancing the multi-canvas consistency, and reconstructing the semantic consistency inherent in MVC. The experimental results show that our proposed EER scheme significantly improves the prediction accuracy of existing diffusion-based 2D HTP methods on the publicly available datasets. We will release the code as open-source at https://github.com/IRMVLab/EER-HTP.

We propose a globally consistent semantic SLAM system (GCSLAM) and a semantic-fusion localization subsystem (SF-Loc), which achieves accurate semantic mapping and robust localization in complex parking lots. Visual cameras (front-view and surround-view), IMU, and wheel encoder form the input sensor configuration of our system. The first part of our work is GCSLAM. GCSLAM introduces a semantic-constrained factor graph for the optimization of poses and semantic map, which incorporates innovative error terms based on multi-sensor data and BEV (bird’s-eye view) semantic information. Additionally, GCSLAM integrates a Global Slot Management module that stores and manages parking slot observations. SF-Loc is the second part of our work, which leverages the semantic map built by GCSLAM to conduct map-based localization. SF-Loc integrates registration results and odometry poses with a novel factor graph. Our system demonstrates superior performance over existing SLAM on two real-world datasets, showing excellent capabilities in robust global localization and precise semantic mapping.

Utilizing 2D images for place recognition within 3D point cloud maps presents significant challenges in autonomous driving applications, primarily due to the inherent cross-modal disparity between visual and LiDAR data. In this study, we propose a novel cross-modal visual-LiDAR place recognition method based on Bird’s Eye View (BEV) feature distillation. Our framework is the first end-to-end solution designed to achieve cross-modal place recognition between surround-view images and LiDAR point clouds. By encoding features into a unified BEV representation, our approach effectively bridges the modality gap between 3D and 2D data. Additionally, we introduce a teacher-student distillation training strategy to further enhance the network’s cross-modal generalization capabilities. Extensive experiments on benchmark datasets, including nuScenes and Argoverse, demonstrate that our method achieves state-of-the-art (SOTA) performance in cross-modal place recognition tasks. Furthermore, validation on the SJTU-Sanya dataset confirms the robustness and adaptability of our approach in real-world scenarios. We publicly release our network model and implementation details at https://github.com/IRMVLab/CrossBEV-PR.

We propose SGLoc, a novel localization system that directly regresses camera poses from 3D Gaussian Splatting (3DGS) representation by leveraging semantic information. Our method utilizes the semantic relationship between 2D image and 3D scene representation to estimate the 6DoF pose without prior pose information. In this system, we introduce a multilevel pose regression strategy that progressively estimates and refines the pose of query image from the global 3DGS map, without requiring initial pose priors. Moreover, we introduce a semantic-based global retrieval algorithm that establishes correspondences between 2D (image) and 3D (3DGS map). By matching the extracted scene semantic descriptors of 2D query image and 3DGS semantic representation, we align the image with the local region of the global 3DGS map, thereby obtaining a coarse pose estimation. Subsequently, we refine the coarse pose by iteratively optimizing the difference between the query image and the rendered image from 3DGS. Our SGLoc demonstrates superior performance over baselines on 12scenes and 7scenes datasets, showing excellent capabilities in global localization without initial pose prior. Code will be available at https://github.com/IRMVLab/SGLoc.

We propose SemGauss-SLAM, a dense semantic SLAM system utilizing 3D Gaussian representation, that enables accurate 3D semantic mapping, robust camera tracking, and high-quality rendering simultaneously. In this system, we incorporate semantic feature embedding into 3D Gaussian representation, which effectively encodes semantic information within the spatial layout of the environment for precise semantic scene representation. Furthermore, we propose feature-level loss for updating 3D Gaussian representation, enabling higher-level guidance for 3D Gaussian optimization. In addition, to reduce cumulative drift in tracking and improve semantic reconstruction accuracy, we introduce semantic-informed bundle adjustment. By leveraging multi-frame semantic associations, this strategy enables joint optimization of 3D Gaussian representation and camera poses, resulting in low-drift tracking and accurate semantic mapping. Our SemGauss-SLAM demonstrates superior performance over existing radiance field-based SLAM methods in terms of mapping and tracking accuracy on Replica and ScanNet datasets, while also showing excellent capabilities in high-precision semantic segmentation and dense semantic mapping. Code will be available at https://github.com/IRMVLab/SemGauss-SLAM.

Autonomous exploration in complex multi-agent reinforcement learning (MARL) with sparse rewards critically depends on providing agents with effective intrinsic motivation. While artificial curiosity offers a powerful self-supervised signal, it often confuses environmental stochasticity with meaningful novelty. Moreover, existing curiosity mechanisms exhibit a uniform novelty bias, treating all unexpected observations equally. However, peer behavior novelty, which encode latent task dynamics, are often overlooked, resulting in suboptimal exploration in decentralized, communication-free MARL settings. To this end, inspired by how human children adaptively calibrate their own exploratory behaviors via observing peers, we propose a novel approach to enhance multi-agent exploration. We introduce CERMIC, a principled framework that empowers agents to robustly filter noisy surprise signals and guide exploration by dynamically calibrating their intrinsic curiosity with inferred multi-agent context. Additionally, CERMIC generates theoretically-grounded intrinsic rewards, encouraging agents to explore state transitions with high information gain. We evaluate CERMIC on benchmark suites including VMAS, Meltingpot, and SMACv2. Empirical results demonstrate that exploration with CERMIC significantly outperforms SoTA algorithms in sparse-reward environments.

Visual navigation is a fundamental problem in embodied AI, yet practical deployments demand long-horizon planning capabilities to address multi-objective tasks. A major bottleneck is data scarcity: policies learned from limited data often overfit and fail to generalize OOD. Existing neural network-based agents typically increase architectural complexity that paradoxically become counterproductive in the small-sample regime. This paper introduce NeuRO, a integrated learning-to-optimize framework that tightly couples perception networks with downstream task-level robust optimization. Specifically, NeuRO addresses core difficulties in this integration: (i) it transforms noisy visual predictions under data scarcity into convex uncertainty sets using Partially Input Convex Neural Networks (PICNNs) with conformal calibration, which directly parameterize the optimization constraints; and (ii) it reformulates planning under partial observability as a robust optimization problem, enabling uncertainty-aware policies that transfer across environments. Extensive experiments on both unordered and sequential multi-object navigation tasks demonstrate that NeuRO establishes SoTA performance, particularly in generalization to unseen environments. Our work thus presents a significant advancement for developing robust, generalizable autonomous agents.

The core of multi-robot collision avoidance is to guide robots to avoid collisions with other robots and obstacles in a dynamic multi-robot environment, which has recently gained increasing interest among the main challenges of robotics. However, the current multi-robot navigation policy neural network exhibits weak position encoding capabilities for spatial environmental features in mapping environment states and robot actions, as well as an inability to recurrently infer information on dynamic environmental features in the temporal dimension, leading to insufficient safety and effectiveness in guiding robot motion. In this paper, we propose a novel spatial-temporal RetNet (STR) that encodes reciprocal collision avoidance states between robots in both spatial and temporal dimensions, aiming to enhance the safety and effectiveness of the policy neural network in guiding robots to accomplish specified tasks. The spatial state encoder module is developed based on parallel RetNet structure, which enhances the ability of the neural network in multi-robot navigation policies to extract reciprocal collision avoidance states between robots in spatial dimensions and overcomes the weak position encoding capability of advanced transformer-based multi-robot navigation policy neural networks. A temporal state encoder is designed by introducing the recurrent RetNet structure. This enhances the multi-robot navigation policy neural network’s ability to encode features in the temporal dimension of multi-robot movements and overcomes the transformer-based multi-robot navigation policy neural network’s inability to recurrently infer information in the time dimension. Simulation experiments were designed to demonstrate that the safety and effectiveness of our proposed method outperform the previous state-of-the-art approaches in guiding the robot to complete the task. Physical experiments illustrate that our policy can be effectively applied to real-world systemsNote to Practitioners—Multi-robot navigation has a wide range of real-world applications, such as multi-robot formation flying for search and rescue, autonomous warehouse operations, and robots navigating through human crowds. This paper introduces a novel Spatial-Temporal RetNet (STR) framework aimed at enhancing safety and effectiveness in multi-robot collision avoidance. STR addresses the limitations of existing methods by improving the neural network’s ability to extract reciprocal collision avoidance states in both spatial and temporal dimensions. The spatial state encoder strengthens the extraction of spatial features, while the temporal state encoder improves the handling of time-dependent information. Simulation and physical experiments demonstrate that STR enhances robot navigation in dynamic environments, making it suitable for real-world applications such as multi-robot coordination.

Dense scene reconstruction for photo-realistic view synthesis has various applications, such as VR/AR, and robotics navigation. Existing dense reconstruction methods are primarily designed for small room scenarios, but in practice, the scenes encountered by robots are typically large-scale environments. Most existing methods have difficulties in large-scale scenes due to three core challenges:(a) inaccurate depth input. Depth information is crucial for both scene geometry reconstruction and pose estimation. Accurate depth input is impossible to get in real-world large-scale scenes.(b) inaccurate pose estimation. Existing methods are not robust enough with the growth of cumulative errors in large scenes and long sequences.(c) insufficient scene representation capability. A single global radiance field lacks the capacity to scale effectively to large-scale scenes. To this end, we propose an incremental joint learning framework, which can achieve accurate depth, pose estimation, and large-scale dense scene reconstruction. For depth estimation, a vision transformer-based network is adopted as the backbone to enhance performance in scale information estimation. For pose estimation, a feature-metric bundle adjustment (FBA) method is designed for accurate and robust camera tracking in large-scale scenes and eliminates pose drift. In terms of implicit scene representation, we propose an incremental scene representation method to construct the entire large-scale scene as multiple local radiance fields to enhance the scalability of 3D scene representation. In local radiance fields, we propose a tri-plane based scene representation method to further improve the accuracy and efficiency of scene reconstruction. We conduct extensive experiments on various datasets, including our own collected data, to demonstrate the effectiveness and accuracy of our method in depth estimation, pose estimation, and large-scale scene reconstruction. The code has been open-sourced on https://github.com/dtc111111/incre-dpsr.

In recent years, there have been significant advancements in 3D reconstruction and dense RGB-D SLAM systems. One notable development is the application of Neural Radiance Fields (NeRF) in these systems, which utilizes implicit neural representation to encode 3D scenes. However, the depth images obtained from consumer-grade RGB-D sensors are often sparse and noisy, which poses significant challenges for 3D reconstruction and affects the accuracy of the representation of the scene geometry. Furthermore, existing methods select random pixels for camera tracking, leading to inaccurate localization in real-world indoor environments. To this end, we present NeSLAM, an advanced framework that achieves accurate and dense depth estimation, robust camera tracking, and realistic synthesis of novel views. First, a depth completion and denoising network is designed to provide dense geometry prior and guide the neural implicit representation optimization. Second, we propose a NeRF-based self-supervised feature tracking algorithm for robust real-time tracking. Experiments on various indoor datasets demonstrate the effectiveness and accuracy of the system in reconstruction, tracking quality, and novel view synthesis. Note to Practitioners—Traditional SLAM methods usually use the sparse point cloud to represent the scene, resulting in poor scene representation capability. Our method proposes a neural implicit representation method with depth completion and denoising network and feature tracking method, achieves accurate scene reconstruction and accurate pose estimation in various indoor scenes. The depth completion and denoising network provide accurate depth information associated with depth uncertainty, which is used to improve the geometry consistency. The NeRF-based self-supervised feature tracking method improve the accuracy and robustness for camera tracking. The experimental results demonstrate the accuracy and effectiveness of this method in different scenes.

Magnetic soft flapping milliswimmers hold significant potential for biomedical applications due to their flexible mobility and multifunctionality. However, advanced motion control tasks for such forms remain underexplored, hindering their flexible mobility and various biomedical applications. The multi-physics coupling inherent in the flapping swimming process presents substantial challenges for dynamics modeling and control. In this study, we fabricated a magnetic soft flapping millirobot and conducted dynamics analysis using rigid-body modeling and quasi-static wing-flapping analysis method. The resulting state-space model is employed to build a navigation control framework based on adaptive Model Predictive Control (MPC) method, realizing autonomous navigation and obstacle avoidance performances. To validate the effectiveness of the proposed control framework, the complex path-following task in 3D space is performed, demonstrating superior performance compared to geometry-based following methods. Furthermore, the robot’s navigation ability in complex environments is evaluated in both physical and simulated setups featuring static and dynamics obstacles. The milliswimmer’s 3D flexible mobility and its capability to operate effectively in challenging and intricate environments may reveal great potential for real-world biomedical and other advanced applications.

Existing target segmentation methods are typically considered easy to implement and yield satisfactory results. However, they generally cannot adapt to a new environment without tiring parameter tuning, which leads to poor performance, including issues such as over-segmentation, under-segmentation, missing segmentation, and false positives. To avoid the wrong segmentation in a parameter-tuning-free and user-friendly fashion, we propose a generalized target segmentation (GTS) method based on the image-view representation of point clouds. Specifically, the method avoids devising a Euclidean space-based algorithm that is sensitive to surrounding objects and to the varied point cloud density and intensity in a new environment. The target segment produced by GTS can be used with any target-based extrinsic calibration architecture, based on which this paper further proposes a generalized target-based (in this case, chessboard) extrinsic calibration framework called GTSCalib for a non-repetitive scanning LiDAR and a camera. GTSCalib additionally introduces a novel intensity threshold method based on kernel density estimation (KDE) for 3D corner detection and the SQPnP solver for optimization to achieve more generalized and robust performance. Extensive simulations and experiments demonstrate that GTSCalib has high generalization ability, robustness, and accuracy. The code is released at https://github.com/Natsu-Akatsuki/GTSCalib.Note to Practitioners—Calibration is necessary for many non-repetitive scanning LiDAR-camera systems to enable sensor fusion in the fields of mapping, localization, and perception. Unfortunately, existing target-based (in this case, chessboard) calibration methods are weakly adaptable to the surrounding environment with variable density or intensity of point clouds, resulting in unstable performance, particularly for the target segmentation submodule. To solve this problem, we introduce a new target segmentation approach, GTS, and a more generalized and robust extrinsic calibration framework, GTSCalib. The proposed GTSCalib is very suitable for practitioners looking for a robust and accurate target-based calibration without limits on the target’s pose or its surrounding environment and without the need for time-consuming parameter tuning.

Soft robots face significant control challenges due to the coupling effects of strain and external forces, particularly with external interactions that change their original model. This paper proposes a vision-based controller integrated with an adaptive algorithm to estimate and compensate for external contact disturbances in such environments. Specifically, the adaptive law can online estimate unknown camera parameters, eliminating the need for costly and environment-specific calibration procedures. In parallel, a contact disturbance estimation strategy is introduced to model and compensate for real-time interaction effects without the foreknowledge of the constitutive equation of the soft mechanism. The adaptive algorithm is seamlessly integrated into the adaptive image-based visual servo (IBVS) controller, allowing simultaneous calibration and contact compensation during task execution. We validated the algorithm on a tendon-driven, octoarticular soft robotic manipulator prototype. The experimental results demonstrated the algorithm’s ability to position the end-effector even under external interactions, with the adaptive parameters converging, thereby validating the effectiveness of the online estimation in assessing the impacts of interactions.

Navigation is a fundamental capability for robots to operate in expansive spaces. Reliable navigation in unknown environments is crucial for deploying robots in areas such as disaster rescue and industrial inspection. In such scenarios, it is essential for robots to construct memories based on historical data to support long-term, optimized decision-making. However, many existing techniques focus on memorizing direct features from raw perceptions, often resulting in redundancy due to irrelevant textures and areas. This approach leads to inefficiencies in computation and storage, and produces a memory structure that lacks general applicability. We suggest that it may not be necessary to store specific scene features. Instead, recalling the robot’s episodic movements could provide sufficient cognitive cues for navigation. To address this, we introduce Memory Enhanced Navigation with Embedded Odometry (MENEO), a framework consisting of three steps: ego-motion estimation, memory aggregation, and adaptive policy generation. MENEO offers two main advantages: its streamlined architecture significantly boosts computational and storage efficiency, and its universal design supports various sensor modalities, adapts to multiple navigation tasks, and accommodates different scenarios. We test MENEO in two different environments: maze exploration using a lidar-IMU sensor, and image-goal visual navigation in photorealistic indoor scenes. In both cases, MENEO demonstrates competitive navigation performance, outperforming existing methods by reducing storage and computational requirements. Additionally, MENEO’s compact memory representation not only enhances adaptability across diverse environments but also shows flexibility in real-world applications.Note to Practitioners—In learning-based navigation, the memory mechanism is essential for long-term optimized policies. It allows intelligent robots to make informed decisions by utilizing a wide range of temporal and spatial cues derived from historical data. Traditional methods use various types of memory (such as internal, unstructured, or structured), but these often result in computational and storage inefficiencies due to the direct inclusion of complex and redundant raw scene features. Furthermore, because these memory systems are closely linked to specific scene features, they lack general applicability across different sensor configurations, scene types, and tasks. In this paper, we aim to eliminate the need to directly manage the redundant environmental features found in previous memory structures. Instead, we propose focusing solely on memorizing a robot’s self-movements. Since the pose sequence is streamlined and compact, our approach not only enhances computational and storage efficiency but also improves the interpretability and universality of the navigation system. This advantage enables MENEO to be seamlessly integrated into a wide variety of intelligent navigation systems. It is especially beneficial for small robots with limited computing power, such as those used in search and rescue operations, where enhanced memory can greatly enhance their autonomous navigation capabilities.

Autonomous navigation substantially depends on memory mechanisms for enhancing path optimality. However, the dynamic nature of environments can cause inconsistencies between the stored scene memory and real-time perception, leading to potentially catastrophic navigation errors. Existing memory structures often fall short in addressing these challenges, as they typically only account for object-level dynamics and falter when faced with long-term alterations in environmental structure. To counter these constraints, this paper presents a learning-based framework designed to construct and maintain a geometric memory, thereby facilitating improved visuomotor navigation under structural dynamics. This proposed framework employs visual inputs to construct a geometric representation of the environment, and identifies structural changes by assessing the consistency between the established memory and instantaneous perception. To update the geometric memory efficiently, we introduce a memory updater grounded in a structure storage pool. Furthermore, a two-phase hierarchical planner is proposed to decompose navigation tasks and formulate smooth navigation strategies. Experimental results from photorealistic simulations underscore the efficacy of the proposed system in managing long-term dynamics and navigation control. The effectiveness of the proposed system is further corroborated through deployment and testing in real-world environments.Note to Practitioners—Classic geometry-based navigation methods hinge on the construction of a global map for spatial reasoning and optimized robot control. Within a learning-based pipeline, the dependence on memory information becomes critical for enabling robots to develop spatial and temporal awareness. While the maintenance of memory structures can significantly expand the field of the robot’s perception in time and space, discrepancies between historical memory and real-time perception in dynamic environments can lead to misguided decisions. Unlike most existing research that addresses short-term dynamic issues at the object level, this paper centers on long-term, large-scale dynamics precipitated by changes in environmental structure. We put forward a learning-based framework that incorporates dynamic perception, map maintenance, and hierarchical navigation. The experimental results highlight the efficiency and real-time processing capability of this method in handling structural dynamics, hence enhancing navigation efficiency.

The whole-body control of mobile manipulators for the 6-DOF trajectory following task is the basis of many continuous tasks. However, traditional control strategies rely on accurate models and expert knowledge for solving the trajectory following task. Deep reinforcement learning (DRL) provides a promising model-free solution, but it is sample-inefficient. To this end, we propose Trajectory Following Hindsight Experience Replay (TF-HER), a sample-efficient DRL algorithm for the whole-body coupled trajectory following task with dense rewards. TF-HER builds a multi-trajectory state space, and relabels the low-reward data to generate informative high-reward experiences. Also, the distributional shift caused by the relabeling is corrected by estimating the density ratio of relabeled experiences. Extensive demonstrations on both nonholonomic and holonomic bases in simulation validate that our algorithm can accelerate the model convergence and significantly improve the sample efficiency. Furthermore, we present real-world experiments to demonstrate the effectiveness of our approach. The code is available:https://github.com/IRMV-Manipulation-Group/TF-HER.Note to Practitioners—The whole-body 6-DOF trajectory tracking capability for mobile manipulators is crucial in industrial automation, serving as the foundation for a wide range of continuous and precise operations including automated assembly, welding, and material handling. This paper proposes a reinforcement learning approach to enhance the efficiency and effectiveness of mobile manipulators, requiring no prior model information. Beyond this specific application, this research has promising applications in other robotic systems because it is model-free. By utilizing a gradient-descent-based relabeling method and an adaptive density ratio estimator, we address distributional shifts and mitigate hindsight bias, guiding the mobile manipulator to accurately follow complex 6-DOF trajectories with dense rewards. The experimental results highlight our method’s superior performance over existing model-free techniques, with improved sample efficiency and reduced tracking error across diverse robotic platforms in both simulated and real-world settings. The successful policy transfer from simulation to physical robots with fine-tuning further validates the robustness and practical applicability.

Traditional LiDAR SLAM approaches prioritize localization over mapping, yet high-precision dense maps are essential for numerous applications involving intelligent agents. Recent advancements have introduced methods leveraging neural fields to enhance mapping capabilities; however, these approaches still face several limitations. Firstly, concerning scene representation, they typically employ neural fields with high-dimensional features and multi-layer perceptron decoders utilizing non-continuous activation functions. This results in low learning efficiency and challenges in capturing high-frequency signals. Secondly, in terms of scene organization, these methods often treat the entire scene as a singular neural field, leading to inefficiencies, inflexibility, and difficulties in rectifying accumulated errors when mapping large-scale environments over extended periods. To tackle the first issue, we propose a lightweight continuous SDF regression approach by encoding the scene in single-valued embeddings and decoding SDF values from a Kolmogorov-Arnold Network. By minimizing discrepancies in measuring range, sampling distance, and decoded SDF values, we facilitate iterative frame-to-model tracking and bundle adjustment neural mapping. To mitigate the second challenge, we propose structuring the whole scene into multiple neural SDF submaps. By establishing node-node, node-submap, and loop closure constraints into a global pose graph, the system can create dense neural maps with global consistency across large-scale scenes. Experimental evaluations in both real-world and simulated settings indicate that our system achieves superior mapping completeness and accuracy, enhanced learning efficiency, reduced memory consumption, and greater flexibility compared to its counterparts.

Visual navigation is fundamental for embodied agents operating in expansive workspaces. The cognitive abilities of these agents form the essential basis for creating intelligent behavioral patterns. Memory and reasoning are vital components among these abilities. The former enhances decision-making by preserving a wide array of episodic spatio-temporal perception cues, while the latter allows proactive and advanced probabilistic inference of task distributions based on long-term experiences. Despite individual studies on these two cognitive modalities, their integration for enhanced decision-making presents a considerable challenge due to their substantial differences in representation and behavioral characteristics. In this paper, we introduce Semantic-based Multi-modal Cognitive Graph (SMCG) for intelligent visual navigation. This framework is distinguished by its unified semantic-level representation of both memory and reasoning capabilities. Specifically, SMCG, rather than directly memorizing perceptual features as per previous methods, records observed object sequences. Simultaneously, reasoning is based on a semantic relation graph that represents correlations among objects. We additionally develop a hierarchical cognition extraction (HCE) pipeline and employ it to decode cognitive cues within SMCG and situation-aware subgraphs, thereby enhancing intelligent navigation behavior. Experimental results in image-goal navigation show pronounced performance improvements, credited to the effective induction and rational application of heterogeneous cognitive modalities.

3D scene flow represents the 3D motion of each point in the point cloud, which is a base 3D perception task for autonomous driving, like optical flow for 2D images. As non-learning methods are often inefficient or struggled to learn accurate correspondence in complex 3D real world, recent works turn to supervised learning methods, which require ground truth labels. However, acquiring the ground truth of 3D scene flow is challenging mainly due to the lack of sensors capable of capturing point-level motion and the complexity of accurately tracking each point in real-world environments. Therefore, it is important to resort to self-supervised methods, which do not require ground truth labels. In this paper, a novel unsupervised learning method of scene flow with LiDAR odometry is proposed, which enables the scene flow network can be trained directly on real-world LiDAR data without scene flow labels. In this structure, supervised odometry provides a more accurate shared cost volume for the interframe association of 3D scene flow. In addition, because static and occluded points are more suitable for using the pose transform while dynamic and non-occluded points are more suitable for using the scene flow transform, a static mask and an occlusion mask are designed to classify the states of points and a mask-weighted warp layer is proposed to transform source points in a divide-and-conquer manner. The experiments demonstrate that the divide-and-conquer strategy makes the predicted scene flow more accurate. The experiment results compared to other methods also show the application ability of our proposed method to real-world data. Our source codes are released at:https://github.com/IRMVLab/PSFNet.

Reconstructing deformable soft tissues from endoscopic videos is a critical yet challenging task. Leveraging depth priors, deformable implicit neural representations have seen significant advancements in this field. However, depth priors from pre-trained depth estimation models are often coarse, and inaccurate depth supervision can severely impair the performance of these neural networks. Moreover, existing methods overlook local similarities in input sequences, which restricts their effectiveness in capturing local details and tissue deformations. In this paper, we introduce UW-DNeRF, a novel approach utilizing neural radiance fields for high-quality reconstruction of deformable tissues. We propose an uncertainty-guided depth supervision strategy to mitigate the impact of inaccurate depth information. This strategy relaxes hard depth constraints and unlocks the potential of implicit neural representations. In addition, we design a local window-based information sharing scheme. This scheme employs local window and keyframe deformation networks to construct deformations with local awareness and enhances the model's ability to capture fine details. We demonstrate the superiority of our method over state-of-the-art approaches on synthetic and in vivo endoscopic datasets. Code is available at: https://github.com/IRMVLab/UW-DNeRF.

The fundamental prerequisite for embodied agents to make intelligent decisions lies in autonomous cognition. Typically, agents optimize decision-making by leveraging extensive spatiotemporal information from episodic memory. Concurrently, they utilize long-term experience for task reasoning and foster conscious behavioral tendencies. However, due to the significant disparities in the heterogeneous modalities of these two cognitive abilities, existing literature falls short in designing effective coupling mechanisms, thus failing to endow robots with comprehensive intelligence. This article introduces a navigation framework, the hierarchical topology-semantic cognitive navigation (HTSCN), which seamlessly integrates both memory and reasoning abilities within a singular end-to-end system. Specifically, we represent memory and reasoning abilities with a topological map and a semantic relation graph, respectively, within a unified dual-layer graph structure. Additionally, we incorporate a neural-based cognition extraction process to capture cross-modal relationships between hierarchical graphs. HTSCN forges a link between two different cognitive modalities, thus further enhancing decision-making performance and the overall level of intelligence. Experimental results demonstrate that in comparison to existing cognitive structures, HTSCN significantly enhances the performance and path efficiency of image-goal navigation. Visualization and interpretability experiments further corroborate the promoting role of memory, reasoning, as well as their online learned relationships, on intelligent behavioral patterns. Furthermore, we deploy HTSCN in real-world scenarios to further verify its feasibility and adaptability.

Pushing is an essential nonprehensile manipulation for robots to achieve complex tasks. Until now, object rigidity remains one of the common assumptions in robotic pushing. To endow robots with the advanced capability of pushing deformable objects, we propose a mathematical model and control method for the planar pushing of deformable objects. Given the robotic end-effector velocity or position input, the model predicts the motion and deformation of the pushed object, which is developed based on the quasi-static finite element analysis with reasonable simplification, considering the contact conditions of nodes with both the operator and the contact surface. By combining the designed model to estimate the state of the object and interactions with the environment, we further propose a method based on model predictive control to realize the pushing control. With a specialized simplified model to accelerate prediction, the controller is solved by iterative linear quadratic regulator with a dynamic weight, which balances the object motion and pushing area adjustment. The accuracy and efficiency of the proposed deformable model are validated by comparing the theoretical results with the experimental ones under different conditions, and the controller is verified by simulation and experiments.

In this article, we propose a two-stage shared control framework for physical human–robot interaction (pHRI) that addresses the inconsistency of human–robot commands and consider the influence of environmental information. In the human–robot–environment system, based on the human intention measured by the interaction force, autonomy will actively initiate the replanning when the human control intention is strong, generating a feasible local desired trajectory of the robot. At the same time, we define an index called predicted safety index (PSI) to measure the safety of the system status. When the human has control intention but does not reach the threshold, we propose a shared controller based on cooperative-game theory and PSI. Specially, it is designed within the model predictive control framework, utilizing cooperative game theory to analyze human–robot interaction behavior and treating the Pareto optimal solution as the control input. We conduct comparative experiments to evaluate the assistive performance of the proposed shared control algorithm through a waypoint tracking task with naive human users. User study with objective and subjective measures demonstrate that the algorithm effectively reduces human effort while maintaining tracking accuracy, thus enhancing both performance and safety.

Robot localization using a built map is essential for a variety of tasks including accurate navigation and mobile manipulation. A popular approach to robot localization is based on image-to-point cloud registration, which combines illumination-invariant LiDAR-based mapping with economical image-based localization. However, the recent works for image-to-point cloud registration either divide the registration into separate modules or project the point cloud to the depth image to register the RGB and depth images. In this paper, we present I2PNet, a novel end-to-end 2D-3D registration network, which directly registers the raw 3D point cloud with the 2D RGB image using differential modules with a united target. The 2D-3D cost volume module for differential 2D-3D association is proposed to bridge feature extraction and pose regression. The soft point-to-pixel correspondence is implicitly constructed on the intrinsic-independent normalized plane in the 2D-3D cost volume module. Moreover, we introduce an outlier mask prediction module to filter the outliers in the 2D-3D association before pose regression. Furthermore, we propose the coarse-to-fine 2D-3D registration architecture to increase localization accuracy. Extensive localization experiments are conducted on the KITTI, nuScenes, M2DGR, Argoverse, Waymo, and Lyft5 datasets. The results demonstrate that I2PNet outperforms the state-of-the-art by a large margin and has a higher efficiency than the previous works. Moreover, we extend the application of I2PNet to the camera-LiDAR online calibration and demonstrate that I2PNet outperforms recent approaches on the online calibration task. Source codes are released athttps://github.com/IRMVLab/I2PNet.

Learning 3D scene flow from LiDAR point clouds presents significant difficulties, including poor generalization from synthetic datasets to real scenes, scarcity of real-world 3D labels, and poor performance on real sparse Li-DAR point clouds. We present a novel approach from the perspective of auto-labelling, aiming to generate a large number of 3D scene flow pseudo labels for real-world Li-DAR point clouds. Specifically, we employ the assumption of rigid body motion to simulate potential object-level rigid movements in autonomous driving scenarios. By updating different motion attributes for multiple anchor boxes, the rigid motion decomposition is obtained for the whole scene. Furthermore, we developed a novel 3D scene flow data augmentation method for global and local motion. By perfectly synthesizing target point clouds based on augmented motion parameters, we easily obtain lots of 3D scene flow labels in point clouds highly consistent with real scenarios. On multiple real-world datasets including LiDAR KITTI, nuScenes, and Argoverse, our method outperforms all previous supervised and unsupervised methods without requiring manual labelling. Impressively, our method achieves a tenfold reduction in EPE3D metric on the LiDAR KITTI dataset, reducing it from 0.190m to a mere 0.008m error.

Scene flow estimation, which aims to predict per-point 3D displacements of dynamic scenes, is a fundamen-tal task in the computer vision field. However, previ-ous works commonly suffer from unreliable correlation caused by locally constrained searching ranges, and struggle with accumulated inaccuracy arising from the coarse-to-fine structure. To alleviate these problems, we propose a novel uncertainty-aware scene flow estimation network(DifFlow3D) with the diffusion probabilistic model. Iter-ative diffusion-based refinement is designed to enhance the correlation robustness and resilience to challenging cases, e.g. dynamics, noisy inputs, repetitive patterns, etc. To re-strain the generation diversity, three key flow-related features are leveraged as conditions in our diffusion model. Furthermore, we also develop an uncertainty estimation module within diffusion to evaluate the reliability of esti-mated scene flow. Our DifFlow3D achieves state-of-the-art performance, with 24.0% and 29.1% EPE3D reduction respectively on FlyingThings3D and KITTI 2015 datasets. Notably, our method achieves an unprecedented millimeter-level accuracy (O.0078m in EPE3D) on the KITTI dataset. Additionally, our diffusion-based refinement paradigm can be readily integrated as a plug-and-play module into ex-isting scene flow networks, significantly increasing their estimation accuracy. Codes are released at https:// github.com/IRMVLab/DifFlow3D.

Understanding how the surrounding environment changes is crucial for performing downstream tasks safely and reliably in autonomous driving applications. Recent occupancy estimation techniques using only camera images as input can provide dense occupancy representations of large-scale scenes based on the current observation. However, they are mostly limited to representing the current 3D space and do not consider the future state of surrounding objects along the time axis. To extend camera-only occupancy estimation into spatiotemporal prediction, we propose Cam4DOcc, a new benchmark for camera-only 4D occupancy forecasting, evaluating the surrounding scene changes in a near future. We build our benchmark based on multiple publicly available datasets, including nuScenes, nuScenes-Occupancy, and Lyft-Level5, which provides sequential occupancy states of general movable and static objects, as well as their 3D backward centripetal flow. To establish this benchmark for future research with comprehensive comparisons, we introduce four baseline types from diverse camera-based perception and prediction implementations, including a static-world occupancy model, voxelization of point cloud prediction, 2D-3D instance-based prediction, and our proposed novel end-to-end 4D occupancy forecasting network. Furthermore, the standardized evaluation protocol for preset multiple tasks is also provided to compare the performance of all the proposed baselines on present and future occupancy estimation with respect to objects of interest in autonomous driving scenarios. The dataset and our implementation of all four baselines in the proposed Cam4DOcc benchmark are released as open source at https://github.com/haomo-ai/Cam4DOcc.

We propose SNI-SLAM, a semantic SLAM system utilizing neural implicit representation, that simultaneously performs accurate semantic mapping, high-quality surface reconstruction, and robust camera tracking. In this system, we introduce hierarchical semantic representation to allow multi-level semantic comprehension for top-down structured semantic mapping of the scene. In addition, to fully utilize the correlation between multiple attributes of the environment, we integrate appearance, geometry and semantic features through cross-attention for feature collaboration. This strategy enables a more multifaceted understanding of the environment, thereby allowing SNI-SLAM to remain robust even when single attribute is defective. Then, we design an internal fusion-based decoder to obtain semantic, RGB, Truncated Signed Distance Field (TSDF) values from multi-level features for accurate decoding. Furthermore, we propose a feature loss to update the scene representation at the feature level. Compared with low-level losses such as RGB loss and depth loss, our feature loss is capable of guiding the network optimization on a higher-level. Our SNI-SLAM method demonstrates superior performance over all recent NeRF-based SLAM methods in terms of mapping and tracking accuracy on Replica and ScanNet datasets, while also showing excellent capabilities in accurate semantic segmentation and real-time semantic mapping. Codes will be available at https://github.com/IRMVLab/SNI-SLAM.

Localization using a monocular camera in the pre-built LiDAR point cloud map has drawn increasing attention in the field of autonomous driving and mobile robotics. However, there are still many challenges (e.g. difficulties of map storage, poor localization robustness in large scenes) in accurately and efficiently implementing cross-modal localization. To solve these problems, a novel pipeline termed LHMap-loc is proposed, which achieves accurate and efficient monocular localization in LiDAR maps. Firstly, feature encoding is carried out on the original LiDAR point cloud map by generating offline heat point clouds, by which the size of the original LiDAR map is compressed. Then, an end-to-end online pose regression network is designed based on optical flow estimation and spatial attention to achieve real-time monocular visual localization in a pre-built map. In addition, a series of experiments have been conducted to prove the effectiveness of the proposed method. Our code is available at: https://github.com/IRMVLab/LHMap-loc.

Navigating robots through dynamic multi-robot environments, avoiding collisions with both other robots and obstacles, has emerged as a central challenge in robotics. The existing approaches fall short in allowing the policy network to effectively capture spatial-temporal reciprocal collision avoidance in multi-robot environments, comprising both static and dynamic obstacles, resulting in inadequate safety and efficiency in directing robot movement. In this study, we introduce a novel policy neural network called Spatial-Temporal RetNet (STR), designed to encode reciprocal collision avoidance states between robots in spatial and temporal dimensions. The goal is to improve the safety and efficacy of the policy neural network in directing robots to complete assigned tasks. The spatial state encoder module is built upon a parallel RetNet structure, which strengthens the neural network's capacity in extracting reciprocal collision avoidance states between robots in spatial dimensions. This module addresses the limitations of position encoding in transformer-based multi-robot navigation policy neural networks. We design a temporal state encoder utilizing a recurrent RetNet structure. This innovation bolsters the multi-robot navigation policy neural network's capability to capture features in the temporal dimension of multi-robot movements. It addresses the limitations of transformer-based multi-robot navigation policy neural networks, particularly in recurrently inferring information across time dimensions. Simulation experiments were conducted to showcase the superior safety and effectiveness of our proposed method compared to previous state-of-the-art approaches in guiding robots to accomplish tasks.

To date, multi-robot path planning has primarily been addressed by centralized solvers, typically aiming to maintain optimality. However, given its NP-hard nature, directly applying existing solvers in large and complex scenarios proves inefficient. A promising alternative lies in adopting a divide- and-conquer strategy to break down the problem into manageable sub-problems. In this work, we propose a systematic, universal, and scalable graph partitioning method, aiming to automatically divide any real-world environment into multiple regions. Building upon this, we convert the path planning on the entire graph into distributed sub-region path planning and devise corresponding inter-regional strategies. Our work can be easily implementable in practical systems and effectively enhances the scalability of existing solvers. Experimentally, our approach contributes to a tenfold improvement in computational efficiency while only sacrificing about 10% of optimality.

Recently, Shuttle-based Storage and Retrieval Systems (SBS/RSs) have garnered significant attention from both academia and industry, owing to their high spatial utilization and rapid response speed. However, the weak connectivity of roadmaps in densely stored environments increases the likelihood of congestion and deadlocks when multiple four-way shuttles operate simultaneously, thereby imposing greater demands for collaborative path planning. Instead of exhaustively coordinating the shuttle motions during the off-line planning or online local control stages, we solve the cooperative path planning challenge from the perspective of altering the road graph structure dynamically. More specifically, we propose an approach to automatically transfer the typical road graph of SBS/RSs into a hierarchical graph with a reduced size, and then dynamically adjust its edge properties to prohibit any motion conflicts. In this manner, the planning problem of large-scale shuttle groups can be easily resolved and all the potential congestions can be eliminated inherently. Finally, we build a complete multi-shuttle cooperative path planning system adaptable for large-scale problems.

This paper presents a novel pneumatic continuum module featuring high contraction-ratio, bidirectional actuation, and active stiffness regulation. The module comprises four linear pneumatic actuators integrating rigid polygon origami frame into soft bellow. This integration not only helps to regulate motion but also enhances structural strength, facilitating contraction and bending performance. The contraction resulting from vacuum pressure serves as a limiting layer for one opposing pair of actuators, while the other pair operates in a virtual antagonistic configuration, allowing for active regulation of joint stiffness through pressure control. The paper provides a detailed workflow covering the design, fabrication, and mathematical modeling of the pneumatic module. Furthermore, the paper presents verifications of the module’s actuation capacity and active variable stiffness. The findings of this study can serve as valuable references for the design of manipulators.

When coordinating the motion of connected autonomous vehicles at a signal-free intersection, the vehicles from each direction naturally forms a team and each team seeks to minimize their own traversal time through the intersection, without concerning the traversal times of other teams. Since the intersection is shared by all teams and agent-agent collision must be avoided, the coordination has to trade the traversal time of one team for the other. This paper thus investigates a problem called Multi-Agent Teamwise Cooperative Path Finding (TCPF), which seeks a set of collision-free paths for the agents from their respective start to goal locations, and agents are grouped into multiple teams with each team having its own objective function to optimize. In general, there are more than one teams and hence multiple objectives. TCPF thus seeks the Pareto-optimal front that represents possible trade-offs among the teams. We develop a centralized planner for TCPF by leveraging the Multi-Agent Path Finding techniques to resolve agent-agent collision, and Multi-Objective Optimization to find Pareto-optimal solutions. We analyze the completeness and optimality of the planner, which is then tested in various settings with up to 40 agents to verify the runtime efficiency and showcase the usage in intersection coordination.

Building a self-model for robots, enabling them to simulate their physical selves and predict future states without direct interaction with the physical world, is crucial for robot motion planning and control. Existing self-modeling methods primarily focus on rigid robots and typically require significant time, effort, and resources to gather training data. In this study, we introduce SoftNeRF, a self-supervised visual self-model designed for soft robots. We use a hybrid neural shape representation based on the Signed Distance Function (SDF) to capture both the geometry and complex nonlinear motion of soft robots. By leveraging differentiable rendering, our method learns a self-model from readily available RGB images, similar to how humans understand their physical state through reflection. To improve training efficiency and model accuracy, we propose an error-guided adaptive sampling strategy. SoftNeRF can serve as a plug-in for various downstream tasks, even when trained with data unrelated to those tasks. We demonstrate SoftNeRF’s ability to support shape prediction and motion planning for robots in both simulated and real-world environments. Furthermore, SoftNeRF excels in detecting and recovering from damage, thereby enhancing machine resilience. Code is available at: https://github.com/irmvlab/soft-nerf.

Estimating camera motion and continuously reconstructing dense scenes in deformable environments presents a complex and open challenge. Many existing approaches tend to rely on assumptions about the scene’s topology or the nature of deformable motion. However, these assumptions do not hold true in medical endoscopy applications. To address these challenges, we introduce DDS-SLAM, a novel dense deformable semantic neural SLAM that achieves accurate camera tracking, continuous dense scene reconstruction, and high-quality image rendering in deformable scenes. First, we propose a novel hybrid neural scene representation method capable of capturing both natural and artificial deformations. Additionally, by leveraging the 2D semantic information of the scene, we introduce a semantic loss function based on semantic distance fields. This approach guides network optimization at a higher level, thereby enhancing system performance. Furthermore, we validate our method through a series of experiments conducted on several representative medical datasets, demonstrating its superiority over other state-of-the-art approaches. The code is available at: https://github.com/IRMVLab/DDS-SLAM.

In the context of visual navigation in unknown scenes, both “exploration” and “exploitation” are equally crucial. Robots must first establish environmental cognition through exploration and then utilize the cognitive information to accomplish target searches. However, most existing methods for image-goal navigation prioritize target search over the generation of exploratory behavior. To address this, we propose the Navigation with Uncertainty-driven Exploration (NUE) pipeline, which uses an implicit and compact scene representation, NeRF, as a cognitive structure. We estimate the uncertainty of NeRF and augment the exploratory ability by the uncertainty to in turn facilitate the construction of implicit representation. Simultaneously, we extract memory information from NeRF to enhance the robot’s reasoning ability for determining the location of the target. Ultimately, we seamlessly combine the two generated abilities to produce navigational actions. Our pipeline is end-to-end, with the environmental cognitive structure being constructed online. Extensive experimental results on image-goal navigation demonstrate the capability of our pipeline to enhance exploratory behaviors, while also enabling a natural transition from the exploration to exploitation phase. This enables our model to outperform existing memory-based cognitive navigation structures in terms of navigation performance. Project page: https://github.com/IRMVLab/NUE-NeRF-nav

For aerial swarms, formation flight has been applied in various scenes. However, most existing works do not consider balancing the conflicting requirements among keeping formation, keeping the smoothness of trajectories, and obstacle avoidance within the limited time. To address this issue, we propose a decentralized trajectory planning framework for formation flight in unknown and dense environments. To ensure that feasible trajectories can be found within the limited time, the formation optimization problem is decoupled into formation affine transformation and iterative trajectory generation. Firstly, the optimization problem based on affine transformation is designed to obtain the optimal affine transformation sequence, which provides the formation reference of trajectory optimization. Secondly, the iterative optimization framework of trajectory planning is designed, which balances the conflicting requirements of formation, smooth flight, and obstacle avoidance. Besides, to escape the local minima caused by non-convex dense environments, the method of topological path planning is designed to provide distinctive initial solutions for trajectory optimization. Finally, the proposed methods are proven to be effective through the simulations and real-world experiments.

This paper introduces a 3D point cloud sequence learning model based on inconsistent spatio-temporal propagation for LiDAR odometry, termed DSLO. It consists of a pyramid structure with a spatial information reuse strategy, a sequential pose initialization module, a gated hierarchical pose refinement module, and a temporal feature propagation module. First, spatial features are encoded using a point feature pyramid, with features reused in successive pose estimations to reduce computational overhead. Second, a sequential pose initialization method is introduced, leveraging the high-frequency sampling characteristic of LiDAR to initialize the LiDAR pose. Then, a gated hierarchical pose refinement mechanism refines poses from coarse to fine by selectively retaining or discarding motion information from different layers based on gate estimations. Finally, temporal feature propagation is proposed to incorporate the historical motion information from point cloud sequences, and address the spatial inconsistency issue when transmitting motion information embedded in point clouds between frames. Experimental results on the KITTI odometry dataset and Argoverse dataset demonstrate that DSLO outperforms state-of-the-art methods, achieving at least a 15.67% improvement on RTE and a 12.64% improvement on RRE, while also achieving a 34.69% reduction in runtime compared to baseline methods. Our implementation will be available at https://github.com/IRMVLab/DSLO.

LiDAR point cloud semantic segmentation enables the robots to obtain fine-grained semantic information of the surrounding environment. Recently, many works project the point cloud onto the 2D image and adopt the 2D Convolutional Neural Networks (CNNs) or vision transformer for LiDAR point cloud semantic segmentation. However, since more than one point can be projected onto the same 2D position but only one point can be preserved, the previous 2D projection-based segmentation methods suffer from inevitable quantized information loss, which results in incomplete geometric structure, especially for small objects. To avoid quantized information loss, in this paper, we propose a novel spherical frustum structure, which preserves all points projected onto the same 2D position. Additionally, a hash-based representation is proposed for memory-efficient spherical frustum storage. Based on the spherical frustum structure, the Spherical Frustum sparse Convolution (SFC) and Frustum Farthest Point Sampling (F2PS) are proposed to convolve and sample the points stored in spherical frustums respectively. Finally, we present the Spherical Frustum sparse Convolution Network (SFCNet) to adopt 2D CNNs for LiDAR point cloud semantic segmentation without quantized information loss. Extensive experiments on the SemanticKITTI and nuScenes datasets demonstrate that our SFCNet outperforms previous 2D projection-based semantic segmentation methods based on conventional spherical projection and shows better performance on small object segmentation by preserving complete geometric structure. Codes will be available at https://github.com/IRMVLab/SFCNet.

Multi-Agent Combinatorial Path Finding (MCPF) seeks collision-free paths for multiple agents from their initial locations to destinations, visiting a set of intermediate target locations in the middle of the paths, while minimizing the sum of arrival times. While a few approaches have been developed to handle MCPF, most of them simply direct the agent to visit the targets without considering the task duration, i.e., the amount of time needed for an agent to execute the task (such as picking an item) at a target location. MCPF is NP-hard to solve to optimality, and the inclusion of task duration further complicates the problem. To handle task duration, we develop two methods, where the first method post-processes the paths planned by any MCPF planner to include the task duration and has no solution optimality guarantee; and the second method considers task duration during planning and is able to ensure solution optimality. The numerical and simulation results show that our methods can handle up to 20 agents and 50 targets in the presence of task duration, and can execute the paths subject to robot motion disturbance.

We propose an end-to-end visuomotor navigation framework that leverages Neural Radiance Fields (NeRF) for spatial cognition. To the best of our knowledge, this is the first effort to integrate such implicit spatial representation with embodied policy end-to-end for cognitive decision-making. Consequently, our system does not necessitate modularized designs nor transformations into explicit scene representations for downstream control. The NeRF-based memory is constructed online during navigation, without relying on any environmental priors. To enhance the extraction of decision-critical historical insights from the rigid and implicit structure of NeRF, we introduce a spatial information extraction mechanism named Structural Radiance Attention (SRA). SRA empowers the agent to grasp complex scene structures and task objectives, thus paving the way for the development of intelligent behavioral patterns. Our comprehensive testing in image-goal navigation tasks demonstrates that our approach significantly outperforms existing navigation models. We demonstrate that SRA markedly improves the agent's understanding of both the scene and the task by retrieving historical information stored in NeRF memory. The agent also learns exploratory awareness from our pipeline to better adapt to low signal-to-noise memory signals in unknown scenes. We deploy our navigation system on a mobile robot in real-world scenarios, where it exhibits evident cognitive capabilities while ensuring real-time performance.

Designing a safe and effective collision avoidance policy for multiple robots is essential in decentralized scenarios, where each robot is responsible for generating its own paths, to ensure their safe operation. Recently, the utilization of reinforcement learning to develop decentralized policies that enable multiple robots to move cooperatively and accomplish tasks has yielded positive outcomes. However, the presence of exploration unsafe actions during the reinforcement learning training process results in inadequate safety. We seek to enhance the safety of distributed multi-robot navigation policies and propose a new imitation learning framework based on the variational Bayesian model, which enables robots to learn safe actions by anticipating the subsequent state they are expected to reach. In addition, a new policy neural network structure for multi-robot navigation is proposed by introducing the transformer structure, which encodes the significance of nearby robots in relation to their forthcoming conditions. Experiments demonstrated that our policy can more safely guide robots to navigate in multi-robot environments under conditions of limited information, outperforming the state-of-the-art RL-RVO method in terms of success rate.Note to Practitioners—The motivation of this paper is to address the problem of collision avoidance in a multi-robot environment under limited information, which can also be applied to autonomous driving, crowd simulation, and other related fields. Positive outcomes have been observed in the utilization of reinforcement learning to create decentralized policies that enable multiple robots to move cooperatively and complete tasks. However, inadequate safety remains a challenging task due to the possibility of exploring hazardous actions during training. This article aims to enhance the safety of distributed policies guiding robots to accomplish navigation tasks in dynamic multi-robot environments. To begin with, we introduce a novel framework for imitation learning that is based on the variational Bayesian model. This framework facilitates the learning of safe actions by the policy to improve its performance and guide the robot in navigating and avoiding obstacles more securely. A loss function is proposed that enables the anticipation of the future state expected to be reached by the robot. By incorporating the transformer structure, a new neural network structure is designed for multi-robot navigation that encodes the significance of nearby robots concerning their upcoming conditions. This network structure employs a BiGRUs to facilitate the assimilation of observations from multiple agents by the policy. Compared to existing works such as GA3C-CADRL, SARL, and RL-RVO, our proposed method achieves a higher success rate. In our future research, we will investigate methods to enhance the policy’s performance in guiding robots to complete tasks by focusing on improving travel time and average speed, while also strictly ensuring safe navigation. Furthermore, we plan to extend this approach by addressing navigation challenges in more densely populated multi-robot environments.

This article studies the visual servoing and vibration suppression control for flexible manipulators when the system states are unmeasurable and only the image feedback is available. The dynamic equations of flexible manipulators are decomposed into the slow and fast subsystems based on the singular perturbation theory. The nonlinear observers based on the state transformation using the Lie derivatives are proposed to estimate the unmeasurable system states and unknown camera intrinsic parameters at the same time. Then, the image-based controllers utilizing the estimated states are, respectively, designed in the slow and fast subsystems to regulate the image positions of feature points and suppress the vibration of flexible manipulators simultaneously. In the proposed approach, only the visual feedback is required to generate the control input for flexible manipulators, which simplifies the controller implementation. The stability of the proposed control scheme is proved based on the Lyapunov theory. Finally, experimental results on a flexible single-link manipulator are provided to demonstrate the effectiveness of the proposed control approach.

In this paper, an imitation learning framework is innovatively developed for robot skills learning in violin bowing. The introduction of Dynamic Movement Primitives (DMPs) to model motion addresses issues such as the uncertainty in changing string angles, which traditional methods like physical measurements are highly error-prone and non-generalizable. Alternatively, Conventional rule-based methods often fail to precisely mimic the complex playing techniques of humans, especially in terms of musicality and emotional expression. To address this, we propose a new model named Fuzzy and PCADynamic Movement Primitive (FP-DMP), built upon Type-2 Fuzzy Models and Principal Component Analysis (PCA). This model utilizes variables derived from PCA (bowing plane angles) as inputs for the membership function and employs fuzzy kmeans clustering to identify the angle of the violin bowing plane, independent of physical measurements or positional information. Additionally, we have developed a professional-level musical performance behavior database through a Task-Parameterized Gaussian Mixture Model (TP-GMM). Biomimetic experiments prove that our FP-DMP model can control robots for highprecision violin performances. The core technology, FP-DMP, combines Gaussian Mixture Models (GMM) and Principal Component Analysis (PCA) for efficient trajectory clustering based on bowing angles, uses type-2 fuzzy K-means for nuanced differentiation of violin string trajectories, and innovatively adapts the force term in DMP to account for the non-linear friction encountered in violin playing. This not only advances imitation learning in complex performance tasks for robots but also opens new research avenues for imitation learning in other highly specialized and technically challenging fields.

Creating memory representations is essential for developing viable navigation strategies for intelligent agents. Although neural radiance fields (NeRFs) have shown great promise as a novel method for spatial representation, their potential for integration into learning-based navigation as a memory structure has been largely overlooked in the existing literature. In this article, we introduce a navigation pipeline that incorporates NeRF into visuomotor navigation. Initially, we present a derivative radiance field that facilitates one-shot pose and depth estimation from a single query image. By assuming equivalence between density and space occupancy, we generate a geometric accessibility map based on an offline-constructed NeRF prior. Utilizing the above information, we design a global planner that decomposes long-term tasks by performing waypoint estimation and rendering. Finally, we employ an imitation-learned local controller to achieve a reliable navigation policy. Our pipeline effectively utilizes NeRF's compact spatial representation for task decomposition and action generation, enabling efficient navigation. Experimental results highlight its advantages over recent implicit and explicit memory approaches in image-goal navigation tasks. Moreover, we conduct interpretability studies and apply our algorithm in real-world scenarios to further attest to its practicality and effectiveness.

Deep reinforcement learning (DRL) enables robots to perform some intelligent tasks end-to-end. However, there are still many challenges for long-horizon sparse-reward robotic manipulator tasks. On the one hand, a sparse-reward setting causes exploration inefficient. On the other hand, exploration using physical robots is of high cost and unsafe. In this article, we propose a method of learning long-horizon sparse-reward tasks utilizing one or more existing traditional controllers named base controllers in this article. Built upon deep deterministic policy gradients (DDPGs), our algorithm incorporates the existing base controllers into stages of exploration, value learning, and policy update. Furthermore, we present a straightforward way of synthesizing different base controllers to integrate their strengths. Through experiments ranging from stacking blocks to cups, it is demonstrated that the learned state-based or image-based policies steadily outperform base controllers. Compared to previous works of learning from demonstrations, our method improves sample efficiency by orders of magnitude and improves performance. Overall, our method bears the potential of leveraging existing industrial robot manipulation systems to build more flexible and intelligent controllers.

The field of soft manipulators requires a more promising solution, including efficient structures and controllers. This article presents a novel cable–pneumatic hybrid-driven tapered soft manipulator (TSM) design and control scheme to enhance the performance in actual tasks. This article is the first to present the design with a Bowden tube as a driving tendon and propose a composite tendon with Bowden tubes and cable tendons (BTCTs). Leveraging the principles of hybrid-driven antagonism, the compact TSM integrates the composite tendon with BTCTs and pneumatically actuated tapered bellows. This new hybrid-driven form provides the TSM with excellent resistance to axial extension, tangential bending, and torsion, enhancing the stiffness of the TSM. The variable-stiffness range of the TSM was quantified in tests, including axial stiffness (0.57–10.77 N/mm), tangential bending stiffness (0.01–0.45 N/mm), and torsion stiffness (0.02–0.044 N$\cdot$m/$^\circ$) tests. A deep learning-based neural network approach was utilized to model the inverse kinematics of the TSM. For more precise motion control, using position and orientation feedback from the sensor at the tip, we have designed a closed-loop iterative feedback controller incorporating three algorithms. Experiments on spatial point positioning, trajectory tracking with different constraints, orientation control, and disturbance experiments were conducted on the TSM. Experimental results [spatial point positioning error (mean error of stable region: 0.17 mm), circular trajectory tracking error (mean and standard deviation (SD) of 100 trials: 0.87$\pm$0.57 mm), orientation control error (less than 1$^{\circ }$), and the performance in disturbance experiment] demonstrated that our approach has high control accuracy and strong robustness against external disturbances. We conducted experiments involving teleoperation control, collision-free precise operations in cluttered and constrained environments, and disturbance-adaptive board cleaning testing, ensuring both stability and safety during contact with humans. These experiments intuitively demonstrate the potential of this TSM for executing complex tasks in real-world environments, promising to become a safe collaborative assistant for humans in the future.

This paper presents a novel biomimetic morphing quadrotor design inspired by the morphology of an eagle claw during prey capture. The arms of the quadrotor are capable of vertical folding to enable dynamic grasping, mimicking the transition of the eagle claw from an open to a closed state. This transition is achieved through the rotation of a central servomotor and the associated movement of 20 links. Thanks to the closed-loop multi-link structure of the frame, the propellers of the quadrotor remain in a fixed orientation when the arms are folded, allowing for system stabilization at any arm rotation angle. The geometric property of the whole frame is analyzed to determine the relationships and constraints of the links, which is important in experimental vehicle fabrication. To handle possible physical property changes and external disturbances during grasping, the adaptive sliding mode controllers are applied. To deal with objects of unknown size in grasping tasks, an admittance filter is proposed for adaptive morphology. While in flight, our proposed morphing quadrotor is able to rapidly or continuously transition to any configuration within its range smoothly. Experimental results show the ability of the quadrotor to dynamically grasp various unknown objects at 0.4m/s without additional tools, as well as its versatility in traversal of narrow spaces and perching.

Recently, transformer architecture has gained great success in the computer vision community, such as image classification, object detection, etc. Nonetheless, its application for 3D vision remains to be explored, given that point cloud is inherently sparse, irregular, and unordered. Furthermore, existing point transformer frameworks usually feed raw point cloud of N×3 dimension into transformers, which limits the point processing scale because of their quadratic computational costs to the input size N. In this paper, we rethink the structure of point transformer. Instead of directly applying transformer to points, our network (TransLO) can process tens of thousands of points simultaneously by projecting points onto a 2D surface and then feeding them into a local transformer with linear complexity. Specifically, it is mainly composed of two components: Window-based Masked transformer with Self Attention (WMSA) to capture long-range dependencies; Masked Cross-Frame Attention (MCFA) to associate two frames and predict pose estimation. To deal with the sparsity issue of point cloud, we propose a binary mask to remove invalid and dynamic points. To our knowledge, this is the first transformer-based LiDAR odometry network. The experiment results on the KITTI odometry dataset show that our average rotation and translation RMSE achieves 0.500°/100m and 0.993% respectively. The performance of our network surpasses all recent learning-based methods and even outperforms LOAM on most evaluation sequences.Codes will be released on https://github.com/IRMVLab/TransLO.

Although point cloud registration has achieved remarkable advances in object-level and indoor scenes, large-scale registration methods are rarely explored. Challenges mainly arise from the huge point number, complex distribution, and outliers of outdoor LiDAR scans. In addition, most existing registration works generally adopt a two-stage paradigm: They first find correspondences by extracting discriminative local features and then leverage estimators (eg. RANSAC) to filter outliers, which are highly dependent on well-designed descriptors and post-processing choices. To address these problems, we propose an end-to-end transformer network (RegFormer) for large-scale point cloud alignment without any further post-processing. Specifically, a projection-aware hierarchical transformer is proposed to capture long-range dependencies and filter outliers by extracting point features globally. Our transformer has linear complexity, which guarantees high efficiency even for large-scale scenes. Furthermore, to effectively reduce mismatches, a bijective association transformer is designed for regressing the initial transformation. Extensive experiments on KITTI and NuScenes datasets demonstrate that our RegFormer achieves competitive performance in terms of both accuracy and efficiency. Codes are available at https://github.com/IRMVLab/RegFormer.

Robust estimation is a crucial and still challenging task, which involves estimating model parameters in noisy environments. Although conventional sampling consensus-based algorithms sample several times to achieve robustness, these algorithms cannot use data features and historical information effectively. In this paper, we propose RLSAC, a novel Reinforcement Learning enhanced SAmple Consensus framework for end-to-end robust estimation. RLSAC employs a graph neural network to utilize both data and memory features to guide exploring directions for sampling the next minimum set. The feedback of downstream tasks serves as the reward for unsupervised training. Therefore, RL-SAC can avoid differentiating to learn the features and the feedback of downstream tasks for end-to-end robust estimation. In addition, RLSAC integrates a state transition module that encodes both data and memory features. Our experimental results demonstrate that RLSAC can learn from features to gradually explore a better hypothesis. Through analysis, it is apparent that RLSAC can be easily transferred to other sampling consensus-based robust estimation tasks. To the best of our knowledge, RLSAC is also the first method that uses reinforcement learning to sample consensus for end-to-end robust estimation. We release our codes at https://github.com/IRMVLab/RLSAC.

Point clouds are naturally sparse, while image pixels are dense. The inconsistency limits feature fusion from both modalities for point-wise scene flow estimation. Previous methods rarely predict scene flow from the entire point clouds of the scene with one-time inference due to the memory inefficiency and heavy overhead from distance calculation and sorting involved in commonly used farthest point sampling, KNN, and ball query algorithms for local feature aggregation. To mitigate these issues in scene flow learning, we regularize raw points to a dense format by storing 3D coordinates in 2D grids. Unlike the sampling operation commonly used in existing works, the dense 2D representation 1) preserves most points in the given scene, 2) brings in a significant boost of efficiency, and 3) eliminates the density gap between points and pixels, allowing us to perform effective feature fusion. We also present a novel warping projection technique to alleviate the information loss problem resulting from the fact that multiple points could be mapped into one grid during projection when computing cost volume. Sufficient experiments demonstrate the efficiency and effectiveness of our method, outperforming the prior-arts on the FlyingThings3D and KITTI dataset. Our source codes will be released on https://github.com/IRMVLab/DELFlow.

Depth estimation is one of the most important tasks in scene understanding. In the existing joint self-supervised learning approaches of depth-pose estimation, depth estimation and pose estimation networks are independent of each other. They only use the adjacent image frames for pose estimation and lack the use of the estimated geometric information. To enhance the depth-pose association, we propose a monocular multi-frame unsupervised depth estimation framework, named PLPE-Depth. There are a depth estimation network and two pose estimation networks with image input and pseudo-LiDAR input. The main idea of our approach is to use the pseudo-LiDAR reconstructed from the depth map to estimate the pose of adjacent frames. We propose depth re-estimation with a better pose between the image pose and the pseudo-LiDAR pose to improve the accuracy of estimation. Besides, we improve the reconstruction loss and design a pseudo-LiDAR pose enhancement loss to facilitate the joint learning. Our approach enhances the use of the estimated depth information and strengthens the coupling between depth estimation and pose estimation. Experiments on the KITTI dataset show that our depth estimation achieves state-of-the-art performance at low resolution. Our source codes will be released on https://github.com/IRMVLabIPLPE-Depth.

Highly accurate and efficient map in unknown and complex environments is essential for robotics navigation. Traditionally, mobile robot platforms are often computationally constrained when using multiple sensors to process large amounts of input data. In previous works, some of them have been deployed to embedded platforms in real-time. However, how to balance accuracy and efficiency while reducing the computational resources and the memory footprint is still the bottleneck. Motivated by these challenges, we proposed a mapping framework called VDBblox to incrementally build Euclidean Signed Distance Fields (ESDFs) map from Truncated Signed Distance Fields (TSDFs) mapping. We use a novel weight function to update the non-projective TSDFs, thus improving the quality of the mesh reconstruction with higher accuracy than up-to-date methods. Meanwhile, the generated ESDFs map is maintained by the least recently used (LRU) cache to dynamically handle the obstacle changes with less runtime than state-of-the-art. We show VDBblox performance in terms of accuracy and efficiency by benchmark comparison on RGB-D and LiDAR public datasets. Moreover, we demonstrate that VDBblox can be integrated into a completed quadrotor system as a sub-module. Then we validate it through online obstacle avoidance and high-quality mesh reconstruction in real-world experiments. Finally, we release our method as open-source code to the community11Code - https://github.com/yinloonga/vdbblox.

Different environments pose a great challenge to the outdoor robust visual perception for long-term autonomous driving, and the generalization of learning-based algorithms on different environments is still an open problem. Although monocular depth prediction has been well studied recently, few works focus on the robustness of learning-based depth prediction across different environments, e.g. changing illumination and seasons, owing to the lack of such a multi-environment real-world dataset and benchmark. To this end, the cross-season monocular depth prediction dataset and benchmark, SeasonDepth, is introduced to benchmark the depth estimation performance under different environments. We investigate several state-of-the-art representative open-source supervised and self-supervised depth prediction methods using newly-formulated metrics. Through extensive experimental evaluation on the proposed dataset and cross-dataset evaluation with current autonomous driving datasets, the performance and robustness against the influence of multiple environments are analyzed qualitatively and quantitatively. We show that long-term monocular depth prediction is still challenging and believe our work can boost further research on the long-term robustness and generalization for outdoor visual perception. The dataset is available on https://seasondepth.github.io.

Traditional methods based on Lyapunov analysis and learning-based approaches such as reinforcement learning (RL) are two powerful tools in visual servo tasks. Traditional methods are interpretable and their stability can be guar-anteed by Lyapunov analysis. However, they tend to have a high dependency on an accurate system dynamic model. RL approaches learn to act based on past experiences and thus have higher adaptability on disturbances and errors. However, the training process is long and its safety or stability is generally hard to guarantee, making real-world training risky. In this paper, we propose a residual RL framework for training a multicopter to finish visual servo tasks under disturbances, guided by the system safety in terms of system Lyapunov function. Such an approach compensates for the lack of disturbance-rejection ability of the traditional method, and optimizes stability explicitly so the RL agent makes safer actions both during the training and in the final policy. A comparison between our approach and the baselines is provided in simulation, and real-world experiments on a multicopter are also carried out to show our effectiveness. We believe that this work moves one step toward achieving RL applications on real-world robotic systems.

An efficient 3D point cloud learning architecture, named EfficientLO-Net, for LiDAR odometry is first proposed in this article. In this architecture, the projection-aware representation of the 3D point cloud is proposed to organize the raw 3D point cloud into an ordered data form to achieve efficiency. The Pyramid, Warping, and Cost volume (PWC) structure for the LiDAR odometry task is built to estimate and refine the pose in a coarse-to-fine approach. A projection-aware attentive cost volume is built to directly associate two discrete point clouds and obtain embedding motion patterns. Then, a trainable embedding mask is proposed to weigh the local motion patterns to regress the overall pose and filter outlier points. The trainable pose warp-refinement module is iteratively used with embedding mask optimized hierarchically to make the pose estimation more robust for outliers. The entire architecture is holistically optimized end-to-end to achieve adaptive learning of cost volume and mask, and all operations involving point cloud sampling and grouping are accelerated by projection-aware 3D feature learning methods. The superior performance and effectiveness of our LiDAR odometry architecture are demonstrated on KITTI, M2DGR, and Argoverse datasets. Our method outperforms all recent learning-based methods and even the geometry-based approach, LOAM with mapping optimization, on most sequences of KITTI odometry dataset. We open sourced our codes at: https://github.com/IRMVLab/EfficientLO-Net.

Contact-based aerial interaction control in unknown environments is a challenging issue. To solve this problem, this paper presents an image-based impedance control scheme for an aerial manipulator. Unlike points and image moments, the line features considered in this paper cannot uniquely determine the pose of the onboard camera. Therefore, a nonlinear observer is proposed to online provide supplementary 3D information of the system. After that, a hierarchical tracking controller, which is equipped with an image-space impedance filter, is designed to implement a trajectory tracking task with tunable compliance, while at the same time achieving a subtask that aims to determine the final pose of the aerial manipulator. By planning task-space trajectories, desired interaction behavior can be specified without relying on the global position of the system. Furthermore, the effectiveness criterion of the observer is derived, and the stability of the system is proven. Several experiments, including a push-and-slide task, robustness analysis, and a comparison with the state of the art have been conducted to demonstrate the feasibility of the method. Note to Practitioners—This article presents a method for controlling an aerial manipulator to interact with unknown environments without measuring the global position of both the environment and the aerial manipulator. Furthermore, unlike the existing methods, the proposed visual servoing method is based on line features, which are generally more robust than point features. Moreover, it would be intuitive for users to specify the approaching path and the interaction behavior of the aerial manipulator by just planning the image feature trajectories and the desired distance between the aerial manipulator and the environment. This distance is online estimated using the visual feedback of a monocular camera, instead of relying on any costly 3D-vision sensors or lidars, making it an economical solution for aerial manipulation in field environments.

This paper addresses the landing problem of a quadrotor on an unpredictable moving vehicle, using a robust image-based visual servoing (IBVS) method. The circle-based image moments are defined to construct image dynamics, and the passivity-like property of the circle features is preserved by reprojecting to the virtual image plane. The landing control system is decoupled into translation and rotation modules due to the rotation invariance of the proposed circle features. First, by exploiting the error transformation in the image space, a robust IBVS controller can overcome the lack of both the desired depth information of target features and the velocity feedback of the target. Next, an adaptive geometric attitude controller is developed directly using rotation matrices to avoid the singularities of Euler-angles and the ambiguity of quaternions. One benefit of the proposed scheme is that it can potentially improve the camera visibility, guarantee the transient and steady-state behaviors in image space, and be efficiently implemented on the low-cost quadrotor. Finally, The stability analysis is presented using Lyapunov stability theory on cascaded systems, and the effectiveness of the proposed control strategy is demonstrated through simulations and experiments. Note to Practitioners—The motivation of this paper is to investigate a practical control strategy for the image-based landing control of underactuated quadrotors on an unpredictable moving vehicle. In most of the existing image-based landing control schemes for underactuated quadrotors, having the prior predictive model of the moving landing vehicle to provide a feed-forward compensation during the landing maneuver. However, due to the fact that the landing environment and vehicle are primarily stochastic, resulting in no predictive models are valid in practice. Therefore, this paper suggests a robust image-based landing control strategy without the model or state of the moving landing vehicle. In particular, a novel virtual circle feature, possessing the characteristic of rotation invariance, is designed for the landing of underactuated quadrotors, which decouples the landing system and simplifies the control design. Moreover, the image feature errors are directly retained within prescribed performance funnels in the image space. As a result, the transient and steady-state landing behaviors can be implicitly guaranteed in Cartesian space. The stability and convergence of the system are analyzed mathematically and the experiment using quadrotors provides promising results. In ongoing research, we are addressing the issues of collision avoidances and unknown disturbances to provide a more realistic setup for the autonomous deployment and recovery of underactuated quadrotors in GPS-denied environments.

In this paper, an image-based impedance control strategy for force tracking of an unmanned aerial manipulator (UAM) is presented. Firstly, image features with nice decoupling characteristics are designed and the relationship between the camera motion and the image features is derived. Then, a two-stage strategy is proposed to achieve force tracking of the UAM on a planar object in an arbitrary pose. The first stage drives the end-effector perpendicular to the object’s planer surface by pure visual servoing. To achieve force tracking under the visual guidance, an adaptive visual impedance control method which adjusts the target stiffness according to the force tracking error and the visual feature error is proposed in the second stage. The closed-loop system is proved asymptotically stable by means of Lyapunov analysis. Further, the stability in free flight phase of the stage two is also analyzed and ensured. Finally, experiments were carried out including a whiteboard cleaning task in different poses. The experimental results illustrate the validity and effectiveness of the proposed approach. Note to Practitioners—This work is motivated by the contact force tracking problem of an unmanned aerial manipulator (UAM) without the position measurement. In recent years, impedance control is widely used in force tracking problems. However, position measurement is needed both for the robot and the object in most studies. To generate desired force under visual guidance, image-based visual servoing is combined with a novel impedance controller with variable stiffness. The idea is intuitive. For a human in contact with a wall, the contact force is controlled by adjusting his arm stiffness, which means the stiffness is adapted to the difference between the desired and actual contact force. Furthermore, the stiffness is also adapted to the visual tracking error, removing the assumption that no tracking error is in the inner loop of the impedance controller in the previous works. The proposed strategy is a promising solution for real applications and is validated by a board cleaning experiment in this paper.

This paper presents a finite-time visual servoing control strategy for the autonomous landing of a quadrotor onto a tilting and moving landing vehicle. The proposed method, called Finite-Time Dynamic Visual Servo (FTDVS) control, utilizes a computer vision technique called Virtual Reticle Image Plane (VRIP) to track four observable features on the landing plane called landmarks. To represent the sea state, target-plane tilting motion is modelled using monochromatic sinusoidal waves. VRIP exploits the sinusoidal tilting pose of the landing plane and calculates the time period of the tilting motion. Based on this time period, the proposed FTDVS control strategy enables the quadrotor to search for suitable landing windows on a tilting target plane. The proposed finite-time controller converges the tracking errors to zero in finite-time during a landing maneuver. First, a rigorous stability analysis of the FTDVS control law is presented, followed by simulations demonstrating acceptable performance in terms of tracking accuracy when compared to existing controllers for autonomous landing on moving targets. The proposed method has been experimentally validated in an indoor environment.

Depth and ego-motion estimations are essential for the localization and navigation of autonomous robots and autonomous driving. Recent studies make it possible to learn the per-pixel depth and ego-motion from the unlabeled monocular video. In this paper, a novel unsupervised training framework is proposed with 3D hierarchical refinement and augmentation using explicit 3D geometry. In this framework, the depth and pose estimations are hierarchically and mutually coupled to refine the estimated pose layer by layer. The intermediate view image is proposed and synthesized by warping the pixels in an image with the estimated depth and coarse pose. Then, the residual pose transformation can be estimated from the new view image and the image of the adjacent frame to refine the coarse pose. The iterative refinement is implemented in a differentiable manner in this paper, making the whole framework optimized uniformly. Meanwhile, a new image augmentation method is proposed for the pose estimation by synthesizing a new view image, which creatively augments the pose in 3D space but gets a new augmented 2D image. The experiments on KITTI demonstrate that our depth estimation achieves state-of-the-art performance and even surpasses recent approaches that utilize other auxiliary tasks. Our visual odometry outperforms all recent unsupervised monocular learning-based methods and achieves competitive performance to the geometry-based method, ORB-SLAM2 with back-end optimization. The source codes will be released soon at:https://github.com/IRMVLab/HRANet.

Multirobot path planning leads multiple robots from start positions to designated goal positions by generating efficient and collision-free paths. Multirobot systems realize coordination solutions and decentralized path planning, which is essential for large-scale systems. The state-of-the-art decentralized methods utilize imitation learning and reinforcement learning methods to teach fully decentralized policies, dramatically improving their performance. However, these methods cannot enable robots to perform tasks efficiently in relatively dense environments without communication between robots. We introduce the transformer structure into policy neural networks for the first time, dramatically enhancing the ability of policy neural networks to extract features that facilitate collaboration between robots. It mainly focuses on improving the performance of policies in relatively dense multirobot environments under conditions where robots do not communicate with each other. Furthermore, a novel imitation reinforcement learning framework is proposed by combining contrastive learning and double deep Q-network to solve the problem of difficulty training policy neural networks after introducing the transformer structure. We present results in the simulation environment and compare the resulting policy against advanced multirobot path-planning methods in terms of success rate. Simulation results show that our policy achieves state-of-the-art performance when there is no communication between robots. Finally, we experimented with a real-world case using a total of three robots in our robotic laboratory.

3-D scene flow characterizes how the points at the current time flow to the next time in the 3-D Euclidean space, which possesses the capacity to infer autonomously the nonrigid motion of all objects in the scene. The previous methods for estimating scene flow from images have limitations, which split the holistic nature of 3-D scene flow by estimating optical flow and disparity separately. Learning 3-D scene flow from point clouds also faces the difficulties of the gap between synthesized and real data and the sparsity of LiDAR point clouds. In this article, the generated dense depth map is utilized to obtain explicit 3-D coordinates, which achieves direct learning of 3-D scene flow from 2-D images. The stability of the predicted scene flow is improved by introducing the dense nature of 2-D pixels into the 3-D space. Outliers in the generated 3-D point cloud are removed by statistical methods to weaken the impact of noisy points on the 3-D scene flow estimation task. Disparity consistency loss is proposed to achieve more effective unsupervised learning of 3-D scene flow. The proposed method of self-supervised learning of 3-D scene flow on real-world images is compared with a variety of methods for learning on the synthesized dataset and learning on LiDAR point clouds. The comparisons of multiple scene flow metrics are shown to demonstrate the effectiveness and superiority of introducing pseudo-LiDAR point cloud to scene flow estimation.

Physical human–robot interaction in the field of aerial vehicles has received more research attention in recent years. In this article, a visual impedance control strategy for human–aerial robot cooperative transportation with a tethered vehicle is presented. Without a positioning system, the aerial vehicle is controlled to follow the human partner by using cable force and visual features of the object as feedback. Furthermore, being aware of human motion is important to improve efficiency and smoothness of the cooperation. Without measuring velocities of the aerial vehicle and the human, we propose to directly estimate the relative velocity of them by a vision-based velocity observer. This estimated velocity is then integrated into a visual impedance scheme. The stability of the system is rigorously proved by Lyapunov analysis and passivity analysis. Indoor experiments where a human participant transports a long bar with a tethered aerial vehicle are conducted. Results of experiments under different human velocities and intentions demonstrate the effectiveness and reliability of the proposed method.

Mobile robot autonomous navigation in large-scale environments with crowded dynamic objects and static obstacles is still an essential yet challenging task. Recent works have demonstrated the potential of using deep reinforcement learning to enable autonomous navigation in crowds. However, only considering the human-robot interactions results in short-sighted and unsafe behaviors, and they typically use hand-crafted features and assume the global observation range, leading to large performance declines in large-scale crowded environments. Recent advances have shown the power of graph neural networks to learn local interactions among surrounding objects. In this paper, we consider autonomous navigation task in large-scale environments with crowded static and dynamic objects (such as humans). Particularly, local interactions among dynamic objects are learned for better-understanding their moving tendency and relational graph learning is introduced for aggregating both the object-object interactions and object-robot interactions. In addition, local observations are transformed into graphical inputs to achieve the scalability to various number of surrounding dynamic objects and various static obstacle patterns, and the globally guided reinforcement learning strategy is introduced to achieve the fixed-sized learning model even in large-scale complex environments. Simulation results validate our generalizability to various environments and advanced performance compared with existing works in large-scale crowded environments. In particular, our method with only local observations performs better than the benchmarks with global complete observability. Finally, physical robotic experiments demonstrate our effectiveness and practical applicability in real scenarios.

Multiple Object Tracking (MOT) is a significant task in autonomous driving. Nonetheless, relying on one single sensor is not robust enough, because one modality tends to fail in some challenging situations. Texture information from RGB cameras and 3D structure information from Light Detection and Ranging (LiDAR) have respective advantages under different circumstances. Therefore, feature fusion from multiple modalities contributes to the learning of discriminative features. However, it is nontrivial to achieve effective feature fusion due to the completely distinct information modality. Previous fusion methods usually fuse the top-level features after the backbones extract the features from different modalities. The feature fusion happens solely once, which limits the information interaction between different modalities. In this paper, we propose multi-scale interactive query and fusion between pixel-wise and point-wise features to obtain more discriminative features. In addition, an attention mechanism is utilized to conduct soft feature fusion between multiple pixels and points to avoid inaccurate match problems of previous single pixel-point fusion methods. We introduce PointNet++ to obtain multi-scale deep representations of point clouds and make it adaptive to our proposed interactive feature fusion between multi-scale features of images and point clouds. Through the interaction module, each modality can integrate more complementary information from the other modality. Besides, we explore the effectiveness of pre-training on each single modality and fine-tuning on the fusion-based model. Our method can achieve 90.32% MOTA and 72.44% HOTA on the KITTI benchmark and outperform other approaches without using multi-scale soft feature fusion.

Exact position control of the flexible-joint manipulator (FJM) is a challenging task due to the manipulator’s nonlinearity and underactuated characteristic. For the three-dimensional (3-D) rigid-link FJM, the image-based visual servoing (IBVS) approach with link-position output constraint and unknown camera parameters is investigated in this article. The controller is designed to deal with three problems. First, the visual servoing control law is divided into two domains based on the singular perturbation method, where the control input for the fast subsystem is developed to damp out the flexible joint’s vibration. Second, the adaptive updating law for estimating the unknown camera parameters is presented in the slow subsystem. Third, to guarantee that the rigid-link position keep in set constraints, the controller for the slow subsystem is designed by introducing a barrier Lyapunov function. According to the Lyapunov theorem of stability, the proposed controller for the FJM is theoretically proved to be asymptotically stable. Several numerical simulation experiments are provided to illustrate that the presented control scheme is effective.

Learning from demonstration holds the promise of enabling robots to learn diverse actions from expert experience. In contrast to learning from observation-action pairs, humans learn to imitate in a more flexible and efficient manner: learning behaviors by simply “watching.” In this article, we propose a “watch-and-act” imitation learning pipeline that endows a robot with the ability of learning diverse manipulations from visual demonstrations. Specifically, we address this problem by intuitively casting it as two subtasks: 1) understanding the demonstration video and 2) learning the demonstrated manipulations. First, a captioning module based on visual change is presented to understand the demonstration by translating the demonstration video into a command sentence. Then, to execute the captioning command, a manipulation module that learns the demonstrated manipulations is built upon an instance segmentation model and a manipulation affordance prediction model. We validate the superiority of the two modules over existing methods separately via extensive experiments and demonstrate the whole robotic imitation system developed based on the two modules in diverse scenarios using a real robotic arm. Supplementary video is available athttps://vsislab.github.io/watch-and-act/.

In this paper, we propose a novel network, Fusion-Net, which can estimate the extrinsic calibration matrix between LiDAR and a monocular RGB camera with high accuracy and robustness. FusionNet is a coarse-to-fine method, providing an online and end-to-end solution that can automatically detect and correct the decalibration without any specially designed targets or environments. First, the network applies deep-learning-based technologies to extract the features of LiDAR point clouds and RGB images. Then a novel method is adopted to fuse the features got from different sensors by projecting LiDAR features onto RGB feature maps, searching for the RGB features with the projected points as centers and concatenating the extracted RGB features with LiDAR features. To increase the accuracy, we apply a coarse-to-fine method in the network, by transforming LiDAR points and estimating the extrinsic calibration matrices from the coarse scale to the fine scale. The network is trained on random artificial decalibration matrices. Compared to existing approaches, our method doesn't need to train additional iterative networks, but it can also adapt to different ranges of decalibration.

Regular maintenance of pipelines is an important task to ensure oil transportation and other operation (sewers, nature gas). Precise localization of pipeline damage can greatly improve the efficiency of maintenance work. Since the texture similarity and illumination change of pipe, traditional local descriptors for image matching like SIFT, SURF and ORB are easy to suffer from false correspondences. As to remove the false matches, the local neighborhood constraints (LNC) that contain spatial constructs around feature points are proposed. Good correspondences are essential for the high-accuracy localization and mapping solution given the limited textures and illumination in the pipes. The LNC method is also integrated into the state-of-the-art visual SLAM system. The proposed LNC image matching method and the SLAM system are evaluated on datasets gathered from the pipe environment. Compared with other state-of-the-art methods, our LNC image matching method achieves similar or better performance in precision, recall and runtime. The SLAM system provides state estimation and map reconstruction of the pipe in real-time, and the localization error is within 1%.

This article investigates the multirobot cooperative navigation problem based on raw visual observations. A fully end-to-end learning framework is presented, which leverages graph neural networks to learn local motion coordination and utilizes deep reinforcement learning to generate visuomotor policy that enables each robot to move to its goal without the need of environment map and global positioning information. Experimental results show that, with a few tens of robots, our approach achieves comparable performance with the state-of-the-art imitation learning-based approaches with bird-view state inputs. We also illustrate our generalizability to crowded and large environments and our scalability to ten times number of the training robots. In addition, we demonstrate that our model trained for multirobot case can also improve the success rate in the single-robot navigation task in unseen environments. Note to Practitioners—With the development of intelligent industrial and logistic systems, robotic transportation systems are widely implemented. However, existing multirobot path coordination and navigation approaches are basically under some unreasonable assumptions, which are very hard to be implemented in practical scenarios. This article aims to greatly promote the real application of learning-based multirobot cooperative navigation approach, in order to achieve the following. First, we introduce an end-to-end reinforcement learning framework instead of the commonly used imitation learning strategy, as the latter one needs exhaustive training data to cover all the scenarios and does not have the required generalizability. Second, we directly use the raw sensor data instead of the commonly used bird-eye-view semantic observations, as the latter one is generally not representative of practical application scenario from the robot perspective and cannot solve the occlusion issue. Third, we interpret our learned model to illustrate which parts of the input and shared observations contribute most to the robots’ final actions. The above interpretability ensures predictability (thus safety) of our visuomotor policy in practical applications. Our learned visuomotor policy has the ability to coordinate dozens of robots by only using raw visual observations in unknown environments without map nor global localization information, this is the first time in the literature. Our future work includes solving the sim-to-real issue and conducting physical experiments.

Artificial intelligence-enhanced autonomous unmanned systems, such as large-scale autonomous robot networks, are widely used in logistic and industrial applications. In this article, we address the integrated task assignment, path planning, and coordination problem applied for large-scale robot networks with the existence of uncertainties. In particular, a novel generalized conflict graph is designed which encodes the traveling time cost of the subsequent path planning result of each task-robot assignment and also includes the predicted path conflicts of each two assignments. An integrated optimization problem which aims to minimize the total traveling cost and potential path conflicts simultaneously is first formulated and then transformed into a linear programming instance to obtain the optimal solution. In particular, to satisfy the real-time requirement in large-scale systems, a greedy solution is presented which has the near-optimal performance but can decrease the computational complexity by orders of magnitude. The optimality, scalability, robustness, and efficiency of our approach are demonstrated by comprehensive comparisons with existing state-of-the-art approaches. Note to Practitioners—With the development of artificial intelligence techniques, large-scale autonomous robot networks are increasingly used in the logistic warehouses, unmanned container terminals, and intelligence transportation systems. This article considers the large-scale networks with hundreds or even thousands of unmanned robots which are implemented in lifelong transportation systems with uncertainties existed in practical execution process. Our main concept is to simultaneously minimize the total time cost of all the tasks and the potential motion conflicts among all the robots in the subsequent execution stage, thus alleviating robot congestions, balancing traffic distributions, increasing system efficiency, and improving the robustness and scalability. Lifelong simulations with thousand robots illustrate that our approach can reduce more than 30% of the time steps consumed for coordinating robot motion conflicts, and in the meantime, the throughput, and overall system efficiency are improved. However, simulation results show that the system improvement decreases in the presence of extreme high uncertainties (such as temporary motion and communication failures of the robot), due to the inaccurate conflict prediction in the integrated optimization stage. Our future work includes the deep-learning-based traffic evolution prediction and the online reallocation and planning in highly dynamic scenarios.

Visual servoing and vibration suppression of spatial flexible-link manipulators with a fixed camera setup are addressed in this article. The singular perturbation method is adopted to decouple the dynamic equations of the flexible manipulator; hence, two subsystems that represent the rigid robot motion and flexible-link vibration are obtained, respectively. Then, for the slow subsystem related to the rigid motion, an image-based controller is designed to converge the image errors with the consideration of compensating for the errors of approximating the Jacobian matrix. For the fast subsystem corresponding to the elastic vibration, to eliminate the requirements of measuring the vibration states, an observer is designed to estimate the fast states and then a feedback controller of the fast subsystem is presented to suppress the vibration of the flexible manipulator by using the estimation values. The closed-loop stabilities of the slow and fast subsystem are both proved by employing the Lyapunov theory. Numerical simulations demonstrate the effectiveness of the proposed controller, which shows that the image errors approach zero with the vibration of the flexible manipulator damped out simultaneously.

We consider the uncalibrated vision-based control problem of robotic manipulators in this work. Though lots of approaches have been proposed to solve this problem, they usually require calibration (offline or online) of the camera parameters in the implementation, and the control performance may be largely affected by parameter estimation errors. In this work, we present new fully uncalibrated visual servoing approaches for position control of the 2DOFs planar manipulator with a fixed camera. In the proposed approaches, no camera calibration is required, and numerical optimization algorithms or adaptive laws for parameter estimation are not needed. One benefit of such features is that exponential convergence of the image position errors can be ensured regardless of the camera parameter uncertainties. Generally, existing uncalibrated approaches only can guarantee asymptotical convergence of the position errors. Moreover, different from most existing approaches which assume that the robot motion plane and the image plane are parallel, one of the proposed approaches allows the camera to be installed at a general pose. This also simplifies the controller implementation and improves the system design flexibility. Finally, simulation and experimental results are provided to illustrate the effectiveness of the presented fully uncalibrated visual servoing approaches.

The semantic segmentation of point clouds is an important part of the environment perception for robots. However, it is difficult to directly adopt the traditional 3-D convolution kernel to extract features from raw 3-D point clouds because of the unstructured property of point clouds. In this article, a spherical interpolated convolution operator is proposed to replace the traditional grid-shaped 3-D convolution operator. In addition, this article analyzes the defect of point cloud interpolation methods based on the distance as the interpolation weight and proposes the self-learned distance-feature density by combining the distance and the feature correlation. The proposed method makes the feature extraction of the spherical interpolated convolution network more rational and effective. The effectiveness of the proposed network is demonstrated on the 3-D semantic segmentation task of point clouds. Experiments show that the proposed method achieves good performance on the ScanNet dataset and Paris-Lille-3D dataset. The comparison experiments with the traditional grid-shaped 3-D convolution operator demonstrated that the newly proposed feature extraction operator improves the accuracy of the network and reduces the parameters of the network. The source codes will be released on https://github.com/IRMVLab/SIConv.

In this article, the finite-time cooperative control problem for leader-following bearing-defined formation tracking of multiagent systems with double-integrator dynamics is investigated. Different from the existing works on finite-time containment control, our objective is to make followers track leaders' trajectories and form a shape-preserving formation rather than a convex hull. The target formation is defined by both leaders' motions and bearing constraints among neighboring agents, which enables the formation not only to form and preserve a geometric pattern but also to have the ability to achieve both translational and scaling formation maneuver. To satisfy the bearing constraints, a matrix-weighted estimator/controller is developed. The finite-time stabilization of the target formation is achieved, though the matrix-weighted design makes the stability analysis complicated. Finally, an illustrative example is presented to demonstrate the effectiveness.

With the rapid development of autonomous vehicle technologies, how to perform high-precision localization in unknown complex outdoor environment has become an important issue. Visual odometry is one of the low-cost and the most widely utilized localization methods. Traditional methods predict relative pose based on the principle of multi-view geometry, which is sensitive to camera parameters and environmental changes. This paper studies deep learning-based methods which can be more robust. A novel end-to-end unsupervised visual odometry framework based on confidence evaluation is proposed. Its process can be divided into two stages. The first is predicting the initial relative pose transformation with the help of confidence mask which is generated by measuring the relative similarity of geometric corresponding regions in associated images. The second is evaluating the confidence of the output pose estimate based on the trajectory geometric consistency and then refining it. Quantitative and qualitative evaluation of the proposed approach on KITTI dataset are presented to demonstrate its effectiveness in improving pose estimation accuracy and robustness.

The research on loop closure detection has been carried out for the last many years; however, loop closure detection efficiency is still not that good and has been affected by many factors, including illumination conditions, weather conditions, seasons, and viewpoint changes. The research on loop closure detection from a different viewpoint is still an open research problem. The paper proposes an efficient solution to loop closure detection from different viewpoints by using landmarks instead of whole frames and taking the deep and robust features of deep learning instead of handcrafted features. A different kind of training approach is used to train the deep CNN to get highly abstract embeddings of input landmarks. The approach has many advantages over the traditional training approach and can solve many complex problems. This paper has used this approach to force similar or closer embeddings for similar landmarks and is forced to have a large gap in embeddings of different landmarks. The proposed method endeavors viewpoint invariant features, and the astonishing power of deep learning makes the features robust to viewpoint changes, occlusions, and illumination variations. The proposed visual SLAM system is tested on six publicly available datasets, and the results are compared with the most popular Bag of Words methods like DBoW2, DBoW3, and state-of-the-art deep learning methods AlexNet, ResNeXt, FlyNet, AFDPR and Siamese network. The results show that our method is efficient in finding loop closures candidates from different viewpoints. Code is available athttps://github.com/IRMVLab/Step-wise-Learning.

Optical flow estimation is a fundamental problem of computer vision and has many applications in the fields of robot learning and autonomous driving. This paper reveals novel geometric principles of optical flow based on the insight and detailed definition of non-occlusion. Then, two novel loss functions are proposed for the unsupervised learning of optical flow based on the geometric principles of non-occlusion. Specifically, after the occlusion part of the images are masked, the flowing process of pixel coordinate points is carefully considered and geometric constraints are conducted based on the geometric principles of optical flow. First, the optical flow vectors of neighboring pixel coordinate points in the first frame will not intersect during the pixel displacement to the second frame. Secondly, when the cluster containing adjacent four pixel coordinate points in the first frame moves to the second frame, no other pixel coordinate points will flow into the quadrilateral formed by them. According to the two geometrical constraints, the optical flow non-intersection loss and the optical flow non-blocking loss in the non-occlusion regions are proposed. Two loss functions punish the irregular and inexact optical flows in the non-occlusion regions. The experiments on datasets demonstrated that the proposed unsupervised losses of optical flow based on the geometric principles in non-occlusion regions make the estimated optical flow more refined in detail, and improve the performance of unsupervised learning of optical flow. In addition, the experiments training on synthetic data and evaluating on real data show that the generalization ability of optical flow network is improved by our proposed unsupervised approach. Our codes are open sourced at:https://github.com/IRMVLab/NccFlow.

In autonomous driving, monocular sequences contain lots of information. Monocular depth estimation, camera ego-motion estimation and optical flow estimation in consecutive frames are high-profile concerns recently. By analyzing tasks above, pixels in the middle frame are modeled into three parts: the rigid region, the non-rigid region, and the occluded region. In joint unsupervised training of depth and pose, we can segment the occluded region explicitly. The occlusion information is used in unsupervised learning of depth, pose and optical flow, as the image reconstructed by depth-pose and optical flow will be invalid in occluded regions. A less-than-mean mask is designed to further exclude the mismatched pixels interfered with by motion or illumination change in the training of depth and pose networks. This method is also used to exclude some trivial mismatched pixels in the training of the optical flow network. Maximum normalization is proposed for depth smoothness term to restrain depth degradation in textureless regions. In the occluded region, as depth and camera motion can provide more reliable motion estimation, they can be used to instruct unsupervised learning of optical flow. Our experiments in KITTI dataset demonstrate that the model based on three regions, full and explicit segmentation of the occlusion region, the rigid region, and the non-rigid region with corresponding unsupervised losses can improve performance on three tasks significantly. The source code is available at:https://github.com/guangmingw/DOPlearning.

In this article, to solve trajectory tracing problem and vibration suppression for a planar two-link rigid-flexible manipulator subject to joint-velocity measurement noise, a novel uncalibrated visual servoing control is proposed. To begin with, the manipulator’s dynamic model is established by the assumed mode method (AMM). On this basis, based on the singular perturbation theory, two subsystem controllers are designed, one is slow subsystem controller, and the other one is fast subsystem controller. In the slow subsystem, to cope with the complication of the camera calibration, an adaptive algorithm is formulated to evaluate the parameters of a fixed camera online. Aiming to overcome the challenge that exact joint-velocity measurement may be disturbed by external noise, a nonlinear sliding observer is developed to estimate the state of joint velocity accurately. The asymptotic convergence of image tracking error is proved by means of Lyapunov analysis. Additionally, for the purpose of restraining the flexible beam’s elastic vibration, a linear quadratic regulator (LQR) approach is adopted in the fast subsystem control design. The realistic comparing simulation experiments are presented to demonstrate the performance of the proposed controller.

A novel 3D point cloud learning model for deep LiDAR odometry, named PWCLO-Net, using hierarchical embedding mask optimization is proposed in this paper. In this model, the Pyramid, Warping, and Cost volume (PWC) structure for the LiDAR odometry task is built to refine the estimated pose in a coarse-to-fine approach hierarchically. An attentive cost volume is built to associate two point clouds and obtain embedding motion patterns. Then, a novel trainable embedding mask is proposed to weigh the local motion patterns of all points to regress the overall pose and filter outlier points. The estimated current pose is used to warp the first point cloud to bridge the distance to the second point cloud, and then the cost volume of the residual motion is built. At the same time, the embedding mask is optimized hierarchically from coarse to fine to obtain more accurate filtering information for pose refinement. The trainable pose warp-refinement process is iteratively used to make the pose estimation more robust for outliers. The superior performance and effectiveness of our LiDAR odometry model are demonstrated on KITTI odometry dataset. Our method outperforms all recent learning-based methods and outperforms the geometry-based approach, LOAM with mapping optimization, on most sequences of KITTI odometry dataset. Our source codes will be released on https://github.com/IRMVLab/PWCLONet.

Estimating the 6D pose of parts is a core step for robot bin-picking tasks. The problem is that various parts are usually randomly stacked with heavy occlusion in real applications. In this work, we propose a novel method to regress 6D poses by feeding RGB-D images through a two-stage neural network. In the first stage, a network for the scenes with dense parts is proposed to get instance segmentation and corresponding bounding-box. In the second stage, we propose a novel loss function and add a special module and improve the 6D pose estimation performance of the baseline network dramatically. To solve the expensive annotation cost, a simulation method is employed to generate a synthetic dataset. In experiments, our method outperformed the baseline networks (Mask R-CNN and DenseFusion) with a remarkable gap. Besides, the grasping experiments demonstrated that the method also works well in real industrial robot bin-picking applications.

This study proposes a hybrid vision/force control scheme for interaction with the inner surface of the bottle-like object. Based on the geometry of the object, a new generalized constraint called the bottleneck (BN) constraint is proposed, which ensures the tool passes through a fixed 3-D region and avoid collisions with the boundary of the region. To realize the hybrid vision/force control under the BN constraint, a novel dynamic controller is designed inspired by the hierarchical operational space, which can complete the different tasks defined in the decoupled subspace. To enhance the robustness of the algorithm, we develop a data-driven method and an adaptive method to estimate the Jacobian matrix online in force space and image space, respectively. The asymptotic stability of the closed-loop system is rigorously proved by the Lyapunov theory. Experiments are conducted to validate the performance of the proposed method.

Scene flow represents the motion of points in the 3D space, which is the counterpart of the optical flow that represents the motion of pixels in the 2D image. However, it is difficult to obtain the ground truth of scene flow in the real scenes, and recent studies are based on synthetic data for training. Therefore, how to train a scene flow network with unsupervised methods based on real-world data shows crucial significance. A novel unsupervised learning method for scene flow is proposed in this paper, which utilizes the images of two consecutive frames taken by monocular camera without the ground truth of scene flow for training. Our method realizes the goal that training scene flow network with real-world data, which bridges the gap between training data and test data and broadens the scope of available data for training. Unsupervised learning of scene flow in this paper mainly consists of two parts: (i) depth estimation and camera pose estimation, and (ii) scene flow estimation based on four different loss functions. Depth estimation and camera pose estimation obtain the depth maps and camera pose between two consecutive frames, which provide further information for the next scene flow estimation. After that, we used depth consistency loss, dynamic-static consistency loss, Chamfer loss, and Laplacian regularization loss to carry out unsupervised training of the scene flow network. To our knowledge, this is the first paper that realizes the unsupervised learning of 3D scene flow from monocular camera. The experiment results on KITTI show that our method for unsupervised learning of scene flow meets great performance compared to traditional methods Iterative Closest Point (ICP) and Fast Global Registration (FGR). The source code is available at: https://github.com/IRMVLab/3DUnMonoFlow.

Soft robots have been applied widely to various constrained scenarios due to the advantages over traditional rigid manipulators such as softness, deformability and adaptability to constrained surroundings. To make full use of this merit, this paper proposes a method that integrates collision detection, localization and force estimation for a cable-driven soft manipulator without any prior geometrical knowledge of its surroundings. First of all, a collision detection algorithm is presented based upon Cosserat-rod statics by a threshold method through using the cable tension and the shape information, which are obtained by the load cells and the Vicon system, respectively. Secondly, a collision localization and force estimation method is proposed through optimizing the discrepancy between the actual and the theoretical shapes. Finally, experiments are carried out to validate these algorithms. The experimental results demonstrate that the site, the magnitude as well as the direction can be estimated.

In this paper, the distributed rendezvous control problem of networked uncertain robotic systems with bearing measurements is investigated. The network topology of the multi-robot systems is described by an undirected graph. The dynamics of robots is modeled by Euler-Lagrange equation with unknown inertial parameters, which is more general than simple kinematics considered in existing works on rendezvous problem of multi-robot systems. To achieve rendezvous, a distributed adaptive force/torque control law is developed for each robot, which uses bearings with respect to its neighbors instead of relative displacements or distances. It is shown that the resulting closed-loop multi-robot systems are globally asymptotically stable. Then, the rendezvous control problem of multiple wheeled mobile robots is further solved by the proposed approach. Finally, on-site experiment on networked TurtleBot3 Burger mobile robots is conducted and the results demonstrate effectiveness of the proposed approach.

Timely fault detection and identification (FDI) of soft manipulators are critical in the design of surgical systems to improve reliability. However, due to the intrinsic compliance of soft manipulators, their end effectors vibrate during the dynamic control process, which introduces noise into the measured signals and makes FDI of soft manipulators challenging. This paper proposes a novel method to accomplish these tasks based on Long Short Term Memory (LSTM) recurrent neural network. Based on LSTM network, a new Attention-based Noise Compensation (ANC) module is proposed to enable the network to filter the noise merged with signals input in a self-supervision manner. Moreover, weighted cross entropy loss is introduced to balance the normal and faulty samples in the training set. Of the 9930 samples presented to the model, 9489 are correctly diagnosed in less than 1.0 second, which implies that the method can learn the spatial and temporal dependence of the signals and distinguish the healthy modes from the faulty ones. Finally, we compare the ANC-based method with the vanilla LSTM method and the state-of-art Bruin et al. method. From the comparison, we conclude that the ANC-based method proposed in this paper not only shortens the time cost of the FDI process but also suppresses the sensitivity of diagnosis results to noise. Source code, pre-trained models and dataset are available on https://github.com/IRMVLab/ANC-LSTM-fault-detection.

This paper puts forward a novel sensor-less fault detection method with only task errors feedback and applies it to visual servoing tasks of soft robot manipulators. The method is developed by introducing a suitably designed endogenous accessory signal (EAS). On the one hand, EAS transforms the change of jacobian matrix led by faults into the change of task errors, which enables the fault to be directly measured and detected; on the other hand, EAS adjusts the state trajectories according to the distance between states and their boundaries, so that state saturation is avoided. To enhance the robustness of the method, we introduce an artificial potential field that keeps the states from the undesired hyperplanes that lead to the loss of effectiveness of the method. Due to the uncalibrated feature point, its coordinates used in control laws and artificial potential filed are unknown. An adaptive algorithm is developed to guarantee the stability of the system and the convergence of the image errors. Experiments are conducted to validate the performance of the proposed method in both healthy and faulty systems.

In the field of large-scale SLAM for autonomous driving and mobile robotics, 3D point cloud based place recognition has aroused significant research interest due to its robustness to changing environments with drastic daytime and weather variance. However, it is time-consuming and effort-costly to obtain high-quality point cloud data for place recognition model training and ground truth for registration in the real world. To this end, a novel registration-aided 3D domain adaptation network for point cloud based place recognition is proposed. A structure-aware registration network is introduced to help to learn features with geometric information and a 6-DoFs pose between two point clouds with partial overlap can be estimated. The model is trained through a synthetic virtual LiDAR dataset through GTA-V with diverse weather and daytime conditions and domain adaptation is implemented to the real-world domain by aligning the global features. Our results outperform state-of-the-art 3D place recognition baselines or achieve comparable on the real-world Oxford RobotCar dataset with the visualization of registration on the virtual dataset.

Estimation of the motion of an agent and its environment concurrently is done by simultaneous localization and mapping (SLAM). In the recent past, SLAM has made rapid and exciting progress and is used in different fields such as unmanned aerial vehicle (UAV), medical surgeries, and endoscopic procedures. The aim of this article is to devise a more accurate physiotherapy exercise monitoring device on the basis of analysis from eight different SLAM algorithms with criteria including power and memory consumption, CPU heat, and CPU utilization. This article provides a comprehensive evaluation on an embedded platform and is first of its kind, especially providing that SLAM systems ego-motion estimation has never been done so explicitly before. Based on the results of the prior analysis, we proposed a stereo visual-inertial tracking (S-VIT) for lower limb tracking in physiotherapy applications. Our proposed algorithm has significantly improved results compared with the state-of-the-art algorithms. Data sets of various physiotherapy rehabilitation exercises for leg are also collected for detailed validations where the ground truth is acquired with a state-of-the-art motion tracking system, Vicon.Note to Practitioners—Accurate real-time tracking in an unknown environment is a challenging task, especially if high accuracy is needed. In physiotherapy, analyzing the daily recorded data (data acquired from a patient’s body movement) will be beneficial in the process of rehabilitation. However, keeping the daily record of patient body motion during exercise is a difficult task in most of the circumstances, which is due to the nonavailability of precise portable devices for accurate motion tracking. In this article, a solution for tracking the patient’s motion (during physiotherapy) using a small hand-held device is presented. For this purpose, simultaneous localization and mapping (SLAM) is used, which, according to the best of our knowledge, is not used in the field of physiotherapy before. In the first part of this article, we present an extensive analysis of a few SLAM algorithms based on power, memory consumption, CPU heat, and CPU utilization. Based on these results, we select the best algorithm EMoVI-SLAM, and then, we extend the work of this article by modifying EMoVI-SLAM. We propose a new SLAM algorithm called stereo visual-inertial tracking (S-VIT). The proposed algorithm is compared with the EMoVI-SLAM on our data set. We collect the data set of various movements of the lower limb. The result shows that S-VIT outperforms EMoVI-SLAM.

In this article, we focus on resolving the traffic flow prediction, robot path planning, and motion coordination problems in large-scale warehousing robotics systems with thousand-robot networks. The warehousing environment is partitioned into several sectors, and a hierarchical framework is developed, which includes a centralized prediction and planning level and a decentralized local coordination level. In the centralized level, a traffic flow prediction algorithm is first proposed to predict the evolution of the robot density distribution in a future horizon and estimate the future traffic heat value of each sector. Based on this, the sector-level robot path can be generated in the time-expended sector graph by comprehensively considering the traveling distance and the predicted traffic heat value and will be dynamically updated by considering the most recent traffic information. In the coordination level, local cooperative A* algorithm, incorporated with the conflict-based searching strategy, is implemented within each sector to generate conflict-free road-level paths for all the robots in the sector simultaneously, and the rolling planning scheme is utilized in order to immediately react to robot motion uncertainties and communication disconnections. The effectiveness and practical applicability of the proposed approach are validated by large-scale simulations with more than one 1000 robots and real laboratory experiments.Note to Practitioners—Considering practical situations and requirements in industrial warehouses and automated logistics systems, this article resolves the life-long planning and coordination problems of large-scale robot networks and ensures the practical execution performance in the presence of robot motion uncertainties and temporary communication disconnections. Our main idea is to reduce robot congestions and improve warehouse working efficiency by balancing the traffic flow in the whole environment. To achieve this, we present a traffic flow prediction algorithm to estimate the robot density distribution in a future horizon and take this information into consideration in sector-level path planning. The reliability, scalability, and the real-time performance of the proposed solution are achieved by the presented hierarchical system framework and the dynamic planning scheme. The proposed concept and approach can also be used to coordinate other large-scale systems with multirobot or multi-AGV networks. Simulation and experimental results suggest that the proposed solution is effective and practically applicable, but a saturation phenomenon of the system capacity can be observed under a very heavy workload. In the future, we will investigate the relation between the maximum system capacity and the environment structure and make further efforts to optimize the environment structure and road layout in order to improve the warehouse working efficiency.

Long-Term visual localization under changing environments is a challenging problem in autonomous driving and mobile robotics due to season, illumination variance, etc. Image retrieval for localization is an efficient and effective solution to the problem. In this paper, we propose a novel multi-task architecture to fuse the geometric and semantic information into the multi-scale latent embedding representation for visual place recognition. To use the high-quality ground truths without any human effort, the effective multi-scale feature discriminator is proposed for adversarial training to achieve the domain adaptation from synthetic virtual KITTI dataset to real-world KITTI dataset. The proposed approach is validated on the Extended CMU-Seasons dataset and Oxford RobotCar dataset through a series of crucial comparison experiments, where our performance outperforms state-of-the-art baselines for retrieval-based localization and large-scale place recognition under the challenging environment.

Scene flow represents the 3D motion of every point in the dynamic environments. Like the optical flow that represents the motion of pixels in 2D images, 3D motion representation of scene flow benefits many applications, such as autonomous driving and service robot. This paper studies the problem of scene flow estimation from two consecutive 3D point clouds. In this paper, a novel hierarchical neural network with double attention is proposed for learning the correlation of point features in adjacent frames and refining scene flow from coarse to fine layer by layer. The proposed network has a new more-for-less hierarchical architecture. The more-for-less means that the number of input points is greater than the number of output points for scene flow estimation, which brings more input information and balances the precision and resource consumption. In this hierarchical architecture, scene flow of different levels is generated and supervised respectively. A novel attentive embedding module is introduced to aggregate the features of adjacent points using a double attention method in a patch-to-patch manner. The proper layers for flow embedding and flow supervision are carefully considered in our network designment. Experiments show that the proposed network outperforms the state-of-the-art performance of 3D scene flow estimation on the FlyingThings3D and KITTI Scene Flow 2015 datasets. We also apply the proposed network to the realistic LiDAR odometry task, which is a key problem in autonomous driving. The experiment results demonstrate that our proposed network can outperform the ICP-based method and shows good practical application ability. The source codes will be released on https://github.com/IRMVLab/HALFlow.

This article proposes a novel image-based visual servoing for a flexible aerial refueling boom with an eye-in-hand camera. The dynamic model of the flexible refueling boom is established by absolute nodal coordinate formulation (ANCF), after which the dynamic model is decomposed into a slow subsystem and a fast subsystem based on the singular perturbation approach. With respect to slow subsystem, the image feedback is used to control the flexible refueling boom so that the projection of the point marker on the back of the receiver converges to the desired position. With respect to fast subsystem, linear quadratic regulator (LQR) is applied to stabilize the vibration of the flexible refueling boom. The asymptotic convergence of the image error to zero is verified based on the Lyapunov theory. Simulation is used to demonstrate the effectiveness of the proposed method.

Cooperative adaptive cruise control (CACC) of intelligent vehicles contributes to improving cruise control performance, reducing traffic congestion, saving energy and increasing traffic flow capacity. In this paper, we resolve the CACC problem from the viewpoint of synchronization control, our main idea is to introduce the spatial-temporal synchronization mechanism into vehicle platoon control to achieve the robust CACC and to further realize the non-stop intersection control. Firstly, by introducing the cross-coupling based space synchronization mechanism, a distributed control algorithm is presented to achieve the single-lane CACC in the presence of vehicle-to-vehicle (V2V) communications, which enables autonomous vehicles to track the desired platoon trajectory while synchronizing their longitudinal velocities to keeping the expected inter-vehicle distance. Secondly, by designing the enter-time scheduling mechanism (temporal synchronization), a high-level intersection control strategy is proposed to command vehicles to form a virtual platoon to pass through the intersection without stopping. Thirdly, a Lyapunov-based time-domain stability analysis approach is presented. Compared with the traditional string stability based approach, the proposed approach guarantees the global asymptotical convergence of the proposed CACC system. Experiments in the small-scale simulated system demonstrate the effectiveness of the proposed approach.

Accurate, robust and real-time localization under constrained-resources is a critical problem to be solved. In this paper, we present a new sparse pose-graph visual-inertial SLAM (SPVIS). Unlike the existing methods that are costly to deal with a large number of redundant features and 3D map points, which are inefficient for improving positioning accuracy, we focus on the concise visual cues for high-precision pose estimating. We propose a novel hierarchical quadtree based optical flow tracking algorithm, it achieves high accuracy and robustness within very few concise features, which is only about one fifth features of the state-of-the-art visual-inertial SLAM algorithms. Benefiting from the efficient optical flow tracking, our sparse pose-graph optimization time cost achieves bounded complexity. By selecting and optimizing the informative features in sliding window and local VIO, the computational complexity is bounded, it achieves low time cost in long-term operation. We compare with the state-of-the-art VIO/VI-SLAM systems on the challenging public datasets by the embedded platform without GPUs, the results effectively verify that the proposed method has better real-time performance and localization accuracy.

Imitation learning enables robots to learn a task by simply watching the demonstration of the task. Current imitation learning methods usually require the learner and demonstrator to occur in the same context. This limits their scalability to practical applications. In this paper, we propose a more general imitation learning method which allows the learner and the demonstrator to come from different contexts, such as different viewpoints, backgrounds, and object positions and appearances. Specifically, we design a robotic system consisting of three models: context translation model, depth prediction model and multi-modal inverse dynamics model. First, the context translation model translates the demonstration to the context of learner from a different context. Then combining the color observation and depth observation as inputs, the inverse model maps the multi-modal observations into actions to reproduce the demonstration, where the depth observation is provided by a depth prediction model. By performing the block stacking tasks both in simulation and real world, we prove the cross-context learning advantage of the proposed robotic system over other systems.

3D Point cloud registration is still a very challenging topic due to the difficulty in finding the rigid transformation between two point clouds with partial correspondences, and it's even harder in the absence of any initial estimation information. In this paper, we present an end-to-end deep-learning based approach to resolve the point cloud registration problem. Firstly, the revised LPD-Net is introduced to extract features and aggregate them with the graph network. Secondly, the self-attention mechanism is utilized to enhance the structure information in the point cloud and the cross-attention mechanism is designed to enhance the corresponding information between the two input point clouds. Based on which, the virtual corresponding points can be generated by a soft pointer based method, and finally, the point cloud registration problem can be solved by implementing the SVD method. Comparison results in ModelNet40 dataset validate that the proposed approach reaches the state-of-the-art in point cloud registration tasks and experiment resutls in KITTI dataset validate the effectiveness of the proposed approach in real applications.

In this paper, we propose a dynamic state estimation framework for lateral control of a heavy tractor-trailers system using only mass-produced low-cost sensors. This issue is challenging since the lateral velocity of the lead tractor is difficult to measure directly. The performance of existing dynamic model-based estimation methods will also be degraded, as different trailers and payloads cause the tractor model parameters to change. We address this issue by incorporating a kinematic estimator into a dynamic model-based estimation scheme. Accurate and reliable tire cornering stiffness and dynamics-informed lateral velocity of the lead tractor can be output in real-time by using our method. The stability and robustness of the proposed method are theoretically proved. The feasibility of our method is verified by full-scale experiments. It is also verified that the estimated model parameters and lateral states do improve the control performance by integrating the estimator into a lateral control system.

To make the controller implementation easier and to enhance the system robustness and control performance in the presence of the camera parameter uncertainties, it is very desired to develop vision-based control approaches without any offline or online camera calibration. In this article, we propose a new calibration-free image-based trajectory tracking control scheme for nonholonomic mobile robots with a truly uncalibrated fixed camera. By developing a novel camera-parameter-independent kinematic model, both offline and online camera calibration can be avoided in the proposed scheme, and any knowledge of the camera is not needed in the controller design. The proposed trajectory tracking control scheme can guarantee exponential convergence of the image position and velocity tracking errors. To illustrate the performance of the proposed scheme, experimental results are provided in this article. Note to Practitioners-This article was motivated by the vision-based motion control problem of mobile robots in uncalibrated environments. Existing vision-based motion control approaches for nonholonomic mobile robots generally depend on offline precise/coarse or online numerical/adaptive calibration of the camera intrinsic and extrinsic parameters and require precise or coarse knowledge of the camera in their implementation. This article presents a novel calibration-free image-based trajectory tracking control scheme, which can be implemented easily in real environments without any offline or online calibration of the camera parameters and can be used to efficiently control the motion of nonholonomic mobile robots with an arbitrarily placed and truly unknown overhead camera. Experimental results show that the proposed scheme can achieve satisfactory trajectory tracking control performance despite the lack of any knowledge about the camera intrinsic and extrinsic parameters and the presence of unknown camera lens distortions, and hence, can provide a simple but efficient solution to the vision-based motion control problem of nonholonomic mobile robots.

Although many vision-based control methods have been proposed for nonholonomic mobile robots, in their implementation, it is usually necessary to calibrate the camera intrinsic and/or extrinsic parameters using offline/online parameter estimation algorithms or online adaptation laws. To avoid the tediousness of camera calibration and to make the system performance highly robust to camera parameter uncertainties, in this article, we propose novel image-based pose stabilization control approaches for nonholonomic mobile robots with a truly uncalibrated overhead fixed camera. In the proposed approaches, only image position information of three feature points from an overhead camera is used for controller design, while information from other sensors (such as wheel encoders) is not required. Furthermore, either offline or online camera calibration is not necessary, and no knowledge about the camera intrinsic and extrinsic parameters is needed, which also can greatly simplify the controller implementation. Simulation and experimental results are given to demonstrate the feasibility and effectiveness of the proposed purely image-based pose stabilization approaches.

In this paper, we consider the self-repairing problem from the viewpoint of robotics and our objective is not only to restore the logical network topology but also to maintain the motion synchronization of the physical mobile robot formation. A gradient-based self-repairing algorithm which only relies on the local interactions among coupling robots is presented. More specifically, aiming to optimize the repair path in a distributed manner, a gradient generation and diffusion mechanism is presented first, which can generate a stable gradient distribution in the robot formation. Then, based on the recursive self-repairing technique and the proposed gradient distribution, several self-repairing rules as well as the corresponding individual control method are presented to solve the self-repairing problem. The improvement of the proposed algorithm on the motion synchronism of the robot formation and the optimality of the selected repair path are proved by theoretical analyses. Finally, the effectiveness and the practical applicability of the proposed algorithm are validated by simulations and real experiments.

Point cloud based place recognition is still an open issue due to the difficulty in extracting local features from the raw 3D point cloud and generating the global descriptor, and it's even harder in the large-scale dynamic environments. In this paper, we develop a novel deep neural network, named LPD-Net (Large-scale Place Description Network), which can extract discriminative and generalizable global descriptors from the raw 3D point cloud. Two modules, the adaptive local feature extraction module and the graph-based neighborhood aggregation module, are proposed, which contribute to extract the local structures and reveal the spatial distribution of local features in the large-scale point cloud, with an end-to-end manner. We implement the proposed global descriptor in solving point cloud based retrieval tasks to achieve the large-scale place recognition. Comparison results show that our LPD-Net is much better than PointNetVLAD and reaches the state-of-the-art. We also compare our LPD-Net with the vision-based solutions to show the robustness of our approach to different weather and light conditions.

In this paper, we study the cooperative path planning and motion coordination problems of the multi-robot system with large number of robots, aiming for practical applications in robotic warehouses and automated transportation systems. Particularly, we solve the life-long planning problem and guarantee the coordination performance in the presence of robot motion uncertainties. A hierarchical path planning and motion coordination structure is presented. The environment is divided into several sectors and a traffic heat-map is presented to describe the current sector-level traffic condition. In path planning level, the sector-level path is calculated by considering the path distance, the current traffic condition and the current robot uncertainty. In motion coordination level, local cooperative A* algorithm and conflict-based searching strategy are utilized within each sector to generate the collision-free local path of each robot in a rolling planning manner. The effectiveness and practical applicability of the proposed approach are validated by simulations with more than one thousand robots and real experiments.

A new unsupervised learning method of depth and ego-motion using multiple masks from monocular video is proposed in this paper. The depth estimation network and the ego-motion estimation network are trained according to the constraints of depth and ego-motion without truth values. The main contribution of our method is to carefully consider the occlusion of the pixels generated when the adjacent frames are projected to each other, and the blank problem generated in the projection target imaging plane. Two fine masks are designed to solve most of the image pixel mismatch caused by the movement of the camera. In addition, some relatively rare circumstances are considered, and repeated masking is proposed. To some extent, the method is to use a geometric relationship to filter the mismatched pixels for training, making unsupervised learning more efficient and accurate. The experiments on KITTI dataset show our method achieves good performance in terms of depth and ego-motion. The generalization capability of our method is demonstrated by training on the low-quality uncalibrated bike video dataset and evaluating on KITTI dataset, and the results are still good.

Most existing controllers for Car-Like Mobile Robots (CLMR) are designed to handle dynamic effects by decoupling speed and steering controls, also assume that full states are accessible, which are unrealistic for real-world applications. This paper presents a combined speed and steering control system for CLMR. To provide the essential state for the controller, a newly developed visual algorithm is adopted for estimating the high-update rate longitudinal and lateral velocities of the robot which cannot be accurately measured by wheel encoders due to the skidding and slipping effects. The stability of the proposed system can be guaranteed by Lyapunov method since the velocity estimation error, the speed tracking error and the lateral deviation converging to zero simultaneously. Real-world experiments are conducted on an electric autonomous tractor with online estimation to demonstrate the feasibility of the approach.

Visual localization is a crucial problem in mobile robotics and autonomous driving. One solution is to retrieve images with known pose from a database for the localization of query images. However, in environments with drastically varying conditions (e.g. illumination changes, seasons, occlusion, dynamic objects), retrieval-based localization is severely hampered and becomes a challenging problem. In this paper, a novel domain-invariant feature learning method (DIFL) is proposed based on ComboGAN, a multi-domain image translation network architecture. By introducing a feature consistency loss (FCL) between the encoded features of the original image and translated image in another domain, we are able to train the encoders to generate domain-invariant features in a self-supervised manner. To retrieve a target image from the database, the query image is first encoded using the encoder belonging to the query domain to obtain a domain-invariant feature vector. We then preform retrieval by selecting the database image with the most similar domain-invariant feature vector. We validate the proposed approach on the CMU-Seasons dataset, where we outperform state-of-the-art learning-based descriptors in retrieval-based localization for high and medium precision scenarios.

Place recognition and loop-closure detection are main challenges in the localization, mapping and navigation tasks of self-driving vehicles. In this paper, we solve the loop-closure detection problem by incorporating the deep-learning based point cloud description method and the coarse-to-fine sequence matching strategy. More specifically, we propose a deep neural network to extract a global descriptor from the original large-scale 3D point cloud, then based on which, a typical place analysis approach is presented to investigate the feature space distribution of the global descriptors and select several super keyframes. Finally, a coarse-to-fine strategy, which includes a super keyframe based coarse matching stage and a local sequence matching stage, is presented to ensure the loop-closure detection accuracy and real-time performance simultaneously. Thanks to the sequence matching operation, the proposed approach obtains an improvement against the existing deep-learning based methods. Experiment results on a self-driving vehicle validate the effectiveness of the proposed loop-closure detection algorithm.

In this paper, we propose a novel pose optimizer which can be inserted into either supervised or unsupervised end-to-end visual odometry for the purpose of local pose optimization. The pose optimizer is an analogue of the pose graph optimization used in traditional VSLAM algorithms. Local pose optimization is performed by an attention-based neural network which iteratively refines the predicted pose estimates of an image snippet. Instead of complicated graph convolutional network, the attention mechanism based on geometric consistency of trajectory constraint is utilized because pose features whose spatial distribution is not important can be flattened to vectors and then processed. The pose optimizer is aimed at improving pose estimation accuracy by redistributing errors of pose estimates. Quantitative and qualitative evaluation of the proposed approach on the KITTI Odometry dataset [1] is presented to demonstrate its effectiveness in improving pose estimation accuracy and minimizing pose drift.

With the advance in the field of mobile robots, autonomous robots are required for long-term deployment in dynamic and complex environments. However, the performance of Visual Inertial SLAM systems in long-term operation is not satisfactory, and most long-term SLAM systems assumes periodic changes in the environment. This paper presents a novel solution for long-term monocular VI SLAM system in dynamic environment based on autoregression(AR) modeling and map prediction. Map points are first classified into static and semi-static map points according to a memory model. Modeling and prediction of the different states of semi-static map points are performed that are derived from time series models. The predicted map is then fused with the current map to achieve a better forecast for the next frame if the prediction is not satisfactory enough. Experiments are carried out on an embedded system. The results indicate that the map prediction is reliable and the proposed approach improves the performance of long-term localization and mapping in dynamic environments.

Learning actions from human demonstration is an emerging trend for designing intelligent robotic systems, which can be referred as video to command. The performance of such approach highly relies on the quality of video captioning. However, the general video captioning methods focus more on the understanding of the full frame, lacking of consideration on the specific object of interests in robotic manipulations. We propose a novel deep model to learn actions from human demonstration video for robotic manipulation. It consists of two deep networks, grasp detection network (GNet) and video captioning network (CNet). GNet performs two functions: providing grasp solutions and extracting the local features for the object of interests in robotic manipulation. CNet outputs the captioning results by fusing the features of both full frames and local objects. Experimental results on UR5 robotic arm show that our method could produce more accurate command from video demonstration than state-of-the-art work, thereby leading to more robust grasping performance.

Cardiovascular disease remains a primary cause of morbidity globally. Percutaneous coronary intervention plays a crucial role in the treatment. The radiation exposure of surgeons during the cardiovascular intervention can be avoided by master-slave surgical robots. This paper introduces a master- slave guidewire and catheter robotic system to protect the surgeons from X ray radiation to the most extent. And the jitters of master manipulators are mitigated by Kalman filtering algorithm. With two master manipulators, it helps to retain the surgeon's traditional operating habits. Also, a vascular model trial was conducted to validate that this interventional robotic system could complete the alternate progress and rotation of interventional guidewire and catheter.

Robot malfunction is inevitable in practical applications of the robot formation control due to uncontrolled crashing, system malfunction or communication loss. In this paper, we study the synchronous formation control problem in the presence of robot malfunctions. Our main idea is to improve the network connectivity and motion synchronism of the robot formation through a series of topology switchings and robot replacements. Firstly, the synchronous formation control method is introduced which enables the robots to tracking their desired trajectories while keeping predefined formation shapes. Secondly, a recursive switched topology control strategy is proposed to restore the formation shape as well as to improve the network connectivity and motion synchronism in the presence of robot malfunctions. Thirdly, the convergence analysis of the proposed control system is presented and a sufficient condition is obtained under an average dwell time scheme. What's more, the proposed approach is fully distributed and the communication delays between neighboring robots also have been taken into consideration. Simulation results demonstrate the effectiveness of the proposed approach.

Place recognition is one of the major challenges for the LiDAR-based effective localization and mapping task. Traditional methods are usually relying on geometry matching to achieve place recognition, where a global geometry map need to be restored. In this paper, we accomplish the place recognition task based on an end-to-end feature learning framework with the LiDAR inputs. This method consists of two core modules, a dynamic octree mapping module that generates local 2D maps with the consideration of the robot's motion; and an unsupervised place feature learning module which is an improved adversarial feature learning network with additional assistance for the long-term place recognition requirement. More specially, in place feature learning, we present an additional Generative Adversarial Network with a designed Conditional Entropy Reduction module to stabilize the feature learning process in an unsupervised manner. We evaluate the proposed method on the Kitti dataset and North Campus Long-Term LiDAR dataset. Experimental results show that the proposed method outperforms state-of-the-art in place recognition tasks under long-term applications. What's more, the feature size and inference efficiency in the proposed method are applicable in real-time performance on practical robotic platforms.

Most existing trajectory tracking controllers are based on non-skidding and non-slipping assumptions, also assume that full states are accessible, which is unrealistic for real-world applications due to tire-road interaction. This paper presents a novel vision-based approach to achieve high performance tracking control of a Car-Like Mobile Robot (CLMR) with wheel skidding and slippage. A visual estimation algorithm is proposed to provide reliable position, velocity, skidding and slipping information to close the control loop. The stability of the proposed system can be guaranteed by Lyapunov method since the position tracking error and the estimation error converge to zero simultaneously. Simulation is made to validate the effectiveness of the developed controller in the presence of skidding and slipping with online visual estimator.

Most existing formation control approaches are based on the assumption that the global/relative position and/or velocity measurements of mobile robots are directly available. To extend the application domain and to improve the formation control performance, it is extremely necessary to avoid the use of position and velocity measurements in the design of formation controllers. In this paper, we propose new leader-following formation tracking control schemes for nonholonomic mobile robots with onboard perspective cameras, without using both position and velocity measurements. To address the unavailability issue of position measurements, the leader-follower kinematics model in the image space is developed, which can facilitate the complete elimination of measurement/estimation of the position information. Furthermore, feedback information from the perspective camera of the follower robot is used to design adaptive observers to estimate the leader linear velocity for feedforward compensation, which can handle the absence of velocity measurements such that the proposed schemes can be applied to control formations of mobile robots without mutual communication abilities. By using the Lyapunov stability theory, a rigorous stability analysis based on the nonlinear formation dynamics is provided to show that the global stability of the combined observer-controller closed-loop system can be guaranteed. Both simulation and experimental results are also given to demonstrate the performance of the proposed formation tracking control schemes.

Extrinsic calibration is one of the most important problems in robotics, whose objective is to estimate the relative poses among the sensors and the robot. Currently, most on-line solutions of extrinsic calibration are based on the Gaussian filters, which estimate all the system states iteratively. However, a large number of the system states are not related to calibration. Estimating all these redundant states may highly increase the dimensionality of the problem, and therefore, undermine the calibration performance in both efficiency and robustness. In this paper, we propose an innovative on-line calibration algorithm, called constraint Gaussian filter with virtual measurements (VMCGF). The nature of VMCGF is a filter with a compact state vector containing only the states of interest, also called essential states. Violating the modeling principles of the traditional Gaussian filters, the measurements cannot be expressed by the observation function with the essential states solely. Exploiting the constraints between the measurements and the essential states, virtual measurements are generated according to the properties of the generalized chi-square distribution. Although originally developed to solve the calibration problem, VMCGF is a general filtering algorithm, that can be applied to solve other problems that might suffer from redundant states. The implementation of VMCGF on a camera odometry calibration problem is introduced, and its observability properties are analyzed. Both simulations and experiments are conducted to validate our algorithm.

This paper studies the problem of image-based leader-follower formation control for mobile robots, where the controller is designed independently of the leader's motion. An adaptive control scheme, which is suitable for both omnidirectional and perspective cameras, is proposed. The proposed approach avoids the need for accurate calibration of the extrinsic parameters of the omnidirectional camera as well as the intrinsic and extrinsic parameters of perspective camera. Additionally, the coefficients of the plane where the feature point moves relative to the camera frame can be uncertain. These uncertain constant parameters are estimated using an adaptive estimator. Uniform Semi-global Practical Asymptotic Stability (USPAS) of the system is shown using the Lyapunov approach. Experimental results are presented to demonstrate the effectiveness of the proposed control scheme.

Inspired by natural earthworm locomotion behavior and segmental muscle motion mechanism, this paper presents our recently developed compliant modular earthwormlike robot with the novel segmental muscle-mimetic design unit that is capable of efficiently mimicking earthworms' segmental muscle contraction and extension functions. The new class of segmental muscle-mimetic design unit relies on the curvature of scissor mechanisms that can be extended and contracted smoothly through controlled servo motors. The paper starts reviewing natural earthworm locomotion behavior, details the bio-inspired concept and design of both the segmental muscle-mimetic unit and the multi-segment earthworm-like robot prototype, and then presents the robot's locomotion models and analysis of the locomotion efficiency of the robot. Simulation and experimental results validate that both the design and the prototyped multi-segment earthworm-like robot have the excellent performance such as its muscle-like contractions behavior, peristaltic locomotion behavior, and highly competitive moving speed.

In this paper, we address the 3D position control of RCM-constrained instruments with monocular cameras. To compute the instrument's position from a single 2D image, we develop an innovative gradient descent algorithm which rotates and translates a line segment (over the plane spanned by the imaged instrument and the optical centre) until it best aligns with the manipulated tool. In contrast with other approaches in the literature, our algorithm only requires to simultaneously observe two feature points; the proposed iterative algorithm is not based on the exact solution, therefore it can still work with noisy image measurements. We derive a kinematic controller that uses the proposed position estimator to guide the 3D motion of a robotic instrument with a monocular camera. We evaluate the performance of our approach with numerical simulations and experiments.

Customized archwires are demanded in the lingual orthodontic treatment for patients suffering from malocclusion. Traditionally, these archwires could only be bent by experienced orthodontists manually. This pattern requires a specialized skills training and occupies long charside time, but still cannot ensure the accuracy of appliances. Therefore, a novel robotic system was developed for automatic and accurate preparation in our study. First, the implementation of hardware system was designed. Second, a modular and ROS-integrated control system was developed to control automatic bending. Third, an adaptive sampling-based bending planner with collision checker in a time-varying environment was established and realized in control system architecture. Preliminary validation of the developed robot system and its control system have been conducted both in simulation and physical robotic system. Experimental results have shown that the developed robotic system with its ROS-integrated control system was able to accomplish automatic and accurate orthodontic archwire preparation.

In this paper, we address the problem of the image-based 3D pose computation of a semi-circle suturing needle using monocular image feedback for laparoscopy. We propose a constrained two-degree-of-freedom (2-DOF) geometry-based modelling method to parametrise the needle's 6-DOF pose, including depth information. The modelling solely relies on the simultaneous observation of the needle's apparent tip and junction. No external markers are needed for extra constraints. An adaptive controller combining gradient descent and vector-flow method is introduced to iteratively guide the needle's initial guessing pose to its real pose by minimizing image-based position errors. Experiments have been conducted using both numerical simulations and simulated laparoscopic scenarios to evaluate the performance of the algorithm.

In this paper, the task-space cooperative tracking control problem of networked robotic manipulators without task-space velocity measurements is addressed. To overcome the problem without task-space velocity measurements, a novel task-space position observer is designed to update the estimated task-space position and to simultaneously provide the estimated task-space velocity, based on which an adaptive cooperative tracking controller without task-space velocity measurements is presented by introducing new estimated task-space reference velocity and acceleration. Furthermore, adaptive laws are provided to cope with uncertain kinematics and dynamics and rigorous stability analysis is given to show asymptotical convergence of the task-space tracking and synchronization errors in the presence of communication delays under strongly connected directed graphs. Simulation results are given to demonstrate the performance of the proposed approach.

This paper presents a multirobot pairwised transportation (MRPWT) approach for factory automated material and product deliveries. We consider MRPWT from the viewpoint of robotics and incorporate practical factory application constraints in the transportation method design. The proposed MRPWT approach is a two-level hybrid planning method, consisting of an incidental delivery based single robot level planner and a simulated annealing based robot group level planner. Each robot resolves its individual transportation plan incidentally to reduce the transportation cost, whereas the group level planner utilizes predefined random actions to search the task assignment solution space and then incorporates the simulated annealing algorithm to resolve the MRPWT problem as a combinatorial optimization problem. By implementing a distributed auction mechanism, the proposed MRPWT approach can be further extended to resolve the online task allocation or reallocation problem in dynamic environments. Experiments performed on a group of mobile robots successfully demonstrate the effectiveness and the practical applicability of the proposed MRPWT approach for factory automated material and product deliveries.

In this paper, we investigate the self-healing problem of mobile robot formation after some robots have been damaged, and present a gradient-based algorithm which enables mobile robots to restore the topology of the formation through local interactions among neighboring robots. Firstly, in order to optimize the repair path in a distributed manner, a gradient generation and diffusion mechanism is proposed to generate a specific gradient distribution in the formation. Then, utilizing several predefined path selection rules, a path selection algorithm is presented to guarantee the optimality of the selected repair path. Furthermore, several optimization indices are presented to quantitatively characterize the performance of self-healing algorithms. Finally, the effectiveness of the proposed algorithm is validated by numerical simulations and the simulation results show that the proposed algorithm can restore the topology of the formation with the fewer repair robots and lower energy consumptions.

In this paper, we will study the uncalibrated vision-based positioning problem of wheeled mobile robots by using a ceiling-mounted camera. A new image-based visual servoing scheme will be proposed, which can cope with the unknown intrinsic and extrinsic parameters of the camera and the uncertain distance parameter of the feature point from geometric center of the mobile robot. The presented approach is developed via extending the depth-independent interaction matrix framework for robot manipulators to mobile robots such that the nonlinear dependence on unknown parameters can be removed from the image-Jacobian matrix and then, we can linearly parameterize uncertain parameters in the closed-loop system. In this way, an estimation scheme for the online updating of uncertain parameters can be developed very efficiently. To show that image errors can be guaranteed to be asymptotically convergent, stability analysis will be carried out by using Lyapunov theory. To validate the performance of the presented approach, simulation and experimental results will also be provided.

Aim at enhancing dexterous and safe operation in unstructured environment, a cable-driven soft robotic manipulator is designed in this paper. Due to soft material it made of and nearly infinite degree of freedom it owns, the soft robotic manipulator has higher security and dexterity than traditional rigid-link manipulator, which make it suitable to perform tasks in complex environments that is narrow, confined and unstructured. Though the soft robotic manipulator possesses advantages above, it is not an easy thing for it to achieve precise position control. In order to solve this problem, a kinematic model based on piecewise constant curvature hypothesis is proposed. Through building up three spaces and two mappings, the relationship between the length variables of 4 cables and the position and orientation of the soft robotic manipulator end-effector is obtained. Afterwards, a depth-independent image Jacobian matrix is introduced and an image-based visual servo controller is presented. Applied by adaptive algorithm, the controller could estimate unknown position of the feature point online, and then Lyapunov theory is used to prove the stability of the proposed controller. At last, experiments are conducted to demonstrate rationality and validity of the kinematic model and adaptive visual servo controller.

This paper presents a new adaptive controller for visual tracking control of a robot manipulator in 3D general motion with a fixed camera whose intrinsic and extrinsic parameters are uncalibrated. In addition to camera parameters, the feature positions in 3D space are also assumed unknown. Based on the fact that the unknown parameters appears linearly in the closed-loop dynamics of the system if the depth-independent interaction matrix is adopted to map the image errors onto the joint space of the manipulator, we developed a new adaptive algorithm to estimated the unknown parameters on-line. With a full consideration of dynamic responses of the robot manipulator, we employ the Lyapunov method to prove asymptotic convergence of the image errors. Experimental results are used to demonstrate the performance of the proposed approach.

This paper presents a new controller for locking a moving object in 3-D space at a particular position (for example the center) on the image plane of a camera mounted on a robot by actively moving the camera. The controller is designed to cope with both unknown robot dynamics parameters and unknown motion of the object. Based on the fact that the unknown position of the moving object appears linearly in the closed-loop dynamics of the system if the depth-independent image Jacobian is used, we developed a nonlinear observer to estimate the 3-D motion of the object on-line and an adaptive algorithm to estimate the robot dynamic parameters. With a full consideration of dynamic responses of the robot, we employed the Lyapunov method to prove asymptotic convergence of the image errors. Experimental results are presented to support the approach in this paper.

This paper presents a novel adaptive controller for image-based visual servoing using generalized image features. The key idea lies in the development of the depth-independent interaction matrix and the proposal of an adaptive algorithm for estimating the unknown geometric parameters of the features in the 3-D space. Furthermore, we derive the conditions for the asymptotic stability of the proposed controller and demonstrate that the conditions are satisfied for six types of common image features: points, lines, distances, angles, areas, and centroids. Experiments have been conducted to validate the proposed controller.

This paper presents a fully distributed algorithm for target tracking using a mobile sensor network. It tries to maintain the target being visible to the mobile network all the time while consuming as little motion energy as possible. Meanwhile the network connectivity is maintained. At every time, only the nodes around the target are activated while other nodes keep idle. Certain functions are defined to quantify the main aspects in the tracking such as the target escaping probability and the network connectivity status. They transform the tracking into a multi-objective optimization problem. To solve this global problem, a local motion strategy is proposed. Simulation results show that our algorithm yields good performance.

This paper presents a novel adaptive controller for image-based visual servoing of robots with an uncalibrated eye-in-hand camera using line features. The controller is developed based on three key ideas. First, we propose a new method that is similar to the Plucker coordinates, to represent projections of the lines features. The new representation leads to a depth-independent image Jacobian matrix and an error vector between real images and estimated projections of the lines, which are both linear to the unknown camera parameters. Second, an adaptive algorithm is developed to estimate the unknown camera parameters and the 3-D coordinates of the lines on-line. Third, a simple controller using the depth-independent image Jacobian is designed to control the projections of the lines to desired positions and orientations. The Lyapunov theory is used to prove the asymptotic convergence of the image error to zero based on the nonlinear robot dynamics. Finally, experiments have been conducted to demonstrate the performance of the proposed approach.

This paper presents a novel approach for image-based visual servoing of a robot manipulator with an eye-in-hand camera when the camera parameters are not calibrated and the 3-D coordinates of the features are not known. Both point and line features are considered. This paper extends the concept of depth-independent interaction (or image Jacobian) matrix, developed in earlier work for visual servoing using point features and fixed cameras, to the problem using eye-in-hand cameras and point and line features. By using the depth-independent interaction matrix, it is possible to linearly parameterize, by the unknown camera parameters and the unknown coordinates of the features, the closed-loop dynamics of the system. A new algorithm is developed to estimate unknown parameters online by combining the Slotine-Li method with the idea of structure from motion in computer vision. By minimizing the errors between the real and estimated projections of the feature on multiple images captured during motion of the robot, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values up to a scale. On the basis of the nonlinear robot dynamics, we proved asymptotic convergence of the image errors by the Lyapunov theory. Experiments have been conducted to demonstrate the performance of the proposed controller.

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera. The controller is designed to cope with the case when the intrinsic and extrinsic parameters of the camera as well as the robot parameters are not calibrated. The controller employs the depth-independent image Jacobian to map the errors on the image plane onto the joint space. By using the depth-independent image Jacobian, it is possible to linearly parameterize the unknown camera parameters in the closed loop dynamics of the system. A new algorithm is developed to estimate unknown parameters on-line. We have proved asymptotic convergence of the image errors by Lyapunov method with a full consideration of dynamic responses of the robot manipulator and demonstrated the performance by experiments.

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera. It is assumed that the intrinsic and extrinsic parameters of the camera are not calibrated. A new adaptive algorithm is developed to estimate the unknown parameters online, based on three original ideas. First, we use the pseudoinverse of the depth-independent interaction matrix to map the image errors onto the joint space of the manipulator. By eliminating the depths in the interaction matrix, we can linearly parameterize the closed-loop dynamics of the manipulator. Second, to guarantee the existence of the pseudoinverse, the adaptive algorithm introduces a potential force to drive the estimated parameters away from the values that result in a singular Jacobian matrix. Third, to ensure that the estimated parameters are convergent to their true values up to a scale, we combine the Slotine-Li method with an online algorithm for minimizing the error between the estimated projections and real image coordinates of the feature points. We have proved asymptotic convergence of the image errors to zero by the Lyapunov theory based on the nonlinear robot dynamics. Experiments have been carried out to verify the performance of the proposed controller.

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera, provided that the intrinsic and extrinsic parameters of the camera are not calibrated. A new adaptive algorithm is developed to estimate the unknown parameters on-line based on three original ideas. First, we use the pseudo-inverse of the depth-independent interaction matrix, proposed in our earlier work, to map the image errors onto the joint space so that we can linearly parameterize the closed-loop dynamics of the system. Second, to guarantee existence of the pseudo-inverse, we introduce a potential force to drive the estimated parameters away from the values resulting in singular image interaction matrix. Third, to ensure that the estimated parameters are convergent to their true values up to a scale, we combine the Slotine-Li method with an on-line algorithm for minimizing the errors between the estimated projections and real image coordinates of the feature points. We have proved asymptotic convergence of the image errors by Lyapunov method and demonstrated the performance by experiments

This paper presents a new adaptive controller for dynamic tracking of a robot manipulator without visual velocity when the intrinsic and extrinsic parameters of the camera are not calibrated. Most controllers in the past require the measurement of the visual velocity or differentiation of the visual position. The measurement of the visual velocity is subject to big noises in general due to low sampling rates of the vision loop. To avoid performance decaying caused by measurement errors of the visual velocity, the controller we developed requires estimated visual velocity only. With a full consideration of dynamic responses of the robot manipulator, we employed the Lyapunov method to prove the convergence of the image errors of the trajectory to zero and the convergence of the estimated parameters to the real values up to a scale. Experiments have been conducted to demonstrate good convergence of the trajectory errors of the robot under the control of the proposed method

This paper presents a new adaptive controller for a robot manipulator to control position of projections of unknown targets using the visual feedback from an eye-in-hand camera. The controller is designed to cope with the case when the intrinsic and extrinsic parameters of the camera are not calibrated. The controller employs the depth-independent image Jacobian to map the errors on the image plane onto the joint space. By using the depth-independent image Jacobian, it is possible to linearly parameterize the unknown camera parameters and the unknown coordinates of the target points in the closed loop dynamics of the system. A new algorithm is developed to estimate unknown parameters on-line. By minimizing the errors between the real and estimated projections of the target points on the image plane, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values. With a full consideration of dynamic responses of the root manipulator, we employed the Lyapunov method to prove asymptotic convergence of the image errors. Experiments have been conducted to demonstrate the performance of the proposed controller

This paper presents a new adaptive controller for image-based dynamic control of a robot manipulator using a fixed camera whose intrinsic and extrinsic parameters are not known. To map the visual signals onto the joints of the robot manipulator, this paper proposes a depth-independent interaction matrix, which differs from the traditional interaction matrix in that it does not depend on the depths of the feature points. Using the depth-independent interaction matrix makes the unknown camera parameters appear linearly in the closed-loop dynamics so that a new algorithm is developed to estimate their values on-line. This adaptive algorithm combines the Slotine-Li method with on-line minimization of the errors between the real and estimated projections of the feature points on the image plane. Based on the nonlinear robot dynamics, we prove asymptotic convergence of the image errors to zero by the Lyapunov theory. Experiments have been conducted to verify the performance of the proposed controller. The results demonstrated good convergence of the image errors.

This paper presents a new adaptive controller for dynamic image-based visual servoing of a robot manipulator when the intrinsic and extrinsic parameters of the camera are not calibrated. To cope with nonlinear dependence of the image Jacobian on the unknown parameters, this controller employs a matrix called nonscaled image Jacobian which does not depend on the scale factors determined by the depths of feature points. By removing the scale factors, the camera parameters appear linearly in the close-loop dynamics so that a new algorithm, different from Slotine and Li’s, is developed to estimate their values on-line. In the parameter adaptation, in addition to the regressor term, our algorithm also uses the errors between the real and estimated projections of the feature points on the image plane so as to guarantee the convergence of the estimated parameters to the real values up to a scale. A new Lyapunov function is introduced to prove asymptotic convergence of the image errors based on the robot dynamics. Experiments have been conducted to demonstrate the performance of the proposed controller.

This paper presents a new and novel controller for dynamic image-based trajectory tracking of a robot manipulator in uncalibrated environments. The controller is designed to cope with the case when the homogenous transformation matrix between the root and the vision system is unknown. A new adaptive algorithm, different from the Slotine and Li’s method, has been developed to estimate a set of parameters corresponding to the unknown transformation matrix. With a full consideration of dynamic responses of the robot manipulator, we employed the Lyapunov method to prove the convergence of the image errors of the trajectory to zero and the convergence of the estimated parameters to the real values up to a scale. Simulations and experiments have been conducted to demonstrate good convergence of the trajectory errors of the robot and the estimated parameters under the control of the proposed method.

This paper presents a new adaptive controller for dynamic image-based visual servoing of a robot manipulator when the intrinsic and extrinsic parameters of the camera are not calibrated. To cope with nonlinear dependence of the image Jacobian on the unknown parameters, this controller employs a matrix called nonscaled image Jacobian which does not depend on the scale factors determined by the depths of feature points. By removing the scale factors, the camera parameters appear linearly in the close-loop dynamics so that a new algorithm, different from Slotine and Li's, is developed to estimate their values on-line. In the parameter adaptation, in addition to the regressor term, our algorithm also uses the errors between the real and estimated projections of the feature points on the image plane so as to guarantee the convergence of the estimated parameters to the real values up to a scale. A new Lyapunov function is introduced to prove asymptotic convergence of the image errors based on the robot dynamics. Experiments have been conducted to demonstrate the performance of the proposed controller.