This article addresses the problem of distributed output formation optimal tracking for heterogeneous multiagent systems (MASs). Unlike the main approach in most existing formation tracking studies, which relies on prespecified trajectories, this work aims to enable heterogeneous MASs to achieve desired formation while tracking the optimal reference trajectory generated by a distributed optimization algorithm. First, a distributed time-varying optimization algorithm is proposed as the distributed optimal reference trajectory generator, which accounts for inequality constraints and ensures fixed-time convergence in consensus and asymptotic convergence in optimality. Then, a distributed output formation optimal tracking control protocol is developed for heterogeneous MASs, with the integration of the distributed optimal reference trajectory generator. Subsequently, the convergence of the proposed distributed output formation tracking control algorithm-based on time-varying optimization-is rigorously proven using Lyapunov stability analysis. Finally, simulation examples are provided to validate the theoretical results.

This article investigates an integrated approach of fault diagnosis and initial alignment of redundant strapdown inertial navigation systems (SINSs) under large misalignment angles. A redundant configuration of four hemispherical resonator gyroscopes (HRGs) and four accelerometers is designed. The parity vector method combined with generalized likelihood ratio test is developed for reliable detection and identification of HRG bias faults. For initial alignment, an analytic coarse alignment provides an initial attitude estimate, which is followed by a precise alignment phase using an unscented Kalman filter (UKF). The UKF is specifically designed to handle the nonlinear error model associated with large yaw misalignment angles. Experimental results demonstrate that the proposed fault diagnosis method effectively identifies HRG faults. Furthermore, comparative studies show that while the UKF and extended Kalman filter yield similar performance for small misalignment angles, the UKF achieves significantly superior alignment accuracy, especially under large yaw misalignment angle. This integrated approach enhances system reliability and navigation precision, which achieves a 100% detection rate for the tested bias faults and reduces the yaw error from$20^{\circ }$to below$0.35^{\circ }$.

The generalization ability of visual reinforcement learning, which allows the policy trained in the source domain to guide agents in similar unknown target environments, is one of the cores applied to visual navigation and autonomous driving. Recently, methods such as data augmentation techniques, self-supervised learning methods, and the generative adversarial network were employed to enhance the generalization capability of policy neural networks in visual reinforcement learning. However, current state-of-the-art methods, after utilizing domain-general latent features to train the RL policy, result in the loss of certain state-specific features, leading to diminished policy performance following generalization. To tackle these challenges, we designed a technical framework called self-expert imitation with purifying latent features, which enables the trained policy to effectively guide agents in scenarios similar to the training environment, without compromising the performance of the policy-guided agent in task completion. Additionally, a novel method was developed for separating domain-general and domain-specific latent vectors based on a variational autoencoder, enabling the domain-general component to exhibit strong and stable zero-shot generalization performance in unseen visually similar domains. Extensive experiments on the CarRacing game demonstrated that our approach achieves strong and stable generalization performance in unseen environments, without compromising the performance of the policy in guiding agents to complete tasks.

LiDAR-based Global Localization (LGL) is an essential ingredient for autonomous robots. However, existing LGL methods typically consider only partial information (e.g., geometric features) from LiDAR observations or are designed for homogeneous LiDAR sensors, overlooking the uniformity in LGL. In this work, a uniform LGL method is proposed, termed UniLGL, which simultaneously achieves spatial and material uniformity, as well as sensor-type uniformity. The key idea of the proposed method is to encode the complete point cloud, which contains both geometric and material information, into a pair of Bird's Eye View (BEV) images (i.e., a spatial BEV image and an intensity BEV image), thereby transforming the LGL problem into a cascaded LiDAR Place Recognition (LPR) and pose estimation problem from the perspective of image fusion. An end-to-end multi-BEV fusion network is designed to extract uniform features, equipping UniLGL with spatial and material uniformity. To ensure robust LGL across heterogeneous LiDAR sensors, a viewpoint invariance hypothesis is introduced, which replaces the conventional translation equivariance assumption commonly used in existing LPR networks and supervises UniLGL to achieve sensor type uniformity in both global descriptors and local feature representations. Moreover, UniLGL introduces a pipeline that leverages a pre-trained single-image Vision Foundation Model (VFM) for feature extraction to enhance the multi-BEV fusion LPR network, enabling strong generalization with only a few LiDAR data for fine-tuning. Finally, based on the mapping between local features on the 2D BEV image and the point cloud, a robust global pose estimator is derived that determines the global minimum of the global pose on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{SE}(3)$</tex-math></inline-formula> without requiring additional registration. To validate the effectiveness of the proposed uniform LGL, extensive benchmarks are conducted in real-world environments, and the results show that the proposed UniLGL is demonstratively competitive compared to other State-of-the-Art (SOTA) LGL methods. Furthermore, UniLGL has been deployed on diverse platforms, including full-size trucks and agile Micro Aerial Vehicles (MAVs), to enable high-precision localization and mapping as well as multi-MAV collaborative exploration in port and forest environments, demonstrating the applicability of UniLGL in industrial and field scenarios. The code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/shenhm516/UniLGL</uri>.

Service restoration implemented by multiple distributed energy resources (DERs) is a resilience-enhancing paradigm for modern distribution systems. To address the challenges of complex system modeling and the problem of cooperative control over heterogeneous multiple agents, this article proposes a graph reinforcement learning (G-RL) method based on heterogeneous multiagent systems (MASs). The method leverages graph-structured data to enhance the representation of distribution system states and employs graph attention networks (GATs) to deeply explore the power flow features and spatial characteristics of nodes in the restoration process. Additionally, a multihead self-attention (MHSA) is incorporated to strengthen collaboration among heterogeneous agents, enabling them to focus on relevant information from multiple perspectives during training. Finally, a joint simulation test platform is developed using Python and OpenDSS, and case studies on a 123-bus distribution system are conducted. Experimental results demonstrate that the proposed approach achieves efficient and autonomous service restoration by enhancing spatial feature extraction and improving collaborative decision-making among agents.

This article focuses on resilient time-varying optimal tracking problems of output formation in open multiagent systems (MASs). Agents can join or exit at any time and may be subject to switching between normal and malicious identities. Normal agents in the open MAS aim to minimize the sum of their local time-varying composite objective functions, each consisting of an output-related term and a state-related nonsmooth term. Simultaneously, agents are required to maintain a given output formation configuration. Based on relative outputs from neighbors, a distributed tracking protocol is proposed, combining the subgradient method with proximal mapping and an adaptive aggregation technique. By analyzing the upper bounds of total dynamic regret and individual dynamic regrets, it is proved that resilient optimal tracking of output formation can be achieved without knowledge of agent identities. Simulations validate these results.

Neural implicit scene representations have recently shown promising results in dense visual SLAM. However, existing implicit SLAM algorithms are constrained to single-agent scenarios, and fall difficulty in large indoor scenes and long sequences. Existing multi-agent SLAM frameworks cannot meet the constraints of communication bandwidth. To this end, we propose the first distributed multi-agent collaborative SLAM framework with distributed mapping and camera tracking, joint scene representation, intra-to-inter loop closure, and multi-submap fusion. Specifically, our proposed distributed neural mapping and tracking framework only needs peer-to-peer communication, which can greatly improve multi-agent cooperation and communication efficiency. A novel intra-to-inter loop closure method is designed to achieve local (single-agent) and global (multi-agent) consistency. Furthermore, to the best of our knowledge, there is no real-world dataset for NeRF-based/GS-based SLAM that provides both continuous-time trajectories groundtruth and high-accuracy 3D meshes groundtruth. To this end, we propose the first real-world indoor neural slam (INS) dataset covering both single-agent and multi-agent scenarios, ranging from small room to large-scale scenes, with high-accuracy ground truth for both 3D mesh and continuous-time camera trajectory. This dataset can advance the development of the community. Experiments on various datasets demonstrate the superiority of the proposed method in both mapping, tracking, and communication. The dataset and code will be open-source on https://github.com/dtc111111/MNESLAM.

Accurate Relative Pose Estimation (RPE) is critical for effective collaboration of multi-robot systems. Traditional methods using cameras or LiDARs heavily rely on overlapping Fields of View (FoV) between robots, which is highly demanding in practical applications and may hinder collaboration efficiency. To accommodate this issue, we propose Anchorless UWB-Assisted Relative Pose Estimation (AURPE), a novel approach that leverages ultra-wideband (UWB) technology in an anchorless setup to achieve multi-robot RPE without requiring overlapping FoVs or external infrastructure. AURPE first estimates the initial relative poses between robots using inter-robot UWB ranging combined with a Bayesian framework and constrained optimization. During robot operation, AURPE continuously refines the relative poses by integrating UWB measurements with LiDAR-inertial odometry (LIO) and employs a consensus voting mechanism to identify the most reliable pose estimates. Additionally, a pose graph-based backend optimization is incorporated to enhance the accuracy of both initial and real-time relative pose. Extensive simulations and real-world experiments demonstrate that AURPE achieves accurate RPE even in non-overlapping scenarios where traditional methods fail. Compared to state-of-the-art point cloud registration methods, AURPE shows superior performance in both accuracy and robustness, highlighting its potential to significantly enhance cooperative tasks in multi-robot systems operating in complex environments.

Accurate Six Degrees of Freedom (6-DoF) pose estimation of Ship-To-Shore (STS) quay crane spreaders is crucial for ensuring safe and efficient container handling in port automation. However, existing pose estimation techniques face significant challenges, as camera-based systems either rely on markers, which are prone to damage, or struggle with depth estimation inaccuracies. Additionally, 3D sensor-based approaches, particularly point cloud registration (PCR), face challenges such as initial pose errors, high-latency inference, and difficulties in object identification based purely on geometric features. To address these limitations, we propose LCSPose, a LiDAR-camera fusion-based 6-DoF pose estimation method that is marker-free, accurate, efficient, and scalable. Our approach integrates three key modules: (1) a semantic-geometric segmentation module for spreader segmentation and outlier removal, (2) a spatial consistency template sampling module based on Spatial Consistency Score (SC-Score) for reliable template selection across varying distances, and (3) a multi-view coarse-to-fine pose refinement module which incorporates multi-view PCA alignment for robust initial posture prior estimation and iterative pose refinement strategy for long-range registration. Our method demonstrates a 60% improvement in registration recall over state-of-the-art (SOTA) PCR methods, achieving up to 6 cm in translation error and 0.19 degrees in rotation error, while maintaining real-time processing at 20Hz.

Formulating a multi-robot obstacle avoidance policy is essential for enabling safe and efficient navigation in multi-robot environments, forming a critical component of the effective operation of multi-robot systems. Recently, reinforcement learning has been applied to improve the performance of decentralized, policy-driven robots in task execution. However, ensuring the safety of these agents during movement remains a significant challenge due to the inherent risks associated with the reinforcement learning process, such as frequent collisions. To address this issue and enhance the safety of policy-guided multi-robot navigation, we propose a novel policy based on imitation learning. This framework introduces a novel policy neural network that integrates a graph attention mechanism with the GRU network structure. The key innovation lies in utilizing the interactions between neighboring robots to enhance the safety of their movements. In a multi-robot simulation environment, robot behaviors are directed by the proposed policy. A comparative analysis was conducted between our approach and RL-RVO, one of the advanced methods in the field. The results demonstrate that our approach outperforms RL-RVO, achieving a higher success rate and significantly improving safety performance.

As delivery robots are increasingly integrated into our daily lives, their ability to navigate through crowded spaces demands swift and accurate prediction of pedestrian trajectories, which is crucial for autonomous functionality. However, existing methods face challenges of unstable accuracy and inefficiency in real-world deployment. Trajectory prediction involves both temporal and social dimensions. Recent methods have achieved better results by modeling temporal and social dimensions simultaneously, preventing information loss compared to modeling them separately, which significantly increases computational costs, posing challenges for practical deployment.In this paper, we conceptualize the trajectory prediction task as a deterministic sequence-to-sequence model that produces one precise forecast, aligning with real-world needs while reducing complexity. To improve efficiency and reduce latency for real-time applications, we propose a novel dynamic-stream transformer architecture that categorizes layers into multi-stream and single-stream based on the number of dimensions involved in computation. The single-stream modules attend to all dimensions simultaneously, providing comprehensive information fusion but with higher computational complexity. The multi-stream modules focus on only one dimension, enabling parallel and batched computation, crucial for improving the model’s real-time performance. By combining them strategically, we achieve a balance between accuracy and speed. Extensive experiments on real datasets show that our dynamic-stream transformer architecture significantly reduces computational complexity, achieving a speed increase of 180% to 3180% compared to similar approaches, while also attaining performance close to the state-of-the-art (SOTA) for deterministic trajectory prediction.

Recent work has shown that 3D Gaussian-based SLAM enables high-quality reconstruction, accurate pose estimation, and real-time rendering of scenes. However, these approaches are built on a tremendous number of redundant 3D Gaussian ellipsoids, leading to high memory and storage costs and slow training speed. To address this limitation, we propose a compact 3D Gaussian Splatting SLAM system that reduces the number and the parameter size of Gaussian ellipsoids. A sliding window-based masking strategy is first proposed to reduce the redundant ellipsoids. Then, a novel geometry codebook-based quantization method is proposed to further compress 3D Gaussian geometric attributes. Robust and accurate pose estimation is achieved by a local-to-global bundle adjustment method with reprojection loss. Extensive experiments demonstrate that our method achieves faster training, rendering speed, and low memory usage while maintaining the state-of-the-art (SOTA) quality of the scene representation.

Video streaming based teleoperation often faces a trade-off between bandwidth consumption and the need for high-fidelity telepresence. Higher image resolution or a wider field of view (FOV) substantially increases bandwidth requirements. In this paper, we propose a novel telepresence model for teleoperated vehicles operating in bandwidth-constrained environments. Our approach employs a LiDAR-fused 3D Gaussian Splatting (3DGS) as a compact scene representation to efficiently generate remote views. Initially, a static point cloud map is constructed using LiDAR-based semantic mapping, which serves as the initial Gaussians for optimizing the 3DGS model. During teleoperation, the prebuilt 3DGS is then rendered on the teleoperation platform, while only safety-critical information, such as vehicle pose and dynamic objects, is transmitted from the vehicle to the teleoperator in real-time. The proposed telepresence model significantly reduces data transmission requirements while maintaining photorealistic telepresence, enabling reliable and effective teleoperation even under stringent bandwidth constraints. This capability ensures safe and efficient vehicle teleoperation under challenging environments without relying on traditional high-bandwidth communication, thereby broadening the applicability of teleoperation technology to more demanding and diverse operational scenarios. Real-world experimental results show that the developed system can provide immersive teleoperation experiences at Kbps-level bandwidth consumption.

This article addresses a prescribed-time time-varying output formation tracking (TVOFT) problem for heterogeneous multiagent systems (MASs) under directed topologies. Formation tracking in heterogeneous linear MASs is critical for practical applications, such as cooperative robotics, autonomous transportation, and surveillance. However, many existing related designs often fail to guarantee convergence within a prescribed time. To overcome this limitation, a distributed prescribed-time TVOFT protocol associated with a corresponding design algorithm is presented. In the designed protocol, a distributed output-feedback observer is constructed for each follower to estimate the state of the leader within a prescribed time. Then, a local output-feedback controller is developed by incorporating a local state observer. It is proved that the heterogeneous MASs can achieve the desired TVOFT within the prescribed time. Furthermore, a heterogeneous experimental platform consisting of two autonomous aerial vehicles and three autonomous ground vehicles, is constructed to verify the effectiveness of the proposed prescribed-time TVOFT design. Comparative experiment results highlight the advantages of the proposed design over existing methods from the perspective of accurate prescribed-time convergence and practical feasibility.

Tracking the object 6-DoF pose is crucial for various downstream robot tasks and real-world applications. In this paper, we investigate the real-world robot task of aerial vision guidance for aerial robotics manipulation, utilizing category-level 6-DoF pose tracking. Aerial conditions inevitably introduce special challenges, such as rapid viewpoint changes in pitch and roll and inter-frame differences. To support these challenges in task, we first introduce a robust category-level 6-DoF pose tracker (Robust6DoF). This tracker leverages shape and temporal prior knowledge to explore optimal inter-frame keypoint pairs, generated under a priori structural adaptive supervision in a coarse-to-fine manner. Notably, our Robust6DoF employs a Spatial-Temporal Augmentation module to deal with the problems of the inter-frame differences and intra-class shape variations through both temporal dynamic filtering and shape-similarity filtering. We further present a Pose-Aware Discrete Servo strategy (PAD-Servo), serving as a decoupling approach to implement the final aerial vision guidance task. It contains two servo action policies to better accommodate the structural properties of aerial robotics manipulation. Exhaustive experiments on four well-known public benchmarks demonstrate the superiority of our Robust6DoF. Real-world tests directly verify that our Robust6DoF along with PAD-Servo can be readily used in real-world aerial robotic applications. The project homepage is released at Robust6DoF.

The core of multi-robot collision avoidance is to guide robots to avoid collisions with other robots and obstacles in a dynamic multi-robot environment, which has recently gained increasing interest among the main challenges of robotics. However, the current multi-robot navigation policy neural network exhibits weak position encoding capabilities for spatial environmental features in mapping environment states and robot actions, as well as an inability to recurrently infer information on dynamic environmental features in the temporal dimension, leading to insufficient safety and effectiveness in guiding robot motion. In this paper, we propose a novel spatial-temporal RetNet (STR) that encodes reciprocal collision avoidance states between robots in both spatial and temporal dimensions, aiming to enhance the safety and effectiveness of the policy neural network in guiding robots to accomplish specified tasks. The spatial state encoder module is developed based on parallel RetNet structure, which enhances the ability of the neural network in multi-robot navigation policies to extract reciprocal collision avoidance states between robots in spatial dimensions and overcomes the weak position encoding capability of advanced transformer-based multi-robot navigation policy neural networks. A temporal state encoder is designed by introducing the recurrent RetNet structure. This enhances the multi-robot navigation policy neural network’s ability to encode features in the temporal dimension of multi-robot movements and overcomes the transformer-based multi-robot navigation policy neural network’s inability to recurrently infer information in the time dimension. Simulation experiments were designed to demonstrate that the safety and effectiveness of our proposed method outperform the previous state-of-the-art approaches in guiding the robot to complete the task. Physical experiments illustrate that our policy can be effectively applied to real-world systemsNote to Practitioners—Multi-robot navigation has a wide range of real-world applications, such as multi-robot formation flying for search and rescue, autonomous warehouse operations, and robots navigating through human crowds. This paper introduces a novel Spatial-Temporal RetNet (STR) framework aimed at enhancing safety and effectiveness in multi-robot collision avoidance. STR addresses the limitations of existing methods by improving the neural network’s ability to extract reciprocal collision avoidance states in both spatial and temporal dimensions. The spatial state encoder strengthens the extraction of spatial features, while the temporal state encoder improves the handling of time-dependent information. Simulation and physical experiments demonstrate that STR enhances robot navigation in dynamic environments, making it suitable for real-world applications such as multi-robot coordination.

Dense scene reconstruction for photo-realistic view synthesis has various applications, such as VR/AR, and robotics navigation. Existing dense reconstruction methods are primarily designed for small room scenarios, but in practice, the scenes encountered by robots are typically large-scale environments. Most existing methods have difficulties in large-scale scenes due to three core challenges:(a) inaccurate depth input. Depth information is crucial for both scene geometry reconstruction and pose estimation. Accurate depth input is impossible to get in real-world large-scale scenes.(b) inaccurate pose estimation. Existing methods are not robust enough with the growth of cumulative errors in large scenes and long sequences.(c) insufficient scene representation capability. A single global radiance field lacks the capacity to scale effectively to large-scale scenes. To this end, we propose an incremental joint learning framework, which can achieve accurate depth, pose estimation, and large-scale dense scene reconstruction. For depth estimation, a vision transformer-based network is adopted as the backbone to enhance performance in scale information estimation. For pose estimation, a feature-metric bundle adjustment (FBA) method is designed for accurate and robust camera tracking in large-scale scenes and eliminates pose drift. In terms of implicit scene representation, we propose an incremental scene representation method to construct the entire large-scale scene as multiple local radiance fields to enhance the scalability of 3D scene representation. In local radiance fields, we propose a tri-plane based scene representation method to further improve the accuracy and efficiency of scene reconstruction. We conduct extensive experiments on various datasets, including our own collected data, to demonstrate the effectiveness and accuracy of our method in depth estimation, pose estimation, and large-scale scene reconstruction. The code has been open-sourced on https://github.com/dtc111111/incre-dpsr.

In recent years, there have been significant advancements in 3D reconstruction and dense RGB-D SLAM systems. One notable development is the application of Neural Radiance Fields (NeRF) in these systems, which utilizes implicit neural representation to encode 3D scenes. However, the depth images obtained from consumer-grade RGB-D sensors are often sparse and noisy, which poses significant challenges for 3D reconstruction and affects the accuracy of the representation of the scene geometry. Furthermore, existing methods select random pixels for camera tracking, leading to inaccurate localization in real-world indoor environments. To this end, we present NeSLAM, an advanced framework that achieves accurate and dense depth estimation, robust camera tracking, and realistic synthesis of novel views. First, a depth completion and denoising network is designed to provide dense geometry prior and guide the neural implicit representation optimization. Second, we propose a NeRF-based self-supervised feature tracking algorithm for robust real-time tracking. Experiments on various indoor datasets demonstrate the effectiveness and accuracy of the system in reconstruction, tracking quality, and novel view synthesis. Note to Practitioners—Traditional SLAM methods usually use the sparse point cloud to represent the scene, resulting in poor scene representation capability. Our method proposes a neural implicit representation method with depth completion and denoising network and feature tracking method, achieves accurate scene reconstruction and accurate pose estimation in various indoor scenes. The depth completion and denoising network provide accurate depth information associated with depth uncertainty, which is used to improve the geometry consistency. The NeRF-based self-supervised feature tracking method improve the accuracy and robustness for camera tracking. The experimental results demonstrate the accuracy and effectiveness of this method in different scenes.

Gaussian Splatting SLAMs have made significant advancements in improving the efficiency and fidelity of real-time reconstructions. However, these systems often encounter incomplete reconstructions in complex indoor environments, characterized by substantial holes due to unobserved geometry caused by obstacles or limited view angles. To address this challenge, we present Manhattan Gaussian SLAM, an RGB-D system that leverages the Manhattan World hypothesis to enhance geometric accuracy and completeness. By seamlessly integrating fused line segments derived from structured scenes, our method ensures robust tracking in textureless indoor areas. Moreover, The extracted lines and planar surface assumption allow strategic interpolation of new Gaussians in regions of missing geometry, enabling efficient scene completion. Extensive experiments conducted on both synthetic and real-world scenes demonstrate that these advancements enable our method to achieve state-of-the-art performance, marking a substantial improvement in the capabilities of Gaussian SLAM systems.

Formation constrained optimal tracking problems for heterogeneous multi-agent systems under Byzantine attacks are studied. The objective of the honest agents, unaffected by Byzantine attacks, is to find optimal trajectories that minimize the cumulative local cost functions of all honest agents, while simultaneously satisfying the cumulative local inequality constraints and maintaining the formation configuration. A distributed resilient formation tracking controller utilizing preview control and primal-dual strategy is proposed without requiring each agent to know which agents are affected by Byzantine attacks. The performance of the proposed algorithm is analyzed in terms of the upper bounds of dynamic regret and constraint violations. Numerical simulations and experiments, involving one unmanned aerial vehicle and four unmanned ground vehicles, are performed to verify the effectiveness of the obtained results.

This paper proposes a model predictive control method based on a dynamic event-triggered strategy for high-precision formation control of spacecraft formations using continuous low thrust. An optimal formation control problem is formulated based on the J2 linear time-varying relative dynamics using the Legendre pseudospectral method, and the model predictive controller is designed based on this optimization problem. Thereafter, a dynamic event-triggered strategy is introduced in the model predictive controller, in which the optimization problem is solved only when the event-triggered condition is violated. As a result, the computational pressure of the model predictive control algorithm is greatly reduced without increasing fuel consumption or degrading control accuracy. Meanwhile, the dynamic event-triggered condition allows for relatively even event-triggered intervals, which is more conducive to engineering applications. The feasibility and stability of this control strategy are demonstrated, and the simulation results under two different formation control scenarios show that the proposed controller achieves equivalent control performance to the conventional model predictive controller, and the computational pressure is greatly reduced by designing the dynamic event-triggered condition. Note to Practitioners—This paper presents a study on an event-triggered model predictive control (MPC) strategy for spacecraft formation control. The objective is to achieve high-precision control for spacecraft formations, including formation maintenance and reconfiguration, in the presence of bounded external disturbances. The proposed control strategy significantly reduces the computational burden of the model predictive control by introducing a dynamic event-triggered policy while preserving the control performance of the MPC. Moreover, the design of event-triggered conditions effectively prevents the occurrence of the Zeno phenomenon. The feasibility, system stability, and convergence of the control strategy are rigorously analyzed, and simulation-based verification is conducted in a binary-star formation control scenario. It is important to note that the event-triggered model predictive control strategy presented in this paper is applicable to various leader-follower type formation systems, extending beyond the scope of spacecraft formation control.

In robotic grasp detection, challenges such as uncertainty in object type, size, and placement within the scene diminish grasping accuracy. However, the inability to effectively locate the graspable area and incomplete feature extraction for grasp detection are two key factors that hinder grasp detection accuracy and are not considered in current methods. This paper presents a novel retentive attention-based multiscale perception grasp detection network (RAMPGrasp) to address this constraint. First, we introduce retentive attention in the feature extraction module, which significantly improves the efficiency of attention score computation for long sequences in visual tasks. Second, we propose a multiscale spatial pyramid attention module, which can effectively adjust the importance of multiscale feature sequences and feature channels, while enhancing the correlation of multiscale features. Third, we design the prediction module as a coarse-to-fine framework, improving feature representation for grasp detection by considering the distribution trend of grasp poses. As a result, RAMPGrasp achieves state-of-the-art grasp detection accuracy, with 98.4% and 95.6% on the Cornell and Jacquard datasets, respectively.

Existing Unsupervised Domain Adaptation (UDA) methods often fall short in fully leveraging contextual information from the target domain, leading to suboptimal decision boundary separation during source and target domain alignment. To address this, we introduce GrabDAE, an innovative UDA framework designed to tackle domain shift in visual classification tasks. GrabDAE incorporates two key innovations: the Grab-Mask module, which blurs background information in target domain images, enabling the model to focus on essential, domain-relevant features through contrastive learning; and the Denoising Auto-Encoder (DAE), which enhances feature alignment by reconstructing features and filtering noise, ensuring a more robust adaptation to the target domain. These components empower GrabDAE to effectively handle unlabeled target domain data, significantly improving both classification accuracy and robustness. Extensive experiments on benchmark datasets, including VisDA-2017, Office-Home, and Office31, demonstrate that GrabDAE consistently surpasses state-of-the-art UDA methods, setting new performance benchmarks. By tackling UDA's critical challenges with its novel feature masking and denoising approach, GrabDAE offers both significant theoretical and practical advancements in domain adaptation.

This study proposes a novel framework based on an entropy-informed graph neural network (EIGNN) integrated with Gaussian distribution (GD) to assess the driving risk of intelligent vehicles in typical traffic scenarios. Existing research often overlooks comprehensive spatiotemporal modeling of vehicle interaction characteristics and the quantification of uncertainty in dynamic risk assessments. In this work, vehicle speed and acceleration are probabilistically modeled using GD, while entropy theory is introduced to quantify risk uncertainty. A risk assessment model based on graph neural networks (GNNs) is then designed to capture the spatiotemporal dynamics of multivehicle interactions and predict the potential risk levels of driving strategies. The results demonstrate that the framework accurately quantifies collision risks in multivehicle interactions in complex traffic scenarios, with high accuracy and robustness across typical situations such as cruising, cut-ins, lane changes, overtaking, and different density traffic. By thoroughly analyzing traffic risk characteristics and incorporating them into intelligent driving decision-making, this study provides significant technical insights and theoretical support for enhancing the safety and decision-making efficiency of autonomous driving systems.

This study explores the effect of takeover time (TOT) on decision-making for Level-3 autonomous vehicles (L3-AVs). The existing research on L3-AV lacks an in-depth analysis of the mechanisms affecting TOT, ignores the importance of spatial and temporal variations in features for TOT prediction, and also lacks consideration of TOT in downstream trajectory planning tasks. This study proposed an exponential smoothing transformers (ETS) former model for TOT prediction, and then, the spatial-temporal predictive transformer (ST-Preformer) was employed to forecast the trajectories of surrounding vehicles, assess lane availability, and determine lane-changing probabilities. Ultimately, these evaluations contribute to the decision-making process of L3-AVs. The findings showed that the ETSformer was able to explain more than 83% of the characteristics of the TOT distribution in the TOT prediction task, effectively reducing the absolute percentage error by 0.7%, based on which the decision-making framework was able to make safe and comfortable optimal decisions. Decision-making is closely related to driving conditions and the surrounding traffic state, and TOT has a critical impact on the safety and stability of decision-making. A comprehensive understanding the impact of TOT on decision-making can help improve the safety of autonomous driving and provide guidance for improving decision-making techniques.

Neural implicit scene representations have recently shown encouraging results in dense visual SLAM. However, existing methods produce low-quality scene reconstruction and low-accuracy localization performance when scaling up to large indoor scenes and long sequences. These limitations are mainly due to their single, global radiance field with finite capacity, which does not adapt to large scenarios. Their end-to-end pose networks are also not robust enough with the growth of cumulative errors in large scenes. To this end, we introduce PLGSLAM, a neural visual SLAM system capable of high-fidelity surface reconstruction and robust camera tracking in real-time. To handle large-scale indoor scenes, PLGSLAM proposes a progressive scene representation method which dynamically allocates new local scene representation trained with frames within a local sliding window. This allows us to scale up to larger indoor scenes and improves robustness (even under pose drifts). In local scene representation, PLGSLAM utilizes tri-planes for local high-frequency features with multilayer perceptron (MLP) networks for the low-frequency feature, achieving smoothness and scene completion in unobserved areas. Moreover, we propose local-to-global bundle adjustment method with a global keyframe database to address the increased pose drifts on long sequences. Experimental results demonstrate that PLGSLAM achieves state-of-the-art scene reconstruction results and tracking performance across various datasets and scenarios (both in small and large-scale indoor environments).

Place recognition is crucial for unmanned vehicles in terms of localization and mapping. Recent years have witnessed numerous explorations in the field, where 2D cameras and 3D LiDARs are mostly employed. Despite their admirable performance, they may encounter challenges in adverse weather such as rain and fog. Hopefully, 4D millimeter-wave radar emerges as a promising alternative, as its longer wavelength makes it virtually immune to interference from tiny particles of fog and rain. Therefore, in this work, we propose a novel 4D radar place recognition model, TransLoc4D, based on sparse convolutions and Transformer structures. Specifically, a MinkLoc4D back-bone is first proposed to leverage the multimodal information from 4D radar scans. Rather than merely capturing geometric structures of point clouds, MinkLoc4D additionally explores their intensity and velocity properties. After feature extraction, a Transformer layer is introduced to enhance local features before aggregation, where linear self-attention captures the long-range dependencies of the point cloud, alleviating its sparsity and noise. To validate TransLoc4D, we construct two datasets and set up benchmarks for 4D radar place recognition. Experiments vali-date the feasibility of TransLoc4D and demonstrate it can robustly deal with dynamic and adverse environments.

Traditional object detection methods are limited by single-sensor constraints, high computational requirements, and poor real-time performance. In addition, occlusion often occurs under the condition of restricted single-view. In this paper, we introduce a camera and LiDAR fusion-based object detection method, which achieves excellent detection performance under limited computational resources. We also explores a fusion detection method deployed with multi-view, which can effectively solve the occlusion issue encountered by single view. The proposed method is valuable for single view as well as multi-view in various application scenarios. Our fusion method significantly improves detection accuracy and reliability, and solves the problems of data discrepancy, interference between sensors, and occlusion due to restricted view. Simulations and extensive experiments show that our proposed object detection method exhibited high accuracy and relatively low computational time.

Autonomous exploration has become a crucial technology for mobile robots, and numerous broadly applicable algorithms have emerged. However, few exploration methods effectively utilize the features of specified types of areas to enhance the efficiency of autonomous exploration in a complex environment. In this paper, we propose ACS-MM-Explore, an adaptive-circular-search-based exploration framework for large-scale urban road environments, focusing on extracting and utilizing the boundaries of roads to enhance exploration efficiency. Our approach integrates a multi-modal traversabil-ity analysis module to distinguish between road and non-traversable areas on a 2D costmap. A novel mechanism for gen-erating exploration viewpoints is introduced, efficiently creating exploration viewpoints with a circular search process with an adaptive radius. An optimized viewpoint selection mechanism is included, taking into account the geographical and geomet-ric information of each viewpoint. The framework extends the move base and TEB local planner as a viewpoint-based navigation module. A comprehensive evaluation is concluded in a simulation environment, demonstrating the framework's effectiveness and robustness.

3D object localization and intention prediction have become crucial components in autonomous system applications, such as self-driving car. However, there still faces a lot of challenges, especially for complex and dynamic scenarios where a single modality information is insufficient to effectively and precisely localize the position and analyze the intention of objects. An improved method based on multimodal information fusion has been proposed via leveraging the advantages of 2D image segmentation and 3D geometrical characteristics of LiDAR point cloud. Extensive comparative experiments have been conducted and the results demonstrate that the proposed method significantly enhances both localization and prediction accuracy, comparing with the method where 2D bounding box of object instead of segmentation information is used to be fused with point cloud.

For indoor environments, prior point-based visual SLAM cannot be processed in real time under low texture and illumination. To address this issue, this work proposes PLP-SLAM (Point-Line-Plane-SLAM) with RGB-D camera. Firstly, point and line features are detected in RGB images. For line features, establish length suppression and near line merge strategy to improve the line extraction quality. Secondly, plane features are extracted based on agglomerative hierarchical clustering method in point cloud obtained by RGB-D camera. Point clouds are divided into several nodes, unlike prior methods spend a lot of time to estimate the normal vector for each individual point, this work assumes that points within each node sharing the same plane normal vector, which can significantly improve the computational efficiency. Thirdly, sparse maps including points, lines and planes are established, meanwhile the scenes are reconstructed by creating the dense maps to show plan features directly. Finally, the performance of proposed method is compared against the state-of-the-art SLAM on public datasets to evaluate the pose estimation. All modules are run in real-time on a CPU, experiments clarify that PLP-SLAM can significantly enhance the robustness of 6DoF pose of the camera and simultaneously creating more detailed maps of the environment.

Multi-session mapping serves as the pre-requisite for autonomous robots to fulfill various long-term tasks (e.g., map updating, navigation, collaboration). However, it is challenging to implement multi-session mapping in enclosed or partially enclosed ambiguous environments (e.g., long corridors, industrial warehouses). Existing solutions either depend heavily on the matching of elementary geometric features (e.g., points, lines, and planes), which tends to fail in environments with ambiguous geometric features; or depend on the given guess of the initial transformation matrix of multiple single-session maps, which is not always obtainable and accurate enough. The ambient magnetic field has exhibited ubiquity and high distinctiveness at different location, which makes it suitable for estimating the initial transformation matrix. Thus, this paper proposes a novel probabilistic magnetic-aware Map Matching framework for Multi-session Mapping, namely MM4MM, to estimate the relative transformation of multiple single-session maps and to build the globally consistent maps in ambiguous and perceptually-degraded environments. The key novelties of this work are the designing of the hierarchical probabilistic map matching framework and the Particle Swarm Optimization strategy to associate the magnetic data of multiple sessions. Evaluations on both simulated and real world experiments demonstrate the greatly improved utility, accuracy, and robustness of multi-session mapping over the comparative methods.

As a new sensor, 4D radar (x, y, z, velocity) has great potential for V2X, due to its 3D point cloud, direct doppler velocity output, long distance ranging, low-cost, and more importantly, robust perception in all weathers. However, the extrinsic calibration of multiple long baseline 4D radars is rarely researched in V2X, which is the key to fuse multi-radars. The main reasons are three-folds: (1) New sensor. Thus, it is not surprising that little related work can be found. (2) Long baseline and large viewpoint-difference. Current works are mainly focused on unmanned vehicles, which is short baseline and small viewpoint-difference. (3) Sparse, noisy, and very cluttered 4D radar point cloud. Thus, it is challenging to rapidly and accurately locate the target and extract the feature. In this paper, LB-R2R-Calib (Long Baseline Radar to Radar extrinsic Calibration) is proposed to address these problems. The novelties are: (1) A new target is introduced: an eight-quadrant corner reflector enclosed by a foam sphere. The benefit is the target center is a viewpoint-invariant feature. Thus, it is ideal for large viewpoint-difference calibration. (2) A new feature extraction algorithm is proposed to rapidly locate the target and extract the target center from a very cluttered point cloud, as we observed some important characteristics of 4D radar. Experiments with two 4D radars in real environments with four configurations demonstrate our method is highly accurate and robust.

The localization of autonomous robots in modern enclosed or semi-enclosed environments, such as office/hotel/hospital, supermarket, and indoor car park environments where GPS signals are severely challenged, remains a bottleneck for the deployment of fully autonomous mobile systems. Existing infrastructure-based (e.g., QR codes, RFID) localization methods are troubled by high maintenance cost and inflexibility issues, while onboard sensors-based solutions (e.g., LiDAR/camera-based) suffer from the ambiguous geometric features and view obstructions from crowded dynamic obstacles (e.g., pedestrians). Magnetic field (MF)-based localization has been gradually utilized in recent years due to its independence from positioning infrastructures and geometric features, thus making it ideal for applications such as service robots and security robots. Magnetic map building serves as the basis and prerequisite component for MF-based localization tasks. The well-acknowledged Gaussian Process Regression (GPR) method can be implemented to build magnetic maps but with heavy computational burdens. Thus in this paper, we propose an efficient and accurate magnetic mapping system based on a novel Sliding-GPR (i.e., S-GPR) method, and evaluate different magnetic mapping methods. A unique region-of-interest (ROI) selection technique and a down/up-sampling method are proposed for the S-GPR to dramatically decrease the computational time while maintaining the mapping accuracy. Extensive experiments in a high-fidelity simulated warehouse and real-world car park environments show that our proposed S-GPR mapping method has exhibited the highest accuracy and relatively low computational time compared with the SOTA magnetic mapping methods.

This paper addresses the critical challenge of reliable object detection in autonomous vehicles operating in dynamic urban environments, particularly when facing adversarial attacks on perception systems. It presents a novel dualvalidation methodology leveraging the synergy of image based and point cloud based object detection systems. The approach comprises sensor calibration to align data from both sources, an attack detection algorithm utilizing cross-validation technique to identify inconsistencies, and a self-restoration strategy to ensure correct detection despite malicious manipulation. The methodology has been tested in a complex urban environment with adversarial scenarios including patch and random removal attacks. Experimental results demonstrate the robustness and accuracy of the proposed method in maintaining reliable object detection under adversarial conditions.

Generalizing visual reinforcement learning is fundamental to robot visual navigation, involving the acquisition of a policy from interactions with source environments to facilitate adaptation to analogous, yet unfamiliar target environments. Recent advancements capitalize on data augmentation techniques, self-supervised learning methods, and the generative adversarial network framework to train policy neural networks with enhanced generalizability. However, current methods, upon extracting domain-general latent features, further utilize these features to train the reinforcement learning policy, resulting in a decline in the performance of the learned policy guiding the agent to accomplish tasks. To tackle these challenges, a framework of self-expert imitation with purifying latent features was devised, empowering the policy to achieve robust and stable zero-shot generalization performance in visually similar domains previously unseen, without diminishing the performance of guiding the agent to accomplish tasks. The extraction method of domain-general latent features is proposed to enhance their quality based on the variational autoencoder. Extensive experiments have shown that our policy, compared with state-of-the-art counterparts, does not diminish the performance of the policy guiding the agent to accomplish tasks after generalization.

Navigating robots through dynamic multi-robot environments, avoiding collisions with both other robots and obstacles, has emerged as a central challenge in robotics. The existing approaches fall short in allowing the policy network to effectively capture spatial-temporal reciprocal collision avoidance in multi-robot environments, comprising both static and dynamic obstacles, resulting in inadequate safety and efficiency in directing robot movement. In this study, we introduce a novel policy neural network called Spatial-Temporal RetNet (STR), designed to encode reciprocal collision avoidance states between robots in spatial and temporal dimensions. The goal is to improve the safety and efficacy of the policy neural network in directing robots to complete assigned tasks. The spatial state encoder module is built upon a parallel RetNet structure, which strengthens the neural network's capacity in extracting reciprocal collision avoidance states between robots in spatial dimensions. This module addresses the limitations of position encoding in transformer-based multi-robot navigation policy neural networks. We design a temporal state encoder utilizing a recurrent RetNet structure. This innovation bolsters the multi-robot navigation policy neural network's capability to capture features in the temporal dimension of multi-robot movements. It addresses the limitations of transformer-based multi-robot navigation policy neural networks, particularly in recurrently inferring information across time dimensions. Simulation experiments were conducted to showcase the superior safety and effectiveness of our proposed method compared to previous state-of-the-art approaches in guiding robots to accomplish tasks.

In recent years, infrastructure-based localization methods have achieved significant progress thanks to their reliable and drift-free localization capability. However, the preinstalled infrastructures suffer from inflexibilities and high maintenance costs. This poses an interesting problem of how to develop a drift-free localization system without using the preinstalled infrastructures. In this paper, an infrastructure-free and drift-free localization system is proposed using the ambient magnetic field (MF) information, namely IDF-MFL. IDF-MFL is infrastructure-free thanks to the high distinctiveness of the ambient MF information produced by inherent ferromagnetic objects in the environment, such as steel and reinforced concrete structures of buildings, and underground pipelines. The MF-based localization problem is defined as a stochastic optimization problem with the consideration of the non-Gaussian heavy-tailed noise introduced by MF measurement outliers (caused by dynamic ferromagnetic objects), and an outlier-robust state estimation algorithm is derived to find the optimal distribution of robot state that makes the expectation of MF matching cost achieves its lower bound. The proposed method is evaluated in multiple scenarios1, including experiments on high-fidelity simulation, and real-world environments. The results demonstrate that the proposed method can achieve high-accuracy, reliable, and real-time localization without any pre-installed infrastructures.

Curb following is a key technology for autonomous road sweeping vehicles. Currently, existing implementations primarily involve pre-recording waypoints during human driving and subsequently retracing them autonomously. Moreover, existing research related to this topic predominately focuses on curb detection for driver assistance, yet the resultant curb detection outcomes remain underutilized in the development of autonomous curb following systems. To fill this gap, this paper proposes a real-time path generation and alignment control approach to facilitate autonomous curb following. Firstly, a segmented path generation algorithm is introduced that progressively generates reference path segments while ensuring the overall continuity of the reference path. Secondly, a parameterized alignment control algorithm is developed to accurately navigate the vehicle along the planned reference path with proved stability. Real public road experiments have been conducted to validate the proposed approach. The experimental results demonstrate the efficacy of the proposed methodologies across various curb following scenarios, including common concave, convex, and straight-concave curbs, thereby showcasing the practical viability of our methods in real-world applications.

Vision-assisted container loading of Rubber Tyred Gantry (RTG) cranes are facing two primary challenges. Firstly, the uncertainty inherent in Covolutional Neural Network (CNN) based detection hinders its direct application in the safety-critical operation of such heavy-duty machinery. Secondly, sensor calibration introduces additional complexities and errors into the system. However, existing studies have not adequately addressed these challenges. Motivated by this gap, this paper proposes an integrated approach for target detection and alignment control in container loading of RTG cranes. To ensure reliable target marker identification, a heuristic post-processing algorithm is developed as a complement to CNN-based foreground segmentation, thereby ensuring safety during the container handling process. On this basis, a pixel-based control scheme is designed to align the container with the target markers, which eliminates the need for offline or online sensor calibrations. The proposed approach has been successfully implemented on a real RTG crane manufactured by Shanghai Zhenhua Heavy Industries Co., Ltd. (ZPMC) and validated at the Port of Ningbo, China. Experimental results demonstrate the superiority of the proposed approach over current manual operations in port industries, highlighting its potential for crane automation.

Traditional teleoperation technologies based on video streaming are facing several challenges in practical applications, including limited bandwidth, constrained spatial awareness, and sensitivity to illumination. Existing studies have not adequately addressed these issues. This paper presents a novel non-video based teleoperation framework for autonomous vehicles operating in bandwidth-limited environments. To reduce the amount of data being transmitted, a persistent-transient environment model is proposed for telepresence. Initially, a digital twin of the environment is preconstructed, containing only persistent environmental information. Subsequently, transient information captured by onboard sensors, such as vehicle state and dynamic objects, necessitate real-time transmission. Based on this model, a 3D virtual scene is rendered in front of the teleoperator, offering any desired virtual viewpoint to enhance spatial awareness. This telepresence model only requires real-time transmission of minimal data, i.e., vehicle state and detected objects, and remains unaffected by illumination conditions, enabling teleoperation even in applications with Kbps-level bandwidth constraints. Experimental results showcase the substantial potential of the proposed framework in bandwidth-limited settings.

Designing a safe and effective collision avoidance policy for multiple robots is essential in decentralized scenarios, where each robot is responsible for generating its own paths, to ensure their safe operation. Recently, the utilization of reinforcement learning to develop decentralized policies that enable multiple robots to move cooperatively and accomplish tasks has yielded positive outcomes. However, the presence of exploration unsafe actions during the reinforcement learning training process results in inadequate safety. We seek to enhance the safety of distributed multi-robot navigation policies and propose a new imitation learning framework based on the variational Bayesian model, which enables robots to learn safe actions by anticipating the subsequent state they are expected to reach. In addition, a new policy neural network structure for multi-robot navigation is proposed by introducing the transformer structure, which encodes the significance of nearby robots in relation to their forthcoming conditions. Experiments demonstrated that our policy can more safely guide robots to navigate in multi-robot environments under conditions of limited information, outperforming the state-of-the-art RL-RVO method in terms of success rate.Note to Practitioners—The motivation of this paper is to address the problem of collision avoidance in a multi-robot environment under limited information, which can also be applied to autonomous driving, crowd simulation, and other related fields. Positive outcomes have been observed in the utilization of reinforcement learning to create decentralized policies that enable multiple robots to move cooperatively and complete tasks. However, inadequate safety remains a challenging task due to the possibility of exploring hazardous actions during training. This article aims to enhance the safety of distributed policies guiding robots to accomplish navigation tasks in dynamic multi-robot environments. To begin with, we introduce a novel framework for imitation learning that is based on the variational Bayesian model. This framework facilitates the learning of safe actions by the policy to improve its performance and guide the robot in navigating and avoiding obstacles more securely. A loss function is proposed that enables the anticipation of the future state expected to be reached by the robot. By incorporating the transformer structure, a new neural network structure is designed for multi-robot navigation that encodes the significance of nearby robots concerning their upcoming conditions. This network structure employs a BiGRUs to facilitate the assimilation of observations from multiple agents by the policy. Compared to existing works such as GA3C-CADRL, SARL, and RL-RVO, our proposed method achieves a higher success rate. In our future research, we will investigate methods to enhance the policy’s performance in guiding robots to complete tasks by focusing on improving travel time and average speed, while also strictly ensuring safe navigation. Furthermore, we plan to extend this approach by addressing navigation challenges in more densely populated multi-robot environments.

Heavy rain significantly reduces image visibility, hindering tasks like autonomous driving and video surveillance. Many existing rain removal methods, while effective in light rain, falter under heavy rain due to their reliance on purely spatial features. Recognizing this challenge, we introduce the Wavelet-Spatial Dual Attention Transformer Framework (WSDformer). This innovative architecture adeptly captures both frequency and spatial characteristics, anchored by the wavelet-spatial dual attention (WSDA) mechanism. While the spatial attention zeroes in on intricate local details, the wavelet attention leverages wavelet decomposition to encompass diverse frequency information, augmenting the spatial representations. Furthermore, addressing the persistent issue of incomplete structural detail restoration, we integrate the PriorFormer Block (PFB). This unique module, underpinned by the Prior Fusion Attention (PFA), synergizes residual channel prior features with input features, thereby enhancing background structures and guiding precise rain feature extraction. To navigate the intrinsic constraints of U-shaped transformers, such as semantic discontinuities and subdued multi-scale interactions from skip connections, our Cross Interaction U-Shaped Transformer Network is introduced. This design empowers superior semantic layers to streamline the extraction of their lower-tier counterparts, optimizing network learning. Empirical analysis reveals our method's leading prowess across rainy image datasets and achieves state-of-the-art performance, with notable supremacy in heavy rainfall conditions. This superiority extends to diverse visual challenges and real-world rainy scenarios, affirming its broad applicability and robustness. The source code is available athttps://github.com/Jiongze-Yu/WSDformer.

Robust visual place recognition (VPR) against significant appearance changes is crucial for the life-long operation of mobile robots. Focusing on this task, we propose a Co-Attentive Hierarchical Image Representations (CAHIR) framework for VPR, which unifies attention-sharing global and local descriptor generation into one encoding pipeline. The hierarchical descriptors are applied to a coarse-to-fine VPR system with global retrieval and local geometric verification. To explore high-quality local matches between task-relevant visual elements, a cross-attention mutual enhancement layer is introduced to strengthen the information interaction between the local descriptors. Through the proposed selective matching distillation, the mutual enhancement layer can learn from state-of-the-art local matchers in a distillation manner. After weighted cross-matching of the enhanced local descriptors, geometric verification is applied to evaluate the spatial consistency of the compared image pair. Experiments show CAHIR outperforms the existing global and local representations for VPR in terms of performance and efficiency. Quantitatively, it achieves state-of-the-art results on three city-scale benchmark datasets. Qualitatively, CAHIR proves to attach great importance to task-relevant visual elements and excels at finding local correspondences that are discriminative to the VPR task.

Accurate global localization is an essential ingredient for autonomous mobile robots (AMRs) operating in enclosed or partially enclosed repetitive environments (e.g., office corridors, industrial warehouses, transportation centers). In such environments, the Global Navigation Satellite System (GNSS) signals are unreliable or severely degraded. The highly ambiguous structures in such challenging scenarios would also lead the ordinary geometric feature-based LiDAR/visual localization methods to fail. The ambient magnetic field (MF) has exhibited high distinctiveness at different location, which makes it a viable alternative for infrastructure-free AMR localization. However, few of the previous research has been focused on the orientation-dependency and similar-sequential-route limitations of MF-based localization. Thus, this paper proposes a novel probabilistic global localization system with 2-D LiDAR and rotation-invariant magnetic field for AMRs operating in challenging repetitive and ambiguous environments. The proposed localization system mainly consists of: 1) Two-step Initialization: laser distance and MF sequence based matching, and 2) MF-based Pose Tracking: recursive multi-dimensional MF sequence based matching. Extensive experimental results demonstrate the advantageous localization performances of the proposed localization system over the existing methods.

LiDAR-based SLAM may easily fail in adverse weathers (e.g., rain, snow, smoke, fog), while mmWave Radar remains unaffected. However, current researches are primarily focused on 2D$(x,y)$or 3D ($x, y$, doppler) Radar and 3D LiDAR, while limited work can be found for 4D Radar ($x, y, z$, doppler). As a new entrant to the market with unique characteristics, 4D Radar outputs 3D point cloud with added elevation information, rather than 2D point cloud; compared with 3D LiDAR, 4D Radar has noisier and sparser point cloud, making it more challenging to extract geometric features (edge and plane). In this paper, we propose a full system for 4D Radar SLAM consisting of three modules: 1) Front-end module performs scan-to-scan matching to calculate the odometry based on GICP, considering the probability distribution of each point; 2) Loop detection utilizes multiple rule-based loop pre-filtering steps, followed by an intensity scan context step to identify loop candidates, and odometry check to reject false loop; 3) Back-end builds a pose graph using front-end odometry, loop closure, and optional GPS data. Optimal pose is achieved through$\mathrm{g}2\mathrm{o}$. We conducted real experiments on two platforms and five datasets (ranging from 240m to 4.8km) and will make the code open-source to promote further research at: https://github.com/zhuge2333/4DRadarSLAM

Visual Place Recognition (VPR) is essential for autonomous robots and unmanned vehicles, as an accurate identification of visited places can trigger a loop closure to optimize the built map. The most prevalent methods tackle VPR as a single-frame retrieval task, which uses a CNN-based encoder to describe and compare each individual frame. These methods, however, overlook the temporal information between frames. Other methods improve this by searching the database with consecutive frames, which can greatly reduce false positives. Nevertheless, current sequence-based methods typically assume the consecutive image frames to be captured at an approximately constant speed, which is not always the case in practice. Therefore, we propose an adaptive sequence search strategy (AdaptSeq), which can dynamically alter the step size of adjacent frames in the retrieved sequence trajectory. Furthermore, to address false positive retrieval of input frames, we propose a CNN-based discriminator named DDsNet. It can determine whether the top retrieved candidates are true positives based on the learned statistics rather than an artificial threshold. Overall, we construct a novel sequence-based VPR pipeline named AdaptSeqVPR. It utilizes a CNN-based encoder for frame descriptions, and encompasses AdaptSeq and DDsNet for sequence matching. The experimental results indicate that our AdaptSeqVPR exhibits superior performance compared to the baseline SeqSLAM and SeqVLAD. Notably, our method can robustly handle the sequence-based VPR for vehicles traveling at non-uniform speeds in changing environments.

In V2X (Vehicle-to-Everything), one important work is to extrinsically calibrate multiple 3D LiDARs, which are mounted with a long baseline and large viewpoint-difference at the road-side. Current solutions either require a specific target being set up (e.g., a sphere), or require specific features existing in the environment (e.g., mutually orthogonal planes). However, it is time-consuming, sometimes even inconvenient, to set up specific targets, e.g., at busy intersections and highways. Furthermore, specific features do not always exist in the traffic scenario. Thus, the current solutions are not feasible. To address this problem, a novel extrinsic calibration method is proposed in this paper, namely LB-L2L-Calib 2.0. It is the 2.0 version of our previous work. The novelties are: 1) We propose to use the easily accessible objects on the road as features for calibration (i.e., the vehicles). Thus, it is not necessary to set up any specific targets and we do not need to worry whether specific features exist or not. The key point is we observed that the 3D bounding box centers of the vehicles are viewpoint-invariant from different viewpoints, which makes them ideal features for long baseline and large viewpoint-difference calibration. 2) To establish correct correspondence between the bounding box centers detected from different LiDARs, we propose an exhaustive searching strategy. It can robustly output correct correspondence. Extensive experiments are performed in three scenarios (simulation: intersection, real: carpark and highway), with two types of LiDAR (Velodyne and Livox), demonstrating that LB-L2L-Calib 2.0 is robust, effective, and accurate.

Extrinsic calibration between LiDAR-Camera and LiDAR-LiDAR has been researched extensively, because it is the foundation for sensor fusion. Meanwhile, many projects are open-sourced and significantly promote related research. However, limited solutions can unify the calibration between repetitive scanning and non-repetitive scanning 3D LiDAR, sparse and dense 3D LiDAR, visual and thermal camera. Currently, to achieve that, we normally need to use different targets and extract different features for different sensor combinations. Sometimes, human intervention is required to locate the target. It is inconvenient and time-consuming. In this paper, L2V2T2Calib is introduced and open-sourced as a trial to unify the calibration. 1). A four-circular-holes board is adopted for all sensors. The four circle centers can be detected by all the sensors, thus are ideal common features. Previous works also use this target, but the algorithms don’t consider non-repetitive scanning LiDARs, thus cannot be directly applied. 2). To unify the process, an important step is to automatically and robustly detect the target from different types of LiDARs. However, this does not receive enough attention. We propose a method based on template matching. It is simple, but effective and general to different depth sensors. 3). We provide two types of output, minimizing 2D re-projection error (Min2D) and minimizing 3D matching error (Min3D), for different users. And their performance is compared. Extensive experiments conducted in both simulation and real environment demonstrate L2V2T2Calib is accurate, robust, more importantly, unified. The code will be open-sourced to promote related research at: https://github.com/Clothooo/lvt2calib

As the drone captures image targets at different flying altitudes, their scales may vary significantly, which can pose challenges for the object detection model to accurately detect them. Additionally, tiny objects in the image contain minimal information, making them difficult to distinguish from the background. To overcome these two challenges, we proposed a network architecture that aims to improve the accuracy of tiny object detection in drone images. Specially, we designed a tiny object detector(TOD) that can effectively extract features of tiny objects and distinguish between tiny object features and image background. Furthermore, this TOD module contains a Convolutional Visual Attention Network (CVAN) to better focus on the regions of tiny objects. Experimental results demonstrate that the proposed method achieves [email protected] accuracy of 53.9% on the VisDrone2021-test-dev dataset and improves by 2.8 % compared to YOLOv7.

This paper investigates the cruise control problem of unmanned ground vehicle (UGV) platoons from the implementation perspective. Unlike most existing works related to platoon cruise control which rely on positioning infrastructures such as lane markings, roadside units, and global navigation satellite systems (GNSS), this paper explores a new problem: platoon cruise control in environments without positioning infrastructures. The introduction of this constraint disables most existing cruise control approaches. To address this problem, an integrated localization and planning framework is proposed, which is composed of three modular algorithms. Firstly, to localize multiple vehicles in a common coordinate system, a collaborative localization algorithm is developed through matching local perceptions of different vehicles. Secondly, to maintain the desired platoon configuration, the historical trajectory of the preceding vehicle is reconstructed, based on which the target state is planned for the following vehicle. Finally, a virtual controller based algorithm is designed to generate feasible trajectories for the following vehicle in real time. The proposed framework has two salient features. Firstly, it does not depend on positioning infrastructures and does not introduce additional positioning sensors, such as GNSS/INS modules, ultra-wideband (UWB) devices, magnetic meters and so on, as long as each vehicle is equipped with a perception sensor (Lidar, radar or camera), which however is essential equipment for nowaday autonomous systems. Secondly, the proposed framework does not depend on direct observations between vehicles to achieve relative localization, making it applicable in non-line-of-sight (non-LOS) situations. Real-world experiments have been conducted to validate the effectiveness, robustness and practicality of the proposed framework.

Collaborative heterogeneous robots are expected to perform comprehensive perception, mapping and coordination in search and rescue scenarios. The challenge of collaboration between heterogeneous robots lies in their huge differences in perception, mobility and processing capabilities. In this paper, a novel collaborative UAV-UGV mapping framework is proposed in GNSS-denied and unknown environments. The key novelty of this work is the proposing of a unified framework to formulate the UAV-UGV collaborative mapping problem with a continuous-discrete model, as well as its realization in real robotic systems. In order to project continuous space into discrete space, a novel information gain trigger scheme is pro-posed. The continuous space allows each robot to perform high frequency local map estimation, while discrete space describes the problem of multi-resolution hybrid map fusion. Considering the nature of data heterogeneity, a flexible probabilistic fusion algorithm is proposed that addresses the multi-resolution hybrid map fusion problem, where the local maps generated by UAV and UGV are fused based on Bayesian rule. The proposed UAV-UGV hybrid system is validated in various challenging scenarios, demonstrating its accuracy and utility in practical tasks.

Multi-LiDAR system is an important part of V2X (Vehicle to Everything) to enhance the perception information for unmanned vehicles. To fuse the information from multiple 3D LiDARs, accurate extrinsic calibration between the LiDARs is essential. However, the existing multi-LiDAR calibration methods mainly focus on short baseline scenarios, where multiple LiDARs are closely mounted on a single platform (e.g., an unmanned vehicle). Besides, most methods typically use a planar target for calibration. Some of the methods require the motion of the multi-LiDAR system. The above conditions severely limit the application of these methods to V2X, where LiDARs are non-movable, the baseline and viewpoint difference between the LiDARs can be very large. In order to meet these challenges, we propose an accurate and robust extrinsic calibration method for long baseline multi-LiDAR systems, named LB-L2L-Calib (Large Baseline LiDAR to LiDAR extrinsic Calibration). (1) We use a sphere as the calibration target for multiple LiDARs with large viewpoint difference, leveraging the viewpoint-invariance of the sphere. (2) A improved sphere detection and sphere center estimation strategy is introduced to detect and extract the sphere center from a cluttered point cloud in large-scale outdoor scenario. (3) A extrinsic parameter regression scheme is introduced. Both simulation and real experiments demonstrate that LB-L2L-Calib is highly accurate and robust. Quantitative results show that the rotation and translation error is less than 0.01m and 0.01° (in simulation, Gauss noise 0.03m, the distance and viewpoint difference between two LiDARs is more than 30m and 90°).

Autonomous robots are often required to acquire high-level prior knowledge by continuously reconstructing the semantics and geometry of the surrounding scene, which is the basis of exploration and planning. Most existing continuous semantic mapping algorithms cannot distinguish potential differences in voxels, resulting in an over-inflated map. Furthermore, fixed-size query ranges introduce high computational complexity. Based on the limitation of over-inflation and inefficiency, this paper proposes a novel incremental continuous semantic occupancy mapping algorithm (S-MKI). The key innovation of this work comes from the two models in the preprocessing stage. On the one hand, Redundant Voxel Filter Model utilizes context entropy to filter out redundant voxels to improve the confidence of the final map, where objects have accurate boundaries with sharp edges. On the other hand, Adaptive Kernel Length Model adaptively adjusts the kernel length with class entropy, which reduces the inherent amount of training data. The final multientropy kernel inference function is formulated to integrate these two models to infer sparse noisy sensor data into dense accurate 3D maps. Experimental results conducted in both indoors and outdoors datasets validate that S-MKI outperforms existing methods.

Place recognition is seen as a crucial factor to correct cumulative errors in Simultaneous Localization and Mapping (SLAM) applications. Most existing studies focus on visual place recognition, which is inherently sensitive to environmental changes such as illumination, weather and seasons. Considering these facts, more recent attention has been attracted to use 3-D Light Detection and Ranging (LiDAR) scans for place recognition, which demonstrates more credibility by exerting accurate geometric information. Different from pure geometric-based studies, this paper proposes a novel global descriptor, named SectionKey, which leverages both semantic and geometric information to tackle the problem of place recognition in large-scale urban environments. The proposed descriptor is robust and invariant to viewpoint changes. Specifically, the encoded three-layers key serves as a pre-selection step and a ‘candidate center’ selection strategy is deployed before calculating the similarity score, thus improving the accuracy and efficiency significantly. Then, a two-step semantic iterative closest point (ICP) algorithm is applied to acquire the 3-D pose (x, y, θ) that is used to align the candidate point clouds with the query frame and calculate the similarity score. Extensive experiments have been conducted on public Semantic KITTI dataset to demonstrate the superior performance of our proposed system over state-of-the-art baselines.

Visual Place Recognition (VPR) has become an indispensable capacity for mobile robots to operate in large-scale environments. Existing methods in this field mostly focus on exploring high-performance encoding strategies, while few attempts are devoted to lightweight models that balance per-formance and computational cost. In this work, we propose a Lightweight Self-attentional Distillation Network (LSDNet) aiming to obtain advantages of both performance and efficiency. (1) From a performance perspective, an attentional encoding strategy is proposed to integrate crucial information in the scene. It extends the NetVlad architecture with a self-attention module to facilitate non-local information interaction between local features. Through further visual word vector rescaling, the final image representation can benefit from both non-local spatial integration and cluster-wise weighting. (2) From an efficiency perspective, LSDNet is built upon a lightweight back-bone. To maintain comparable performance to large backbone models, a dual distillation strategy is introduced. It prompts LSDNet to learn both encoding patterns in the hidden space and feature distributions in the encoding space from the teacher model. Through distillation-augmented training, LSDNet is able to rival the teacher model and outperform SOTA global representations with the same lightweight backbone.

Robots that are supposed to interact with or manipulate objects in the world must be able to track the poses of objects in their sensor data. Thus, Detecting and tracking the 6-DoF poses of targeted objects is important for aerial manipulation and is still in the early stage due to the high dynamics and limited onboard capacity of such systems. In this paper, we propose ICK-Track, a novel method for onboard category-level object 6-DoF pose tracking that can be applied to aerial manipulation without using any pre-defined object CAD models. It first utilizes a semi-supervised video segmentation to detect objects in the eye-in-hand RGB-D camera stream to segment the 3D points of objects. Then, canonical keypoints are extracted using iterative farthest point sampling. We propose a novel inter-frame consistent keypoints generation network to generate the corresponding keypoint pairs, which are used together with ICP to estimate the pose changes of objects for tracking. Experimental results show that our method is more robust to viewpoint changes and runs faster than the state-of-the-art methods on category-level pose tracking. We further test our proposed method on a real aerial manipulator. A demo video showing the use of our method on a real aerial manipulator and the implementation of our method are available at: https://github.com/S-JingTao/ICK-Track.

Last-mile delivery robots are usually required to navigate on the sidewalk through a fixed route. The current solutions heavily rely on the image-based perception and GPS localization to successfully complete delivery tasks. However, it is prone to fail and become unreliable when the robot runs in challenging conditions, such as operating in different illuminations, or under canopies of trees or buildings. To address these issues, this paper proposes a novel robust sidewalk navigation method for the last-mile delivery robots with an affordable sparse LiDAR, which consists of two main modules: Semantic Point Cloud Network (SegPCn) and Reactive Nav-igation Network (RNn), as shown in Fig. 1. More specifically, SegPCn takes the raw 3D point cloud as input and predicts the point-wise segmentation labels, presenting a robust perception capability even in the night. Then, the semantic point clouds are fed to RNn to generate an angular velocity to navigate the robot along the sidewalk, where the localization of the robot is not required. Moreover, an autolabeling mechanism is developed to reduce the labor involved in data preparation as well. And the LSTM neural network is explored to effectively leverage the historical context and derive correct decisions. Extensive experiments have been carried out to verify the efficacy of this method, and the results show that this method enables the robot to navigate on the sidewalk robustly during day and night. We open source the code and the data set on https://github.com/lukewenMX/Robust-Navigation-Method.

Recently, intelligent unmanned aerial vehicles (UAVs) have shown great advantages of flexibility and productivity in power line inspection, wherein robust detection of power lines from aerial images for automatic power line following navigation is required. However, identifying power lines accurately from a cluttered background is challenging due to the limited resolution of onboard cameras and the noisy environment. In this letter, we propose a novel power line detection method, denoted by CODNet, for the application of UAV navigation. Unlike existing works, the proposed method can extract features of power lines from cluttered backgrounds automatically and predict centers and orientations of power lines in the scene simultaneously. Besides, we introduce a new clustering method to summarize the average location and orientation of detected power lines as a guide for the automatic navigation of UAVs. Finally, experimental results demonstrate both the effectiveness and the superiority of the CODNet.

This paper investigates the cyber-security problem for autonomous vehicles under sensor attacks. In particular, a model-based framework is proposed which can detect sensor attacks and identify their sources in order to achieve the secure localization of self-driving vehicles. To ensure robustness of the vehicle against cyber-attacks, sensor redundancy is introduced, that is to deploy multiple sensors, each of which provides real-time pose observations of the vehicle. A bank of attack detectors is developed to capture anomalies in each sensor measurement, which is a combination of an extended Kalman filter (EKF) and a cumulative sum (CUSUM) discriminator. EKFs are employed to estimate the vehicle position and orientation recursively, while each CUSUM discriminator is designed to analyze the residual generated by its combined EKF to detect the possible deviation of the sensor measurement from the expected pose derived according to the mathematical model of the vehicle. To monitor the inconsistency amongst multiple sensor measurements, an auxiliary detector is introduced which fuses observations from multiple sensors. Based on the results of all the detectors, a rule-based isolation scheme is developed to identify the source anomalous sensor. The effectiveness of our proposed framework has been demonstrated on real vehicle data.

Stereo matching is a practical method to estimate depth information and retrieve 3D world in robot perception and autonomous driving scenarios. With the development of convolution neural networks (CNNs), deep-learning based stereo matching algorithms have significantly improved the accuracy and dominated most of the online benchmarks. However, limited labels in real world, especially in challenging weather conditions, still hinder the technology from practical usage. In this paper, we propose a new unsupervised learning mechanism for stereo matching, utilizing adversarial iterative learning and novel soft warping loss to promote the effectiveness of the networks in unseen environments. The experiments transferring the stereo matching module from synthetic domain to real-world domain demonstrate the superiority of our proposed method. Extensive experiments in challenging weathers further prove that our method shows great practical potential in strait environments.

The core of visual place recognition (VPR) lies in how to identify task-relevant visual cues and embed them into dis- criminative representations. Focusing on these two points, we propose a novel encoding strategy named Attentional Pyramid Pooling of Salient Visual Residuals (APPSVR). It incorporates three types of attention modules to model the saliency of local features in individual, spatial and cluster dimensions respectively. (1) To inhibit task-irrelevant local features, a semantic-reinforced local weighting scheme is employed for local feature refinement; (2) To leverage the spatial context, an attentional pyramid structure is constructed to adaptively encode regional features according to their relative spatial saliency; (3) To distinguish the different importance of visual clusters to the task, a parametric normalization is proposed to adjust their contribution to image descriptor generation. Experiments demonstrate APPSVR outperforms the existing techniques and achieves a new state-of-the-art performance on VPR benchmark datasets. The visualization shows the saliency map learned in a weakly supervised manner is largely consistent with human cognition.

Large-scale visual place recognition (VPR) is inherently challenging because not all visual cues in the image are beneficial to the task. In order to highlight the task-relevant visual cues in the feature embedding, the existing attention mechanisms are either based on artificial rules or trained in a thorough data-driven manner. To fill the gap between the two types, we propose a novel Semantic Reinforced Attention Learning Network (SRALNet), in which the inferred attention can benefit from both semantic priors and data-driven fine-tuning. The contribution lies in two-folds. (1) To suppress misleading local features, an interpretable local weighting scheme is proposed based on hierarchical feature distribution. (2) By exploiting the interpretability of the local weighting scheme, a semantic constrained initialization is proposed so that the local attention can be reinforced by semantic priors. Experiments demonstrate that our method outperforms state-of-the-art techniques on city-scale VPR benchmark datasets.

Symmetric environment is one of the most intractable and challenging scenarios for mobile robots to accomplish global localization tasks, due to the highly similar geometrical structures and insufficient distinctive features. Existing localization solutions in such scenarios either depend on pre-deployed infrastructures which are expensive, inflexible, and hard to maintain; or rely on single sensor-based methods whose initialization module is incapable to provide enough unique information. Thus, this paper proposes a novel Multi-Sensor based Two-Step Localization framework named MSTSL, which addresses the problem of mobile robot global localization in geometrically symmetric environments by utilizing the measured magnetic field, 2-D LiDAR, and wheel odometry information. The proposed system mainly consists of two steps: 1) Magnetic Field-based Initialization, and 2) LiDAR-based Localization. Based on the pre-built magnetic field database, multiple initial hypotheses poses can firstly be determined by the proposed two-stage initialization algorithm. Then, utilizing the obtained multiple initial hypotheses, the robot can be localized more accurately by LiDAR-based localization. Extensive experiments demonstrate the practical utility and accuracy of the proposed system over the alternative approaches in real-world scenarios.

In unstructured and unknown environments, heterogeneous robots must be able to perceive the environment, coordinate with each other and complete tasks collaboratively with onboard sensors. In this paper, a tightly-coupled perception and navigation framework is proposed for heterogeneous land-air robots, which forms a closed loop of perception-navigation for heterogeneous robots. The key novelty of this work is the proposing of a unified framework to formulate the cooperative mapping and navigation problem, as well as the derivation of high-level coordination strategy and low-level goal-oriented navigation within a fully integrated approach. To provide a comprehensive understanding of the environment, a flexible probabilistic map fusion algorithm is applied to merge local maps generated by hybrid robots. The proposed UAV-UGV hybrid system is validated in challenging experiments, proving its robustness and effectiveness in practical tasks.

Adversarial examples can be imperceptible to human eyes but can easily fool deep models. Such intrigue property has raised security issues for real-world industrial deep learning systems. To combat those malicious attacks, a novel defense strategy has been proposed based on the conditional variational autoencoder (CVAE) and Bayesian network (BN). The main contribution lies in the provided systematic dual-domain-based defense framework, which covers three modules named detection, diagnosis, and recovery. Specifically, the CVAE is first introduced for latent- and residual-domain generation. Subsequently, a composite and hierarchical BN detector is proposed to conduct the adversary detection through feature validation and output justification. Afterwards, a diagnosis strategy has been constructed for residual domain and different attacks can be evaluated in the unified framework. Finally, a two-step recovery mechanism is established on the CVAE that can effectively restore the feature representations and the network predictions from various adversaries. The feasibility of the entire defense diagram has been extensively demonstrated on three real-world recognition problems.

Reinforcement learning has obtained increasing attention in mobile robot mapless navigation in recent years. However, there are still some obvious challenges including the sample efficiency, safety due to dilemma of exploration and exploitation. These problems are addressed in this paper by proposing the Human-in-Loop Policy Search (HILPS) framework, where learning from demonstration, learning from human intervention and Near Optimal Policy strategies are integrated together. Firstly, the former two make sure that expert experience grant mobile robot a more informative and correct decision for accomplishing the task and also maintaining the safety of the mobile robot due to the priority of human control. Then the Near Optimal Policy (NOP) provides a way to selectively store the similar experience with respect to the preexisting human demonstration, in which case the sample efficiency can be improved by eliminating exclusively exploratory behaviors. To verify the performance of the algorithm, the mobile robot navigation experiments are extensively conducted in simulation and real world. Results show that HILPS can improve sample efficiency and safety in comparison to state-of-art reinforcement learning.

As robots are sharing work spaces with human, human-robot teamwork is becoming increasingly important. It is foreseeable that the daily work team will be composed of human and robots. The integration of the appropriate decision-making process is an essential part to design and develop the team. If robots can understand the activities and intents of human, it is convenient for a person to cooperate with robots in a natural manner. This paper proposes a system that enables robots to understand human pose and execute given command. The system provides two options for different hardware systems: the first one is suitable for powerful computational units; the second model is compact and efficient on a normal robot platform. In order to enrich application scenarios, we propose a method to extract human pose from thermal images so that our system can be used in all-weather scenario. In addition, we collected extensive training data and trained a MLP neural network to classify several human poses. The experimental results show the accuracy and efficiency of the proposed MLP neural network in day and night environments.

It is essential in many robot tasks to retrieve depth information, while it still remains a challenging problem to get robust depth in unfavorable conditions such as night or rainy environments. With the development of convolutional neural networks (CNNs), a large number of algorithms have emerged to tackle the problem of dark image enhancement and depth estimation, but there are few works focus on recovering depth map in dark environments and normal light condition. To meet this demand, we proposed a neural network which takes the paired stereo images in all light conditions as input and estimates the fully scaled depth map. The network contains a novel feature extractor and a stereo matching module which follows a light-weight manner to guarantee this work practical for real robotic applications. We introduced the Retinex Theory into depth estimation and trained the decomposition module with LOL dataset. Then it is adapted into depth estimation by fusing the decompose module into stereo matching algorithm. The whole network is then trained in an end-to-end manner. To demonstrate the robustness and effectiveness of our proposed method, we perform various studies and compare our results to the state-of-the-art algorithms in depth estimation as well as direct combination of image enhancement and stereo matching algorithm. We also collect stereo images in real night environments and present the improved performance of our network.

Enabling long-term operation during day and night for collaborative robots requires a comprehensive understanding of the unstructured environment. Besides, in the dynamic environment, robots must be able to recognize dynamic objects and collaboratively build a global map. This paper proposes a novel approach for dynamic collaborative mapping based on multimodal environmental perception. For each mission, robots first apply heterogeneous sensor fusion model to detect humans and separate them to acquire static observations. Then, the collaborative mapping is performed to estimate the relative position between robots and local 3D maps are integrated into a globally consistent 3D map. The experiment is conducted in the day and night rainforest with moving people. The results show the accuracy, robustness, and versatility in 3D map fusion missions.

Performing collaborative semantic mapping is a critical challenge for cooperative robots to maintain a comprehensive contextual understanding of the surroundings. Most of the existing work either focus on single robot semantic mapping or collaborative geometry mapping. In this paper, a novel hierarchical collaborative probabilistic semantic mapping framework is proposed, where the problem is formulated in a distributed setting. The key novelty of this work is the mathematical modeling of the overall collaborative semantic mapping problem and the derivation of its probability decomposition. In the single robot level, the semantic point cloud is obtained based on heterogeneous sensor fusion model and is used to generate local semantic maps. Since the voxel correspondence is unknown in collaborative robots level, an Expectation-Maximization approach is proposed to estimate the hidden data association, where Bayesian rule is applied to perform semantic and occupancy probability update. The experimental results show the high quality global semantic map, demonstrating the accuracy and utility of 3D semantic map fusion algorithm in real missions.

Repetitive environment is a challenging scenario for mobile robot global localization due to its highly similar structures and lack of distinctive features. Existing solutions in such environments rely heavily on pre-installed infrastructures, which are neither flexible nor cost-effective. Besides, few of the previous research have been focused on the implementation of infrastructure-free localization approaches in repetitive scenarios. Thus, this paper serves as a survey to investigate the problem of infrastructure-free mobile robot global localization with low-cost and efficient sensors in repetitive environments. Three of the most popular infrastructure-free localization methods, namely LiDAR-based localization (LBL), vision-based localization (VBL), and magnetic field-based localization (MFL), are analyzed and evaluated. Extensive global localization experiments are conducted in real-world repetitive scenarios and the results demonstrate that VBL methods perform slightly better than LBL and MFL methods. The overall evaluations indicate that infrastructure-free global localization in repetitive environment is still a challenging problem which deserves more research efforts to develop new solutions.

In order to enable a team of robots to operate successfully, retrieving accurate relative transformation between robots is the fundamental requirement. So far, most research on relative localization mainly focus on geometry features such as points, lines and planes. To address this problem, collaborative semantic map matching is proposed to perform semantic perception and relative localization. This paper performs semantic perception, probabilistic data association and nonlinear optimization within an integrated framework. Since the voxel correspondence between partial maps is a hidden variable, a probabilistic semantic data association algorithm is proposed based on Expectation-Maximization. Instead of specifying hard geometry data association, semantic and geometry association are jointly updated and estimated. The experimental verification on Semantic KITTI benchmarks demonstrate the improved robustness and accuracy.

The problem of attitude tracking control for spacecraft with limited communication rate is addressed in this article. To reduce the communication burden, an adaptive event-triggered control scheme is proposed. In the control scheme, only the sampling states at the event-triggering instants are sent to the control module, which can considerably decrease the data transmission rate. To address the inertia uncertainties and external disturbances, a radial basis function neural network (NN) is introduced. The bound of the uncertainties and disturbances is estimated for the proposed control scheme, which can simplify the NN and reduce the computation. Since the event-triggered error signal is discontinuous due to the event-triggered mechanism, the closed-loop system is formulated as an impulsive dynamical system to obtain the stability properties of the system. Finally, simulation results are given to demonstrate the effectiveness of the proposed control scheme.

In order to improve the robot's perception ability in the complicated environment, especially the unstructured environment, a framework of 3D object detection and pose estimation using single shot detector (SSD) and modified LineMOD template matching is proposed, which can detect multiple objects and estimate their pose simultaneously. Firstly, the initial object detection (the first detection) is realized by single shot detector network and therefore the region of interest (RoI) of target objects are generated. LineMOD template matching is then applied to provide candidate templates. These calculated templates are grouped by the designed clustering algorithm. After sorting the clusters according to the descending order of the average similarity, non-maximum suppression removes the similar results and provide the further multiple detection results (the second detection). Finally, based on the results from the second detection, the pose of the object is estimated by using iterative closest point (ICP) algorithm. The object detection experiments show that on Tejani dataset, the average recognition rate of six objects reaches 99.25%. For the object pose estimation, F1 of the proposed method is 21.7% higher than the conventional method in the pose estimation experiments. Also, F1 of the presented algorithm is 9.5% higher than Deep-6Dpose method. Both comparison experiments verify the effectiveness of the proposed framework. Further, this framework for object detection and pose estimation is employed to do robotic grasping. In particular, the workpiece of steel plates is grabbed, which is a necessary procedure of the polishing technique.

In this paper, we address the problem of secure pose estimation of an autonomous vehicle (AV) under cyber attacks. An extended Kalman filter (EKF) is used to fuse measurements from multiple sensors including GPS, LIDAR, and IMU. To deal with the possible sensor attacks, we design a cumulative sum (CUSUM) detector to monitor the inconsistency between the predicted pose via mathematical model and the sensor measurement. An EKF reconfiguration scheme is proposed to mitigate the influence of sensor attacks once the compromised sensor is identified. The feasibility and effectiveness of the proposed secure pose estimation method are validated using a simulation platform built on Autoware and Gazebo.

This paper deals with the problem of recognizing the unique role in dynamic environments. Different from social roles, the unique role refers to those who are unusual in their carrying items or movements in the scene. In this paper, we propose a hierarchical probabilistic reasoning method that relates spatial relationships between interested objects and humans with their temporal changes to recognize the unique individual. Two observation models, Object Existence Model (OEM) and Human Action Model (HAM), are established to support role inference by analyzing the corresponding semantic-interaction features and spatio-temporal features. Then, OEM and HAM results of each person are compared with the overall distribution in the scene, respectively. Finally, we can determine the role through the fusion of two observation models. Experiments are conducted in both indoor and outdoor environments concerning different settings, degrees of clutter, and occlusions. The results show that the proposed method can adapt to a variety of scenarios and outperforms other methods on accuracy and robustness, moreover, exhibiting stable performance even in complex scenes.

Fusion of local 3D maps generated by individual robots to a globally consistent 3D map is one of the fundamental challenges in multi-robot mapping missions. In this paper, we propose a probabilistic mathematical formulation to address the integrated map fusion problem. More specifically, the problem of estimating fused map posterior can be factorized into a product of relative transformation posterior and the global map posterior, which enables us to solve map matching and map merging problems efficiently. In addition, a distributed communication strategy is employed to share map information among robots. The proposed approach is evaluated in indoor and mixed environments, which shows its utility in 3D map fusion for multi-robot mapping missions.

Wind disturbance has alway been a critical challenge for safety flight and high precision control of UAV due to the uncertainty of wind in time and space domain. To this end, an acceleration feedback (AF) enhanced H∞controller is proposed to enhanced the ability of UAV against wind disturbance and it has been deigned and implemented on a hex-rotor. Firstly, The dynamic of UAV system is decoupled into inner-loop (attitude) and outer-loop (position). Then, a hierarchical H∞controller is designed for the decoupled system. Finally, a AF enhanced method is introduced into the system without changing controller structure. The stability of the AF enhanced method for the UAV system is analyzed and can be ensured by H∞theory as well. The comparison results of trajectory tracking performance between H∞controller and AF enhanced H∞controller under continuous and gusty wind verify that the proposed method is not only robust but effective for both types of wind disturbances.

To obtain the 6 DOF extrinsic parameters (rotation and translation matrix) between a 3D ranging sensor and a thermal camera, previous methods require a high-resolution 3D ranging sensor to reliably detect features. Although sparse 3D LiDARs are widely used on autonomous robots, to the best of our knowledge, the extrinsic calibration between a sparse 3D LiDAR (particularly Velodyne VLP-16) and a thermal camera has not been considered in the literature. In this paper, we present a two-step method to address the problem, where a monocular visual camera is used to assist the process. The proposed method decomposes the problem into two steps: extrinsic calibration between a sparse 3D LiDAR and a visual camera; extrinsic calibration between a visual camera and a thermal camera. Experiments are conducted to demonstrate the effectiveness of the proposed two-step method.

We propose a robust recurrent kernel online learning (RRKOL) algorithm based on the celebrated real-time recurrent learning approach that exploits the kernel trick in a recurrent online training manner. The novel RRKOL algorithm guarantees weight convergence with regularized risk management through the use of adaptive recurrent hyperparameters for superior generalization performance. Based on a new concept of the structure update error with a variable parameter length, we are the first one to propose the detailed structure update error, such that the weight convergence and robust stability proof can be integrated with a kernel sparsification scheme based on a solid theoretical ground. The RRKOL algorithm automatically weighs the regularized term in the recurrent loss function, such that we not only minimize the estimation error but also improve the generalization performance through sparsification with simulation support.

This paper introduces a new methodology for environment characterization. This methodology is based on analysis of the eigenvalues of Laplace-Beltrami operator over 3 dimensional point clouds. Recognizing revisited places can be facilitated by characterizing the environment through a descriptor. The idea of analyzing point clouds using the eigenvalues of Laplace-Beltrami operator for characterization of an environment can be used for place detection which is a critical functionality of autonomous mobile robots. Place detection is a requirement for transition detection in multi environment missions, common frame identification in multi robot mapping, and detection of previously visited location in SLAM for loop closure phase.

This paper studies navigation for multiple mobile robots while avoiding collisions and ensuring global network connectivity in unknown environments. It is found that when the traditional navigation function is employed, control input is extremely small if robots work in a large environment, which implies that robots will almost stop at their initial positions. To solve this problem, a new decentralized navigation function is proposed with a novel goal function which is then applied in a multi-robot navigation scenario. Based on the properties of navigation function and dual Lyapunov theorem, a sufficient condition is derived for robots converging to regions surrounding their corresponding goal positions in a collision-free and connectivity-keeping manner. Simulation results demonstrate the efficacy of the proposed method.

Fault severity estimation is an important part in condition-based maintenance, by which a right time and suitable maintenance policy can be planned. For stator winding inter-turn short circuit fault, its severity is represented by two parameters, percentage of shorted turns and fault loop resistance. Research on severity estimation of this fault has been focused on late stage of the fault development process, when the estimation of only one parameter, percentage of shorted turns, is required. As an attempt to overcome this, this paper proposes a method to estimate both parameters employing nonlinear Kalman filter using an equality constraint on them. The constraint is derived from a sequence component model analysis. The estimation of both fault parameters helps track the degradation process from its incipient stage.

So far, most of the facial recognition methods focus on visual image texture and color information. Although they have worked well, most of them still fail to deal with severe illumination changes. In this paper, a novel approach is proposed for facial verification by analyzing thermal data from different organs of the human face. This new thermal facial pattern is free from illumination changes and can even work in a very dark place. In this experiment, facial thermal data were collected from 30 people with diverse genders, ages, and races. Three persons were tracked to verify the consistency of the thermal pattern in normal circumstances, changing light conditions and different physical conditions. A new distance function is introduced for pattern similarity measurement. The proposed approach successfully distinguished different persons with a high verification rate 91.26% according to F-measure and displayed the thermal pattern changes according to different physical conditions.

One of the major challenges in multi-robot exploration is to fuse the partial maps generated by individual robots into a consistent global map. We address 3D volumetric map fusion by extending the well known iterative closest point(ICP) algorithm to include probabilistic distance and surface information. In addition, the relative transformation is evaluated based on Mahalanobis distance and map dissimilarities are integrated using relative entropy filter. The efficiency of the proposed algorithm is evaluated using maps generated from both simulated and real environments and is shown to generate more consistent global maps.

This paper deals with a model-based approach for diagnosis switch faults in multi-level multi-phase voltage source inverters (VSI). The proposed fault detection and isolation (FDI) method requires only one voltage detector per phase leg to accurately allocate a faulty switch among 2m(n-1) switches in an n-level m-phase VSI. The pole voltage of each phase is monitored and digitized into 2n-1 levels. A fault is detected when an odd level of voltage is observed. After the fault is detected, certain switching modes are applied to the inverter and the resulting voltage levels are observed. By comparing finite numbers of the observed voltage levels to the fault signatures, the location of the faulty switch is accurately determined. The control of the switches is then changed to tolerate the faulty condition. This method is robust to the load variations and suitable for closed loop applications. It is applicable to micro inverters for solar systems with numerous semiconductor switches with minimum number of sensors.

Stator winding inter-turn fault is a common fault in induction machines. A substantial number of works have been developed for detection of this fault; however, there are only a few works on fault parameter estimation and faulty phase identification. In particular, there have been no works dealing with estimation of fault loop resistance. This paper presents a method for estimating the fraction of shorted turns and the fault loop resistance and identifying the faulty phase under steady-state condition. It provides a closed-form solution and hence is applicable to online fault detection and identification. Simulation results demonstrate the effectiveness of the proposed method.

In this paper, a novel approach of sensor placement is proposed for the purpose of maximizing fault detectability and isolability. This new approach rests on the basic fact that faults are embedded in the analytical redundancy relations (ARRs) and that the occurrence of a fault will change the consistency of the corresponding ARRs. Based on these basic facts, the minimal isolating (MI) set is introduced to formulate the full/maximal isolability which is the constraint for sensor placement. Consequently, the optimization problem for sensor placement is reformulated as searching an MI set which is related to the least number of candidate sensors. To find the optimal MI set, a low complexity dynamic programming (LCDP) algorithm is developed on the fault set F that consists of system faults and sensor faults. However, sensor faults are varied as different candidate sensors are used. Therefore, another dedicated procedure is proposed to handle this issue. A case study shows that the proposed approach outperforms an existing sensor placement approach in terms of efficiency.

In this paper, an enhanced data-driven optimal terminal iterative learning control (E-DDOTILC) is proposed for a class of nonlinear and nonaffine discrete-time systems. A dynamical linearization approach is first developed with iterative operation points to formulate the relationship of system output and input into a linear affine form. Then, an ILC law is constructed with a nonlinear learning gain, which is a function about the system partial derivative with respect to the time-varying control input. In addition, a parameter updating law is designed to estimate the unknown partial derivatives iteratively. The input signals of the proposed E-DDOTILC are time-varying and updated utilizing not only the terminal tracking error of the previous run but also the input signals of the previous time instants in the current iteration. The proposed approach is a data-driven control strategy and only the I/O data are required for the controller design and analysis. The monotonic convergence and effectiveness of the proposed approach is further verified by both the rigorous mathematical analysis and the simulation results.

In repetitive control systems, the period of the disturbance may not be integral multiple of sampling rate or may change with time slightly. This will affect the performance of RC controller seriously. In this paper, a robust repetitive control method is proposed to handle non-integral delay in RC control systems. Another feed-forward path is added to the conventional odd-harmonic controller to guarantee that the control gain for all odd-harmonics remain satisfied even when the period of the fundamental signal suffered from disturbance or is not integral multiple of the sample period. Stability and convergence of this approach are proved mathematically. Computer simulation results are provided to confirm its effectiveness further.

With the increasing demand of high reliability and safety of modern electric machines, failure prognosis becomes more and more important since it is efficient to increase reliability and reduce downtime cost. In this work, a model-based remaining useful life (RUL) prediction method is developed for induction motor with stator winding short circuit fault. The induction motor model with stator winding short circuit fault is introduced based on reference frame transformation theory. The winding short circuit fault is characterized by the fraction of short turns and the fault loop resistance. In this paper, the motor life is defined as the stator winding insulation life due to thermal stresses because from a thermal point of view, the stator winding insulation is the weakest part of induction motors. A particle filter method is used to realize unknown parameter estimation and RUL prediction. Simulation results are provided to validate the proposed method.

An improved cascade-type repetitive controller with forward channel gain is proposed for a grid-tied inverter with LCL filter to achieve fast transient response and very low total harmonic distortion. The choices of the RC parameters such as RC control gain krc, the phase compensator parameter m have been discussed. The simulation results show the improved cascade-type RC has fast dynamic response, high steady accuracy and good robustness.

This paper presents the detection and localization methods of entrance and staircase markers for the team E-Mobile in TechX Challenge 2013. Autonomous vehicles are required to detect and locate traffic cones beside the indoor entrance and staircase. One big challenge is from the unpredictable lighting conditions and environment. Different practical techniques such as color space selection, segmentation, shape analysis, distance estimation, and detector training are combined to obtain good detection rate and localization accuracy. The proposed methods can achieve satisfactory performance in real-world experiments.

This paper addresses the distributed formation trajectory planning for a group of nonholonomic vehicles. This is realized with a decentralized Model Predictive Control under dynamic virtual structure architecture. A specific limitation of virtual structure based formation method is the necessity of access to the desired reference. To remove this requirement, a distributed estimator is developed so that each vehicle can construct the desired reference based on the local information exchange. In formation trajectory planning, several issues are taken into consideration which includes: (i) distributed formation achievement by a team of nonholonomic vehicles from initial situation. (ii) inter-group collision avoidance. (iii) dynamic formation to obtain flexible manoeuvring during movement in unknown and cluttered environment. (iv) obstacle avoidance. Finally, simulation results are presented to illustrate the performance of the proposed methodology in producing optimal formation trajectory planning for multiple nonholonomic vehicles.

In this paper, a detailed study on a composite derivative is performed. The composite derivative, which is formed from the combination of fractional integration and derivative and performs a $90^\circ$ phase shift as the traditional first derivative does, is applied to edge detection and the results are analyzed, emphasizing the compromise ability between selectivity and noise suppression. Both objective and subjective comparisons with other edge detectors are carried out, including evaluations through the use of the benchmark Berkeley Segmentation Dataset (BSDS500). In contrast with the classical first-order derivative, the composite derivative is order-steerable; one can adjust the orders of fractional integration and derivative to tune magnitude characteristic and reach a compromise between sensitivity to noise and detection accuracy.

Extreme learning machine (ELM) was initially proposed for single-hidden-layer feedforward neural networks (SLFNs). In the hidden layer (feature mapping), nodes are randomly generated independently of training data. Furthermore, a unified ELM was proposed, providing a single framework to simplify and unify different learning methods, such as SLFNs, least square support vector machines, proximal support vector machines, and so on. However, the solution of unified ELM is dense, and thus, usually plenty of storage space and testing time are required for large-scale applications. In this paper, a sparse ELM is proposed as an alternative solution for classification, reducing storage space and testing time. In addition, unified ELM obtains the solution by matrix inversion, whose computational complexity is between quadratic and cubic with respect to the training size. It still requires plenty of training time for large-scale problems, even though it is much faster than many other traditional methods. In this paper, an efficient training algorithm is specifically developed for sparse ELM. The quadratic programming problem involved in sparse ELM is divided into a series of smallest possible sub-problems, each of which are solved analytically. Compared with SVM, sparse ELM obtains better generalization performance with much faster training speed. Compared with unified ELM, sparse ELM achieves similar generalization performance for binary classification applications, and when dealing with large-scale binary classification problems, sparse ELM realizes even faster training speed than unified ELM.

Although simultaneous localization and mapping (SLAM) algorithms are widely appreciated in mobile robot navigation, they can be further improved to suit practical applications in dynamic environmental conditions. One such important improvement is the detection and tracking of moving objects present in the sensor field of view (FOV). In this paper we propose to extend our recently introduced Collaborative Multi-vehicle SLAM (CMSLAM) solution based on the random finite set (RFS) representation of the feature map and measurements, by tracking both static and dynamic features. We represent static features observed during the SLAMprocess, along with dynamic features present in the current sensor FOV, as an augmented RFS. The corresponding probability density is propagated using a Bayes recursion, from which the static feature map and the estimates of dynamic feature locations can be obtained. Measurement update in the CMSLAM process is carried out only using the static feature map to take advantage of obvious accuracy improvements.

This paper presents a new approach for diagnosis of short circuit Insulated Gate Bipolar Transistor (IGBT) faults in a three-phase voltage-fed motor drive. The three-phase Pulsewidth Modulated (PWM) voltage signals are monitored and digitised into two discrete levels. The combination of discrete voltage levels of the three phases composes eight voltage states. For a healthy Sinusoidal PWM (SPWM) motor drive, all eight states are monitored within one carrier period. However, when a fault occurs certain states cannot be obtained. A fault signature set is defined for each switch fault. By comparing the monitored states with the fault signature sets, the faulty switch is detected and isolated. A PWM voltage source inverter that feeds a threephase Permanent Magnet Synchronous motor with IGBT switch faults is simulated and the fault diagnosis algorithm is applied. The results show that the faults can be diagnosed within tens of microseconds. The robustness of the proposed fault diagnosis and isolation (FDI) approach to the load changes is tested by connecting the motor to a variable mechanical torque.

In this work, a new iterative feedback tuning approach is proposed to tune ALINEA's controller gain automatically when there is not enough prior information available to select a proper feedback gain of ALINEA. It is a data-driven method and the ALINEA controller is auto-tuned only depending on the input and output data collected from closed-loop experiments. To mimic a real traffic environment, a simulator is built on the PARAMICS platform. The flow-based ALINEA controller is also considered to illustrate the good tuning performance of IFT comprehensively. The effectiveness of the proposed methods is verified through PARAMICS based simulations.

Hybrid systems operate in various states, which are represented by a set of modes. In each mode, the system is governed by continuous dynamics, and different modes correspond to different continuous models. For hybrid systems, model-based fault detection and isolation is a challenging task due to the fact that the system's prevailing dynamical model and its current mode (discrete state) are mutually dependent and intertwined. In this paper, a new energy-based approach is introduced for mode tracking of hybrid systems, and its associated systematic analysis is based on a hybrid bond graph. Each system's mode is characterized by a concise energy relation that allows mode identification in the new mode tracking method.

Inspired by the biological concept of central pattern generators (CPGs), this paper deals with adaptive walking control of biped robots. Using CPGs, a trajectory generator is designed consisting of a center-of-gravity (CoG) trajectory generator and a workspace trajectory modulation process. Entraining with feedback information, the CoG generator can generate adaptive CoG trajectories online and workspace trajectories can be modulated in real time based on the generated adaptive CoG trajectories. A motion engine maps trajectories from workspace to joint space. The proposed control strategy is able to generate adaptive joint control signals online to realize biped adaptive walking. The experimental results using a biped platform NAO confirm the effectiveness of the proposed control strategy.

Motor drives are widely used in industry for controlling the speed of three phase AC motors. Faults in motor drives degrade motor performance and can cause catastrophic failures. IGBT (Insulated Gate Bipolar Transistor) switch faults are one of the main roots of electrical faults in inverters and motor drives. In this paper, a method based on neural network is implemented to detect and isolate switch faults in a three phase voltage source inverter. Only the output signals of the inverter are monitored. The entropy of the phase current and voltage is selected as the switch fault feature. Single and multiple short and open circuit switch faults are isolable with this method.

This paper presents a novel methodology for the purpose of fault detection and isolation (FDI) to a two-tank system. This new methodology benefits from the basic facts that faults are embedded in the analytical redundancy relations (ARRs) and that the occurrence of a fault will cause the corresponding ARRs to change. Based on these facts, the minimal isolation set as an important concept is introduced to make each fault in the fault set F detectable and isolable. Then, the sensor placement problem consists in determining an optimal minimal isolation set associated with the least number of sensors. A dedicated genetic algorithm is developed to solve the formulated sensor placement problem. A case study of a two-tank system shows that the proposed methodology performs well.

Most of the available results of adaptive iterative learning control (AILC) hitherto have considered the control systems with known linearly parameterized structures. A dynamical linearization approach is developed for a general nonlinear multiple input multiple output systems. And then a discrete-time adaptive ILC approach is presented to deal with the ILC problems of nonlinear MIMO systems with iteration-varying initial error and reference trajectory. The controller design and analysis is completely data-driven without using any modeling information of the plant, but the measured I/O data only. The almost perfect tracking performance is asymptotically guaranteed by rigirous mathematical analysis.

This paper proposes a novel velocity-free nonlinear proportional-integral (PI) control allocation scheme for fault tolerant attitude control of flexible spacecraft under thruster redundancy. More specifically, the nonlinear PI controller for attitude stabilization without using body angular velocity measurements is firstly designed as virtual control of a control allocator to produce the three axis moments, and can guarantee uniform ultimately boundedness of the closed-loop system in the presence of external disturbances and possible stuck faults. The associated stability proof is constructive and accomplished by the development of a passivity filter formulations together with the choice of a Lyapunov function containing cross/mixed terms involving the various states. Then, a robust least squares based control allocation is employed to deal with the problem of distributing the three axis moments over the available thrusters under redundancy, in which the focus of this control allocation is to find the optimal control vector of actuator by minimizing the worst-case residual, under the condition of thruster faults and control constraints like saturation.

In this paper, we integrate information from a hybrid bond graph (HBG) model and discrete event systems (DES) into a fault diagnosis method for hybrid systems. In a pure HBG framework, mode change detection and isolation is handled by the mode change signature and the mode change signature matrix. In a DES approach, discrete states and faults are traced based on observable events and diagnosers. The integration of the two approaches is based on a new diagnoser that is driven by both, observable events and consistency indicators generated by continuous residuals. The proposed method allows not only to effectively trace the system mode, but also to decide whether this mode is faulty or normal. The new method is presented along with a theoretical example.

In this paper we present a novel solution to the Multi-Vehicle SLAM (MVSLAM) problem by extending the random finite set (RFS) based SLAM filter framework using two recently developed multi-sensor information fusion approaches. Our solution is based on the modelling of the measurements and the landmark map as RFSs and factorizing the MVSLAM posterior into a product of the joint vehicle trajectories posterior and the landmark map posterior conditioned the vehicle trajectories. The joint vehicle trajectories posterior is propagated using a particle filter while the landmark map posterior conditioned on the vehicle trajectories is propagated using a Gaussian Mixture (GM) implementation of the probability hypothesis density (PHD) filter.

Trade-off between the information gain and cost has been extensively used as an evaluation criteria for target points in robot exploration strategies where the primary goal is to reduce the mission time. This article introduces a new gain function that has an integrated cost component that can be used in exploration strategies to map the terrain in a balanced way in all directions to generate compact maps. Article also presents an approach to efficiently calculate the gain values. Simulations in both low and high obstacle density environments for single and multi-robot exploration strategies indicate the utility of the new gain function in generating compact maps.

We propose a robust recurrent kernel online learning (RRKOL) algorithm which allows the exploitation of the kernel trick in an online fashion. The novel RRKOL algorithm achieves guaranteed weight convergence with regularized risk management through the recurrent hyper-parameters for a superior generalization performance. To select useful data to be learned and remove redundant ones, a sparcification procedure is developed based on the stability analysis of the system. Two time-series prediction examples are presented.

A Bond Graph model is built for the steering system of automatic vehicle and a set of model equations are derived for further analysis purpose. For identifying several uncertain parameters, an integrative approach that combine least square method with Bp Neural Network algorithm (NN) is proposed, based on features of NN algorithm, two key improvements are bring into the training method of Bp NN: taking the identification result of least square method as initial weight value of network training, and introducing weight factor to improve the convergence property of Bp NN. The effectiveness of proposed approach is verified through experiment, and the result indicates that the reformatory Bp NN algorithm has higher identification accuracy.

This paper presents a locomotion control strategy for quadruped robots based on central pattern generator (CPG). The proposed control architecture consists of a workspace trajectory generator and a motion engine. The CPG-inspired trajectory generator can generate workspace trajectories and the motion engine can calculate the accurate joint control signals. Moreover, entrainment with sensory feedback information from robot-environment interaction, the presented control system can generate adaptive joint control signals. A quadruped platform AIBO is used to validate the proposed control architecture and experimental results confirm the effectiveness of the control system.

This paper deals with the locomotion control of quadruped robots inspired by the biological concept of central pattern generator (CPG). A control architecture is proposed with a 3-D workspace trajectory generator and a motion engine. The workspace trajectory generator generates adaptive workspace trajectories based on CPGs, and the motion engine realizes joint motion imputes. The proposed architecture is able to generate adaptive workspace trajectories online by tuning the parameters of the CPG network to adapt to various terrains. With feedback information, a quadruped robot can walk through various terrains with adaptive joint control signals. A quadruped platform AIBO is used to validate the proposed locomotion control system. The experimental results confirm the effectiveness of the proposed control architecture. A comparison by experiments shows the superiority of the proposed method against the traditional CPG-joint-space control method.

In this paper, a model-based fault prognosis method is developed for hybrid systems with multiple incipient faults. The concept of augmented global analytical redundancy relations is proposed for the identification of degradation of components, such as sensors and actuators, which cannot be described by physical parameters. In addition, multiple incipient faults are considered in a complex hybrid system, and these faults can develop during a mode when the faults are not detectable. The unknown degradation characteristic of each incipient fault is identified with the closest matching one of some prescribed dynamic models. The resultant degradation model will serve as a base for prognosis. In the process of fault detection and isolation, the degradation models and faults are identified using a multiple-adaptive-hybrid-particle-swarm-optimization algorithm. The proposed methodology and algorithm are verified with simulation as well as experiments.

This paper describes a GA approach for solving the SLAM problem. We treat the local scan from laser sensor as an image pattern. Two subsequent scans were matched to find the robot's ego motion. This motion is used as updates for robot's global position estimation in the map. A virtual scan is obtained from the map in the robot's current position. This virtual scan is then matched with the current laser scan to update robot's location in the map. The innovation in this paper is the matching algorithm to determine the relative location of two patterns: (1) two successive scans and (2) current scan with virtual scans taken from the map. The matching algorithm is to find the optimal location of one pattern to the other, namely (dx, dy, de) (two for translation and one for rotation). This is reduced to a three-dimensional search problem. The GA we developed can achieve high accuracy with fast computational time. Experiments are performed under simulated and real data. The performance obtained outperforms ICP.

In this paper, a recurrent neural network (RNN) based robust tracking controller is designed for a class of multiple-input-multiple-output (MIMO) discrete time nonlinear systems. The RNN is used in the closed-loop system to estimate online unknown nonlinear system function. A multivariable robust adaptive gradient-descent training algorithm is developed to train RNN. The proposed neural control system guarantees the stability of the closed-loop system and good tracking performance is achieved.

Image edge detection is a classic problem of machine vision and image processing. Edge detection often uses an integer-order differential operation. The paper adopts fractional differentiation and integration to obtain a new edge detection operator. The performances in terms of detection accuracy and noise immunity of the new operator are compared with those of the traditional operators through examples. The comparison shows that the new operator is promising.

This article studies fault detection and isolation (FDI) and fault estimation in complex hybrid systems. The FDI approach is based on a set of unified constraints, called augmented Global Analytical Redundancy Relations (AGARRs), to detect and isolate the faults. In order to estimate the magnitude of the fault parameter in the fault candidates, a differential evolution (DE) method is employed. This developed method is applicable to estimation of multiple faults of parametric and nonparametric nature. Simulation is carried out to verify the effectiveness of the proposed method in a front steering system of a CyCab mobile robot with multiple faults.

This paper considers the problem of cooperative target tracking with multiple unmanned aerial vehicles (UAVs) subject to input constraints. Cooperation of multiple UAVs to track a moving target can provide better performance and enhance the robustness. However, the physical constraints of the UAVs pose a significant challenge on the UAV controller design. In this paper, the relative course rate controller for a single UAV is firstly developed based on a guidance vector field. Cooperative control strategy of multiple UAVs is studied and a variable airspeed controller is proposed to regulate temporal separation of UAVs. Simulation results are provided to demonstrate the effectiveness of the proposed approach.

This research result consists of two parts: one is general theory on causality assignment for hybrid bond graph (HBG) and another is application of this concept to the quantitative fault diagnosis. From Low et al., 2008, a foundation for quantitative bond graph-based fault detection and isolation (FDI) design using HBG is laid. Useful causality properties pertaining to the HBG from FDI perspectives, and the concept of diagnostic hybrid bond graph (DHBG) which is advantageous for efficient and effective FDI applications are proposed. This paper is a continuation of our previous paper (Low et al., 2008). Here, the DHBG is exploited to analyze the hybrid system's fault detectability and fault isolability. Additionally, a quantitative FDI framework for effective fault diagnosis for hybrid systems is proposed. Simulation and experimental results are presented to validate some key concepts of the quantitative hybrid bond graph-based FDI framework.

Bond graph (BG) is an effective tool for modeling complex systems and it has been proven useful for fault detection and isolation (FDI) for continuous systems. BG provides the causal relations between system's variables which allow FDI algorithms to be developed systematically from the graph. In the same spirit, Hybrid bond graph (HBG) is a BG-based modeling approach which provides an avenue to model complex hybrid systems. However, due to mode-varying causality properties of HBG, HBG has not been efficiently-exploited for fault diagnosis. In this work, a comprehensive study on the HBG from FDI viewpoints is presented. Some properties pertaining to the HBG are gained in the study. Based on these findings, a causality assignment procedure and a model approximation technique are developed to achieve a HBG with a desirable causality assignment that leads a unified description of system's behavior. These results lay a foundation for quantitative FDI design for complex hybrid systems.

In this work we present a new health monitoring method for hybrid systems. The method utilizes the concept of unified constraint relations, named the global analytical redundancy relations (GARRs). Using GARRs for health monitoring of hybrid system requires the system's current mode and this information is provided by a mode tracker. To make the mode tracking more efficient, a unique mode-change isolation process is utilized, this process is based on the mode change signature matrix (MCSM).

Hysteresis characteristics is highly nonlinear, has memory and is common in engineering systems. Its presence introduces uncertainties and nonlinearity in dynamic modelling and thus difficulties in achieving a good control design. This paper studies the suitability of iterative learning control (ILC) to compensate hysteresis uncertainties for a class of continuous-time dynamic systems. We examine dynamic systems with Preisach model hysteresis nonlinearity. It is shown that this class of systems possess properties of continuity and repeatability which are required for ILC. Furthermore, anticipatory iterative learning control (or A-type ILC) is applied to overcome the uncertainties and nonlinearity introduced by hysteresis. Simulation results are presented to validate the effectiveness of ILC laws to eliminate tracking error due to hysteresis uncertainties.

This paper discusses a vision problem for the detection of lift operation panel which is a plane subject to deformation due to viewing angle change. The key problem in this project is the large scale change where the panel detection has to start at a significant distance away, and end up with the camera very near to the panel button. The process has three steps: (1) hunting for the panel using a coarse searching algorithm; (2) guiding the robot arm towards the panel; at close range, a model based matching algorithm is applied to verify (or identify) the panel; (3) after verification, tracking the button to guide the robot towards it. Two algorithms were used and it is shown that the weak perspective model outperforms the affine model.

In this paper, three iterative learning control (ILC) schemes are developed in the multirate signal processing domain. One is pseudo-downsampled ILC, in which the input update rate is different from the sampling rate of feedback system. The second one is a two-mode ILC, in which the input update rates of ILC are different at low and high frequency bands. The third one is a cyclic pseudo-downsampled ILC, which extends the first scheme by shifting downsampling points in different iterations. Theoretical background and design approaches of these multirate schemes are addressed. Experimental results are presented to highlight the traits of each scheme. The advantage is that these schemes have the ability to learn those error component beyond the learnable bandwidth of a conventional ILC and, therefore, can improve the tracking accuracy substantially. Additionally, the multirate ILC schemes have the abilities to produce good learning transient with the presence of initial state error.

This paper aims to give a general and unifying presentation on modeling of wheel mobile robots (WMRs) in the presence of wheel skidding and slipping from the perspective of control design. We present kinematic models that explicitly relate perturbations to the vehicle skidding and slipping. Four configurations of mobile robots are considered, and perturbations due to skidding and slipping are categorically classified as input-additive, input multiplicative, and/or matched/unmatched perturbations. Furthermore, we relate the WMR's maneuverability with the vehicle controllability that provides a measure on the WMR ability to track a trajectory in the presence of wheel skidding and slipping. These classifications and formulations lay a base for the deployments of various control design techniques to overcome the addressed perturbations.

This paper presents a scheme for high-update rate wheel mobile robot (WMR) posture, velocities, and perturbation estimation using real-time kinematic global positioning system (RTK-GPS) and inertial sensors for WMR control in the presence of wheel skidding and slipping. An outdoor estimation system based on Kalman filtering combines the inertial measurements with centimeter accuracy RTK-GPS measurements to provide essential posture, velocities, and perturbation information. The particular contribution of this paper is in designing an estimation system to be able to deal with WMR control problems in the presence of wheel skidding and slipping. The experimental results suggest that with careful modelling of WMR, the estimation scheme is able to provide reliable and high update rate information for WMR control applications in the presence of wheel skidding and slipping.

This paper presents an analysis on wheeled mobile robots in the presence of wheel skidding and slipping from the perspective of control design. The analysis is based on the kinematic models that are recently developed from control perspective (Wang and Low 2006). Four generic mobile robots are considered in this analysis. We relate the robot's maneuverability with its controllability which provides insights on the robot's ability to track a trajectory in the presence of wheel skidding and slipping. These findings lay a base for the deployments of various control design techniques to overcome mobile robot control problems in the presence of wheel skidding and slipping.

In this paper, the stable full-state tracking problem of a four-wheel-steering vehicle based on the output tracking strategies are investigated. Dynamics of such vehicle is nonholonomic and pose challenging problems for control design and the stability analysis. The dynamics formulated in terms of full state tracking errors offer some properties that allow better understanding of internal and zero dynamics of the tracking error system. Sufficient conditions are derived to ensure the stability of the internal dynamics. We show that the stability of the internal dynamics is mainly guaranteed by suitably defining output function and by suitably confining the behavior of the desired trajectory. Since sufficient conditions to confine the desired trajectory behavior are rather mild, we conclude that the output tracking control laws can be applied very well to solve the full-state tracking problem of a nonholonomic four-wheel-steering vehicle

The earlier works by Zhang, B. et al, (2004) on cutoff-frequency phase-in ILC show that the scheme can suppress initial state error/position offset properly and improve tracking accuracy. However, since cutoff frequency is set high in initial phase of operation cycles, the improvement of tracking accuracy is mainly in this phase and the tracking error in later phase of operation cycles can still be large. A uniformly good tracking accuracy is favorable in many applications. In this paper, a sliding cutoff-frequency phase-in ILC is proposed to achieve this goal. In this scheme, cutoff frequency profile moves along the time axis after some cycles according to the assessment of tracking performance. Experimental results on an SCARA robot show that this scheme can further improve the tracking accuracy and generate a uniform tracking error over the entire operation interval

This present paper aims to give a general and unifying presentation on modeling of WMRs in the presence of wheel skidding and slipping from the perspective of control design. We present kinematic models that explicitly relate the perturbations to the vehicle skidding and slipping. Four configurations of mobile robots are considered and the perturbations due to skidding and slipping are categorically classified as input additive, input multiplicative, and/or matched/unmatched perturbations. These classifications and formulations lay a base for the deployments of various control design techniques to overcome the addressed perturbations

This paper presents a fault-tolerant tracking controller for a platoon of two mobile robots. A vehicle model and a unified controller are proposed for both look-ahead and look-behind tracking of a four-wheel-steering vehicle. The controller can handle situations where faults occur at the two steering systems of the vehicle by making use of any operational systems to steer the vehicle. Tracking stability is ensured by the proper selection of design parameters. Experimental results show the control scheme work properly even in the situations when faults are on and off often at different parts of the vehicle

In this paper, a discrete wavelet transform-based cutoff frequency tuning method is proposed and experimental investigation is reported. In the method, discrete wavelet packet algorithm, as a time-frequency analysis tool, is employed to decompose the tracking error into different frequency regions so that the maximal error component can be identified at any time step. At each time step, the passband of the filter is from zero to the upper limit of frequency region where the maximal error component resides. Hence, the filter is a function of time as well as index of cycle. The experimental results show that this method can suppress higher frequency error components at proper time steps. While at the time steps where the major tracking error falls into lower frequency range, the cutoff frequency of the filter is set lower to reduce the influence of noises and uncertainties. This way, learning transient and long-term stability can be improved.

Repositioning is required in mechanical systems performing repetitive tasks. The effect of poor accuracy of repositioning on tracking performance of iterative learning control has been fully understood. The iterative learning ensures the system output to follow the desired trajectory with a specified error bound, proportional to the bound on initial condition offsets. In this paper, varying-order learning method is proposed to enhance tracking performance by lowering the size of the error bound. A discrete-time initial rectifying action is introduced, by which the system output achieves the desired trajectory jointed smoothly with a transient trajectory from the starting position. Practical schemes are presented based on the varying-order learning and illustrated by numerical results of a single rigid link example.

Positioning of a mobile robot is critical to its movement in an outdoor environment to accomplish tasks with success and accuracy. For outdoor automation with a fleet of mobile robots in a local area, a common positioning system for all vehicles is essential for maneuvering and coordination. The positioning system should be referenced to a global coordinate frame and requires few artificial landmarks only for the areas that are out of the GPS positioning system. In addition, positioning system is desirable to have a high update rate for real-time control purposes. In this paper, we propose an integrated positioning system (IPS), which is able to provide an accurate and high update rate position estimate. The integrated system utilizes GPS as the global coordinate positioning system, Nav200 as the local positioning system for areas that are not covered by GPS signals and encoders/compass as the supplementary relative positioning system. The structure of the IPS and techniques are described. Experimental results are presented to show the working and output of the IPS.

This paper presents a platooning controller for a four-wheel-driving four-wheel-steering vehicle to follow another. The controller is based on the full-state tracking theory and utilizes a vehicular model that makes it able to continue to operate when faults are detected at its steering systems or driving motors which are disabled accordingly. The unified controller is also able to track and follow the target either moving forward in front or moving backward in the back of the vehicle making the real-time implementation of different tracking modes simple. Tracking stability is secured by the proper selection of design parameters. Simulations show the proposed control scheme works properly even in the presence of faults at several different parts.

In this paper, a frequency tuning method based on time-frequency analysis of error signal is presented for iterative learning control (ILC). Qualitative analysis and experimental investigation are presented. The method uses wavelet packet algorithm to decompose the error signal so that the maximal error component at any time step can be identified. The cutoff frequency of the filter at each time step is set to cover the frequency band up to the frequency region where the maximal error component resides. The proposed method allows high frequency error components enter the learning at proper time steps. While at other time steps, the cutoff frequency is set low to guarantee the good learning transient and long-term stability.

This paper presents a passive nonlinear observer for station keeping of autonomous underwater vehicles (AUV) operating in shallow water area. Station keeping is the ability of a vehicle to maintain position and orientation with regard to a reference object. The observer is used to estimate the shallow water wave velocity. The observer is shown to be globally exponentially stable. The observer has been illustrated using an AUV and the simulation shows the good performance of the observer.

In this paper, a learning control scheme is proposed to improve robot tracking accuracy. Through the analysis in frequency domain, it is shown that phase lead compensation can broad the learnable frequency band of a learning control system. The phase lead compensation is realized by phase lead filtering the error of last repetition. In theory a filter whose phase difference to the system is within /spl plusmn/90/spl deg/ can be a candidate for the phase lead compensation process. Experimental results on an industrial robot system show that the proposed scheme is both effective and robust against dynamic modeling errors.

Fuzzy logic systems are promising for efficient obstacle avoidance. However, it is difficult to maintain the correctness, consistency, and completeness of a fuzzy rule base constructed and tuned by a human expert. A reinforcement learning method is capable of learning the fuzzy rules automatically. However, it incurs a heavy learning phase and may result in an insufficiently learned rule base due to the curse of dimensionality. In this paper, we propose a neural fuzzy system with mixed coarse learning and fine learning phases. In the first phase, a supervised learning method is used to determine the membership functions for input and output variables simultaneously. After sufficient training, fine learning is applied which employs reinforcement learning algorithm to fine-tune the membership functions for output variables. For sufficient learning, a new learning method using a modification of Sutton and Barto's model is proposed to strengthen the exploration. Through this two-step tuning approach, the mobile robot is able to perform collision-free navigation. To deal with the difficulty of acquiring a large amount of training data with high consistency for supervised learning, we develop a virtual environment (VE) simulator, which is able to provide desktop virtual environment (DVE) and immersive virtual environment (IVE) visualization. Through operating a mobile robot in the virtual environment (DVE/IVE) by a skilled human operator, training data are readily obtained and used to train the neural fuzzy system.

In this paper, position and orientation estimation with high accuracy based on GPS and encoders for a four-wheel-steering vehicle (4WS) mobile robot is addressed. An architecture of position and orientation estimation is proposed, which consists of two Extended Kalman Filters and a processing unit of Runga-Kutta-based dead reckoning. The first EKF fuses data from six encoders to estimate the velocity of vehicle and sideslip angle. The second EKF is applied to the estimation of position and orientation based on the measurement from precise GPS data and output from first EKF. To obtain better accuracy of estimation, an arbitrator is designed to switch between EKF2 and dead reckoning. The results and analysis of experiments are presented to show the effectiveness of the proposed approach.

This paper presents a full-state tracking control scheme for a car-like vehicle. The control law is developed based on a dynamic model of the vehicle together with the definition of a virtual reference point off the vehicle. The proposed controller can track another vehicle moving in front of or track one moving behind it. Simulations show the effectiveness of the proposed controller. The influence of pre-selected parameters on tracking performance is also examined.

In this paper, the design procedure of multi-channel anticipatory learning control is demonstrated via a robot joint example. Technical details such as design of channels, design of filters, are illustrated. One more channel substantially widens the learnable frequency range. Comparisons of the multi-channel learning control with the conventional single channel learning control verify the effectiveness of the learning controller of the additional channel.

This paper develops a trajectory planning algorithm for a four-wheel-steering vehicle based on vehicle kinematics. The flexibility offered by the steering is utilized fully in the trajectory planning. A two-part trajectory planning algorithm consists of the steering planning and velocity planning. Limits of the vehicle mechanism and drive torque are taken into account. Simulation results are presented to illustrate the application of the proposed algorithm.

This paper presents a novel Behavior Fusion method for the navigation of Autonomous Mobile Vehicle in unknown environments. The proposed navigator consists of an Obstacle Avoider (OA), a Goal Seeker (GS) and a Navigation Supervisor (NS). The fuzzy actions inferred by the OA and the GS are weighted by the NS using the local and global environmental information and fused through fuzzy set operation to produce a command action, from which the final crisp action is determined by defuzzification. Simulation shows that the navigator is able to perform successful navigation task in various unknown environments, and it has smooth action and exceptionally good robustness to sensor noise.

In this paper, an iterative learning impedance control problem for robotic manipulators is formulated and solved. A target impedance is specified and a learning controller is designed such that the system follows the desired response specified by the target model as the actions are repeated. A design method for analyzing the convergence of the learning impedance system is developed. A sufficient condition for guaranteeing the convergence of the system is also derived. The proposed learning impedance control scheme is implemented on an industrial selective compliance assembly robot arm (SCARA) robot, SEIKO TT3000. Experimental results verify the theory and confirm the effectiveness of the learning impedance controller.

Most researches on learning control of constrained robots have been focused on the problem of hybrid position/force control where the learning controllers are designed to track the desired motion and force trajectories. The learning impedance control of robotic manipulators, however, has not been developed so far. In this paper, a learning impedance control problem for robotic manipulators is formulated and solved. A target impedance is specified and a learning controller is designed such that the system follows the desired response specified by the target model as the actions are repeated. Sufficient conditions for guaranteeing the convergence of the system are derived. Simulation results of a cylindrical robot are presented to illustrate the performances of the proposed learning impedance controller.

The kinematic redundancy enables a robot to change its joint configuration without changing the pose of the end-effector or the object. In addition to the requirement of path tracking in Cartesian space, many tasks require a manipulator to achieve more than one performance criteria or take into account some manipulation constraints. In this paper, a fuzzy logic approach is proposed for the desired joint path generation for redundant manipulators with consideration of multiple criteria. Performance criteria are fuzzified and their relative importance measures are introduced. The desired joint path is determined based on the relative importance of various criteria and the satisfaction measures of each candidate joint angle to all criteria by a fuzzy logic multiple criteria decision making process. Simulation results are presented to demonstrate that the proposed method can generate a path which can satisfy two performance criteria.>

A design procedure for simultaneous position and force control is developed, using Lyapunov's direct method, for manipulators in contact with a rigid environment that can be described by holonomic constraints. Many manipulators that interact with their environment require taking into account the effects of these constraints in the control design. The forces of constraint play a critical role in constrained motion and are, along with displacements and velocities, to be regulated at specified values. Lyapunov's direct method is used to develop a class of position and force feedback controllers. The conditions for gain selection demonstrate the importance of the constraints. Force feedback has been shown not to be mandatory for closed-loop stabilization, but it is useful in improving certain closed-loop robustness properties.>