With the growing deployment of robotic applications, fault diagnosis at the individual robot level (single-robot fault diagnosis) is increasingly insufficient to meet stringent safety and reliability requirements. To address these challenges, multi-robot fault diagnosis has emerged as a promising approach, which enables robots to collaboratively share sensor data from diverse tasks. This collaboration helps mitigate data scarcity and supports the development of robust global models with improved generalization capabilities. However, multi-robot fault diagnosis presents several key challenges: (1) mitigating negative transfer caused by sensor heterogeneity across different robots; (2) effectively capturing spatial-temporal dependencies within the sensor data; and (3) designing an efficient distributed learning framework that preserves data privacy while enabling collaborative model training. In this paper, we propose a novel federated spatial-temporal fault learning (FSTFL) framework based on a fused adjacency matrix. The adjacency matrix is dynamically updated and initially constructed using domain knowledge to guide the learning process. Experimental evaluations on real-world datasets demonstrate the effectiveness of the proposed FSTFL framework in achieving accurate and privacy-preserving multi-robot fault diagnosis.

Goal-conditioned reinforcement learning (GCRL) is an effective method for multi-goal robotic manipulation tasks. Many studies based on hindsight experience replay (HER) and hindsight goal generation (HGG) have achieved the autonomous acquisition of robotic manipulation in reward-sparse environments and have greatly improved the learning efficiency of GCRL. However, these methods perform poorly in environments with obstacles and distant goals. In this paper, we propose hindsight goal diffusion and graph-based experience replay (HGD-GER) for complex robotic manipulation. First, obstacle-avoiding graphs in environments with obstacles are constructed, and the graph-based distance metric between different goals is established. Second, the proposed HGD approach utilizes the inherent denoising mechanism of diffusion models and obstacle-avoiding graph-based distance to generate exploration goals, thereby promoting the exploration of obstacle-bypassing areas. Then, GER module modifies the reward value of experience replay by graph-based distance, thereby avoiding the bias introduced by HER and improving the learning performance of the RL algorithm under sparse reward conditions. Finally, we conducted experiments on three robotic manipulation tasks with obstacles and distant goals, and the results show that the proposed HGD-GER achieves excellent learning performance. Additionally, the proposed method is deployed on the physical robot.

The imbalanced data limit the effectiveness of mobile robot fault diagnosis, while generating pseudo multi-sensor signals of mobile robot is an effective solution. However, existing generative methods often fail to balance the differences and correlations among channels across multi-sensor signals. To address these issues, a novel multimodal knowledge and multi-channel correlation generative adversarial network (MKMCGAN) is proposed to generate high-quality fault signals. Specifically, wavelet packet decomposition (WPD) are used to extract time-frequency features for each channel, then multi generator-discriminator pair strategy (MGDS) and a time-frequency analysis knowledge module (TFKM) are designed to bring higher similarity between the generated signal and the real signal. Subsequently, we construct a sensor data association graph and design a prior knowledge correlation module (PKM), which effectively consider the impact of inter-channel correlations on generated signals. Eventually, a novel multi-channel correlation generative adversarial network is proposed to extract time-frequency features and consider inter-channel correlations, which can generate high-quality fault signals. The effectiveness of MKMCGAN is thoroughly validated on datasets collected from a real robot fault diagnosis test bench. Experimental results indicate that MKMCGAN generates higher-quality signals compared to state-of-the-art methods.

Wheeled robots play a crucial role in Industrial Internet of Things (IIoT)-enabled manufacturing environments, and ensuring their reliable operation is essential for production efficiency and safety. However, their inherent complexity makes them prone to faults, while limited fault data results in imbalanced datasets, posing challenges for deep-learning-based fault diagnosis models. Existing denoising diffusion probabilistic model (DDPM)-based fault diagnosis methods tend to address single-channel scenarios, ignoring the graph relationships inherent in multichannel sensor data. Furthermore, current graph-based DDPM models also struggle in wheeled robot scenarios due to its multimodal nature. To address these challenges, we propose an enhanced DDPM-based method for imbalanced fault diagnosis of wheeled robots. Our method integrates graph operations into the noise prediction network of the DDPM framework, enabling efficient modeling of the complex spatial–temporal relations in multimodal graphical sequence data via a newly designed spatial–temporal graph U-Net (STGU-Net). Additionally, we present a dynamic degradation mechanism for the prior graph, simulating gradual structural changes during the diffusion process. Extensive experiments on a real-world wheeled robot platform demonstrate the superiority of the proposed model over state-of-the-art methods from multiple perspectives, showcasing its effectiveness in generating high-quality data and mitigating the imbalance problem in wheeled robot fault diagnosis.

Hindsight experience replay (HER) has greatly increased the possibility of using deep reinforcement learning (DRL) for robotic manipulation with sparse rewards. However, there are still concerns about low learning efficiency and poor performance due to its insufficient exploration ability and bias against the initial goal introduced by HER. In this article, to solve this problem, a multigoal robotic manipulation DRL method based on adaptive intrinsic curiosity and universal value network fitting (AIC-UVNF) is proposed to further improve the exploration ability and learning performance. Specifically, this method utilizes an improved curiosity mechanism to construct a joint intrinsic reward and adaptively adjust the proportion, which can enhance exploration ability and avoid excessive pursuit of novel states. In addition, a universal value network fitting approach is proposed to incorporate the initial goal into the value function fitting process, which employs the value of the initial goal to eliminate the bias of HER in the algorithm update. Combined with the off-policy soft actor-critic method, AIC-UVNF is verified on multigoal robotic manipulation tasks. The results show that the proposed method achieves better convergence efficiency and learning performance.

Robotic push-grasping in densely cluttered environments presents significant challenges due to unbalanced synergy and redundancy between both actions, leading to decreased grasp efficiency. In this paper, a novel double-critic deep reinforcement learning framework is introduced to optimize the push-grasping synergy for robotic manipulation in such environments, aiming to significantly reduce pre-grasping redundancy. This framework incorporates two distinct Deep Q-learning critics: Critic I selects the best course of actions based on the current state derived from visual interpretation, whereas Critic II evaluates the success rate of the current state-action pairing. To further refine the push-grasping synergy, an active double-step learning mechanism is introduced to optimize the training reward function for the pushing action, thereby enhancing its effectiveness through increased intentionality. Simulations show that the proposed framework outperforms contemporary counterparts, notably in grasping success rate and action efficiency. Finally, the framework’s generalization and adaptability are demonstrated by conducting real-world experiments using novel objects without the need of retraining.

3D Panoptic perception is essential for the understanding of real-world environment and plays an increasingly important role in the field of robotics. However, most existing methods heavily rely on image panoptic segmentation networks to acquire panoptic information of the environment, which is time-consuming and susceptible to interference. In this paper, we propose a novel and efficient panoptic mapping method based on multi-source information. Specifically, to improve the real-time performance of the system, we first apply lightweight object detection and semantic segmentation to extract 2D semantic and instance information from images. Second, a panoptic inference algorithm is designed that fully utilizes multi-source information, including geometry-based and learning-based information, to simultaneously reason about background and foreground objects in the environment. Finally, we take advantage of the scalability of the framework by introducing a multi-object tracking algorithm into the framework, thus providing the temporal information among consecutive frames to the data association module. Based on two popular datasets, extensive comparison experiments are conducted to illustrate the effectiveness of the proposed method. Experimental results show that compared with state-of-the-art panoptic mapping methods, the proposed method achieves superior performance in accuracy, real-timeness and stability. Furthermore, we also evaluate our method in real-world scenarios and CPU-only device to demonstrate the feasibility of its practical deployment.

Imbalanced data constitute a significant challenge in intelligent fault diagnosis cases because they can result in degraded diagnosis accuracy, which can in turn jeopardize the safety and reliability of industrial equipment. Generative adversarial networks (GANs) have been effectively used as common data augmentation methods to address this issue. However, their training process is difficult to perform and prone to mode collapse. Therefore, this article proposes a novel data augmentation method grounded in a diffusion model. The proposed method generates samples through physical simulation rather than adversarial training, which avoids the instability and mode collapse issues faced by GANs, leading to a more stable training process. Moreover, the proposed method utilizes the characteristics of gradual diffusion and random sampling to enhance the authenticity and diversity of sample generation. In addition, in terms of evaluating generation models, most existing works do not have a unified and thorough evaluation framework. Therefore, a comprehensive evaluation framework is proposed to effectively and comprehensively evaluate the performance of data augmentation models. Finally, the proposed method is evaluated using an open-source dataset and two actual testbeds to validate its effectiveness. The experimental results show that our method can generate higher quality and more diverse pseudosamples, and achieve superior fault diagnosis performance under imbalanced data. Specifically, our approach achieves diagnosis accuracies of 97.00%, 96.48%, and 98.30% on the three different datasets, all of which are superior to those of the compared state-of-the-art data augmentation algorithms.

The critical issue of wheeled robot fault diagnosis is to comprehensively evaluate its health condition using multisensor data, but traditional deep learning-based methods are hard to model the relationships among sensor measurements. Unlike these methods, the graph convolutional network (GCN), which uses the graph-structured data along with the association graph as input, is more efficient for relationship modeling. However, existing GCN-based fault diagnosis methods suffer from the following weaknesses: the association graphs are obtained according to the similarity of data samples or their features, which cannot guarantee accuracy; and these models are focused on spatial correlations and neglect temporal correlations. To address these problems, we propose to construct the association graph based on prior knowledge,i.e., a simplified mathematical model of the wheeled robot. Moreover, we develop a spatial-temporal difference graph convolutional network (STDGCN) for wheeled robot fault diagnosis. This network contains a difference layer that utilizes localized difference properties for feature enhancement, and the spatial-temporal graph convolutional modules are introduced to jointly capture the spatial-temporal correlations. To verify the effectiveness of the STDGCN for fault diagnosis, experiments are carried out, and the results show that the STDGCN achieves superior performance.

We present a real-time, high-accuracy, robust, tightly coupled visual-inertial odometry (VIO) algorithm, including monocular-inertial odometry and stereo-inertial odometry, and uses inertial measurement unit (IMU) pre-integration that is based on fourth-order Runge–Kutta (PK4) and IMU initialization based on maximum a posteriori (MAP) estimation. In particular, we used the multi-state constraint Kalman filter (MSCKF) to fuse vision and IMU measurement data for state estimation. In the optimization stage, we simultaneously considered and optimized all of the historical constraints, and performed multiple iterations to reduce the linearity errors. For further reducing the cumulative error and improving the relocation accuracy, we used a bag-of-words model for global optimization. To lower the computational cost and increase the real-time performance, we set keyframe insertion mechanism and introduced sliding window, and used a new form of Kalman gain that converts the Kalman gain in multi-state constraint Kalman filtering into the inverse of the state dimension. We validated the proposed method by using the EuRoC MAV dataset and KITTI dataset. We performed physics experiments in an outdoor environment with unstable light, to further validate the accuracy and robustness of our method.

To overcome the shortcomings of existing robot localization sensors, such as low accuracy and poor robustness, a high precision visual localization system based on infrared-reflective artificial markers is designed and illustrated in detail in this paper. First, the hardware system of the localization sensor is developed. Secondly, we design a novel kind of infrared-reflective artificial marker whose characteristics can be extracted by the acquisition and processing of the infrared image. In addition, a confidence calculation method for marker identification is proposed to obtain the probabilistic localization results. Finally, the autonomous localization of the robot is achieved by calculating the relative pose relation between the robot and the artificial marker based on the perspective-3-point (P3P) visual localization algorithm. Numerous experiments and practical applications show that the designed localization sensor system is immune to the interferences of the illumination and observation angle changes. The precision of the sensor is ±1.94 cm for position localization and ±1.64◦ for angle localization. Therefore, it satisfies perfectly the requirements of localization precision for an indoor mobile robot.