In all types of surface code decoders, fully neural network-based high-level decoders offer decoding thresholds that surpass decoder-Minimum Weight Perfect Matching (MWPM), and exhibit strong scalability, making them one of the ideal solutions for addressing surface code challenges. However, current fully neural network-based high-level decoders can only operate serially and do not meet the current latency requirements (below 440 ns). To address these challenges, we first propose a parallel fully feedforward neural network (FFNN) high-level surface code decoder, and comprehensively measure its decoding performance on a computing-in-memory (CIM) hardware simulation platform. With the currently available hardware specifications, our work achieves a decoding threshold of 14.22%, and achieves high pseudo-thresholds of 10.4%, 11.3%, 12%, and 11.6% with decoding latencies of 197.03 ns, 234.87 ns, 243.73 ns, and 251.65 ns for distances of 3, 5, 7 and 9, respectively.

This paper presents a novel method for creating a hybrid 2.5D semi-semantic map, merging a 2D geometric map with a sparse 3D object map, specifically designed for expansive indoor environments. The primary motivation behind this method is to tackle the issue of temporal and viewpoint discontinuity in RGB-D observations of individual objects within such environments. Notably, current RGB-D SLAM research mainly focuses on small-scale scenarios, often overlooking this specific issue. To address this challenge, our approach proposes to represent objects with a set of object-specific keyframes, and optimizes the spatial relationships between these keyframes in a deferred offline mode. Another objective is to alleviate the accumulation of trajectory drift, which can adversely impact object association/reidentification in expansive environments. To achieve this, we leverage a 2D pose graph SLAM module that indirectly provides initial poses of the RGB-D sensor, thereby facilitating the construction of the sparse 3D object map. Addressing the challenge arising from the dimensional disparity between the two sub-maps (2D vs 3D), we implement a joint optimization strategy to refine the hybrid map, ensuring accuracy compatibility between them. The effectiveness of our proposed method is validated through experiments conducted in real-world environments, and a time efficiency analysis demonstrates the potential of our algorithm to operate in real-time. Note to Practitioners—The motivation of this paper is to create 3D object maps for expansive indoor environments. The challenge arises from the fact that most existing RGB-D SLAM algorithms are designed for small-scale indoor scenarios, lacking suitability for expansive environments with empty areas. In real expansive environments, the short-range capacity of RGB-D sensors increases the risk of failures in continuously observing objects in empty areas, potentially causing the robot to lose its way. This results in two consequences: 1) discontinuity in the viewpoints of individual objects, challenging the online reconstruction of 3D objects; 2) accumulations of trajectory drift during the robot’s loss complicate the reidentification of an object upon its reappearance. Our approach addresses these issues by concurrently generating a 2D geometric map, effectively mitigating trajectory drift. Additionally, we employ a set of object-specific keyframes to represent object reconstruction, efficiently handling viewpoint discontinuity in object observations. Experimental results demonstrate the applicability of our algorithm to real-world environments beyond room-scale scenarios, closely aligning with the challenges encountered in practical applications. The analysis also indicates the potential for real-time operation, rendering our SLAM algorithm advantageous for practical deployment in robotic systems. An additional benefit is that, apart from the 3D object map, our method creates a 2D geometric map, supporting most basic navigation tasks for industrial robots. In future research, we will further improve the time-consuming process and enhance the efficiency of the algorithm.

The dual-arm space robot usually forms a closed-chain constraint system with a target through collaborative operations when performing tasks. Most previous related research has focused on the performance of open-chain robots themselves. Studying the manipulation performance of a closed-chain robot system is of great significance. This article proposes a reconfigurable space robot (RSR) system for on-orbit servicing. A graph theory based framework for the automatic generation of reconfigurable robot models is proposed to address the characteristics of variable topology structures. Meanwhile, a new index – the closed-chain inertia matching index is proposed to evaluate its configuration effectively. Compared with traditional dynamic manipulability ellipsoid (DME) and manipulating force ellipsoid (MFE), the effectiveness of the proposed closed-chain inertia matching ellipsoid (IME) is verified. Compared with the DME and MFE, the IME effectively considers the influence of load and can effectively express the dynamic torque force/torque transmission efficiency from the joint actuator to the load in a closed-chain system. The IME can not only be used to determine the optimal joint configuration under a specific closed-chain configuration, but also to determine the optimal nonisomorphic configuration of a reconfigurable robot. Finally, the results of configuration optimization of the closed-chain dual-arm reconfigurable space robot are given.Note to Practitioners—This paper aims to study the determination of the optimal configuration for the RSR during task execution. Due to its various configurations, the RSR has a complex modeling process. This article proposes a modeling framework for automatically generating RSR models that can effectively achieve automatic modeling of various configurations. In addition, most of the previous related research focused on the performance of open-chain robots themselves, ignoring the impact of load. This article proposes a new closed-chain system performance indicator to evaluate the robot configuration. The proposed evaluation indicator effectively considers the influence of load, which is of practical significance to improve the efficiency of task execution. The effectiveness of the proposed index in determining the optimal configuration is demonstrated by simulation verification. In the future, we will focus on multi-arm collaborative operations under a specific configuration.

Interaction stability is an important concern in variable admittance control, and it is a prerequisite for achieving the desired compliant interaction with humans or uncertain environments. Different variable admittance controllers with essential stability analyses are required for multiple interaction scenarios in a complex task, where stability analysis is the major difficulty. In this article, we propose a unified stability-guaranteed variable admittance control architecture to decrease the control system complexity, in which we represent the admittance control model as a port-Hamiltonian (pH) system and design an energy tank with the optimization theory. The advantages of the proposed architecture are reflected in several aspects: 1) compatibility with common strategies; 2) perturbation resistance; and 3) passivity as a natural property. As the dissipated energy from the pH system is injected into the energy tank while the energy in the tank is used to support the interaction and perturbation resistance, the best passive approximation of the desired behaviors is generated, which can avoid the control signal failure (control signal failure denotes that the unexpected input$u=0$occurs due to a lack of energy in the tank). Three different groups of experiments are conducted to verify the feasibility, perturbation resistance, and practicality of the proposed architecture.

This paper addresses a long-standing yet well-documented open problem on trajectory tracking control of manipulators, which is simultaneously affected by unknown kinematics and uncertain dynamics. A theoretical framework for implementing exponential tracking control is established by unifying two novel controllers, i.e., the Jacobian matrix adaption (JMA) controller and the observer-based estimation law, into an integrated control system. The proposed JMA controller converts internal, implicit and immeasurable model information to external, explicit and measurable input-output information for adaptive learning of unknown kinematics. The proposed observer-based estimation law can guarantee the global exponential stability of the estimation error for accurately estimating uncertain dynamics. In addition, the inherent measurement noise, hard nonlinearity, and limited sampling period lead to chattering phenomenon and accumulated errors occur during convergence process. Hence, an improved simple model-free adaptive sliding mode control (ASMC) scheme is proposed to compensate these limitations, which has fast adaptability and powerful tracking and chattering suppression capabilities. It is theoretically proved that the integrated control system is globally exponentially stable. Simulation, experiments and comparison verify the convergence performance of the proposed integrated control system.Note to Practitioners—Despite the enormous advantages provided by advanced robotic manipulators, developing effective tracking control schemes remains a challenging problem Unfortunately, however, the assumption of precise kinematic and dynamic parameters during tracking control is practically unrealistic due to the absence of precise parameters and measurements of the interaction between robotic manipulators and external environment. This uncertainty will reduce the control performance of manipulators, such as accuracy and repeatability. It is of practical significance to further solve the tracking control of manipulators while considering both the uncertain kinematics and the uncertain dynamics. Therefore, this paper addresses the tracking control problem of manipulators, which is simultaneously affected by unknown kinematics and uncertain dynamics. By unifying the two novel controllers into an integrated control system, a theoretical framework for implementing exponential tracking control is established. In addition, an improved scheme is proposed to compensate chattering phenomenon and accumulated errors during the convergence process. The superior convergence performance of the integrated control system are verified by the simulation and experiments.

The distributed resilient tracking problem for multiagent systems (MASs) is investigated in the presence of actuator/sensor faults over directed topology. Both actuator fault and sensor fault are taken into account. Meanwhile, using the local information, the fault compensators are introduced. Then, based on the fuzzy-logic systems (FLSs) and modification technique of adaptive law, a novel distributed adaptive resilient control protocol is developed, which can compensate the effect of faults on the actuator and sensor. It turns out that all signals of MASs are bounded, while the tracking errors enter an adjustable bounded region around the origin. Toward the end, two simulations are provided to validate the effectiveness of the theoretical results.

Active shape control for an antenna reflector is a significant procedure used to compensate for the impacts of a complicated space environment. In this article, a physics-guided distributed model predictive control (DMPC) framework for reflector shape control with input saturation is proposed. First, guided by the actual physical characteristics, an overall structural system is decomposed into multilevel subsystems with the help of a so-called substructuring technique. For each subsystem, a prediction model with information interaction is discretized by an explicit Newmark- β method. Then, to improve the system-wide control performance, a coordinator among all the subsystems is designed in an iterative fashion. The input saturation constraints are addressed by transforming the original problem into a linear complementarity problem (LCP). Finally, by solving the LCP, the input trajectory can be obtained. The performance of the proposed DMPC algorithm is validated through an experiment on the shape control of an antenna reflector structure.

Active disturbance rejection control (ADRC) is an efficient control technique to accommodate both internal uncertainties and external disturbances. In the typical ADRC framework, however, the design philosophy is to "force" the system dynamics into a double-integral form by an extended state observer (ESO) and then the controller is designed. Especially, the systems' physical structure has been neglected in such a design paradigm. In this article, a new ADRC framework is proposed by incorporating at a fundamental level the physical structure of the Euler-Lagrange (EL) systems. In particular, the differential feedback gain can be selected considerably small or even 0, due to the effective exploitation of the system's internal damping. The design principle stems from an analysis of the energy balance of EL systems, yielding a physically interpretable design. Moreover, the exploitation of the system's internal damping is thoroughly discussed, which is of practical significance for applications of the proposed design. Besides, a sliding-mode ESO is designed to improve the estimation performance over traditional linear ESO. Finally, the proposed control framework is illustrated through tracking control of an omnidirectional mobile robot. Extensive experimental tests are conducted to verify the proposed design as well as the discussions.

Hand detection is a crucial technology for space human-robot interaction (SHRI), and the awareness of hand identities is particularly critical. However, most advanced works have three limitations: 1) the low detection accuracy of small-size objects; 2) insufficient temporal feature modeling between frames in videos; and 3) the inability of real-time detection. In the article, a temporal detector (called TA-RSSD) is proposed based on the SSD and spatiotemporal long short-term memory (ST-LSTM) for real-time detection in SHRI applications. Next, based on the online tubelet analysis, a real-time identity-awareness module is designed for multiple hand object identification. Several notable properties are described as follows: 1) the hybrid structure of the Resnet-101 and the SSD improves the detection accuracy of small objects; 2) three-level feature pyramidal structure retains rich semantic information without losing detailed information; 3) a group of the redesigned temporal attentional LSTM (TA-LSTM) is utilized for three-level feature map modeling, which effectively achieves background suppression and scale suppression; 4) low-level attention maps are used to eliminate in-class similarity between hand objects, which improves the accuracy of identity awareness; and 5) a novel association training scheme enhances the temporal coherence between frames. The proposed model is evaluated on the SHRI-VID dataset (collected according to the task requirements), the AU-AIR dataset, and the ImageNet-VID benchmark. Extensive ablation studies and comparisons on detection and identity-awareness capacities show the superiority of the proposed model. Finally, a set of actual testing is conducted on a space robot, and the results show that the proposed model achieves a real-time speed and high accuracy.

Gaze tracking is widely used in driver safety driving, visual impairment detection, virtual reality, human robot interaction, and reading process tracking. However, varying illumination, various head poses, different distances between human and cameras, occlusion of hair or glasses, and low-quality images pose huge challenges to accurate gaze tracking. In this article, based on binocular feature fusion and convolution neural network, a novel method of gaze tracking is proposed, in which local binocular spatial attention mechanism (LBSAM) and global binocular spatial attention mechanism (GBSAM) are integrated into the network model to improve the accuracy. Furthermore, the proposed method is validated on the GazeCapture database. In addition, four groups of comparative experiments have been conducted: between binocular feature fusion model and binocular data fusion model; among the local binocular spatial attention model, the local binocular channel attention model, and the model without local binocular attention mechanism; between the model with GBSAM and that without GBSAM; and between the proposed method and other state-of-the-art approaches. The experimental results verify the advantages of binocular feature fusion, LBSAM and GBSAM, and the effectiveness of the proposed method.

Global temporal information and local semantic information are essential cues for high-performance online object detection in videos. However, despite their promising detection accuracy in most cases, most state-of-the-art approaches have following two limitations: invalid background/scale suppression and inadequate temporal information mining between frames. Many jobs currently focus on temporal information learning based on a single frame. In this article, we propose an attentional global–local information learning network; this is one of the first attempts to fully use both types of information between frames. Attention maps are creatively utilized to transfer temporal contexts between frames. This also effectively alleviates the adverse effects of scale changes. Furthermore, empowered by a detailed framework, a proposed detector effectively uses multilevel feature extraction. Given these contributions, the proposed detector achieves state-of-the-art performance on challenging benchmarks. Finally, practical experiments are conducted on a space human–robot interaction platform.

Abstract Hand gesture recognition plays an important role in human–robot interaction. The accuracy and reliability of hand gesture recognition are the keys to gesture‐based human–robot interaction tasks. To solve this problem, a method based on multimodal data fusion and multiscale parallel convolutional neural network (CNN) is proposed in this paper to improve the accuracy and reliability of hand gesture recognition. First of all, data fusion is conducted on the sEMG signal, the RGB image, and the depth image of hand gestures. Then, the fused images are generated to two different scale images by downsampling, which are respectively input into two subnetworks of the parallel CNN to obtain two hand gesture recognition results. After that, hand gesture recognition results of the parallel CNN are combined to obtain the final hand gesture recognition result. Finally, experiments are carried out on a self‐made database containing 10 common hand gestures, which verify the effectiveness and superiority of the proposed method for hand gesture recognition. In addition, the proposed method is applied to a seven‐degree‐of‐freedom bionic manipulator to achieve robotic manipulation with hand gestures.

This article presents a state-of-the-art survey on the robotic systems, sensors, actuators, and collaborative strategies for physical human-robot collaboration (pHRC). This article starts with an overview of some robotic systems with cutting-edge technologies (sensors and actuators) suitable for pHRC operations and the intelligent assist devices employed in pHRC. Sensors being among the essential components to establish communication between a human and a robotic system are surveyed. The sensor supplies the signal needed to drive the robotic actuators. The survey reveals that the design of new generation collaborative robots and other intelligent robotic systems has paved the way for sophisticated learning techniques and control algorithms to be deployed in pHRC. Furthermore, it revealed the relevant components needed to be considered for effective pHRC to be accomplished. Finally, a discussion of the major advances is made, some research directions, and future challenges are presented.

This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record.This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article.Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

In this paper, a station-keeping controller is designed for a multi-vectored (four thrusts) propeller airship by using path following approach. Firstly, the nonlinear models of the airship are introduced, including kinematics and dynamics, and the T-S fuzzy model is derived using linearization. Then, the station-keeping controller is designed, which consists of two modules: a guidance loop and a robust model predictive control(MPC). The guidance loop uses the guidance-based path following(GBPF) principle to calculate the desired yaw rate and velocities of the airship, while the robust MPC calculates a feedback control law to track the desired yaw rate and velocities by solving linear matrix inequalities (LMIs). Simulation results for the multi-vectored propeller airship are carried out to illustrate the effectiveness of the proposed method.

Hyper-redundant manipulator has more degrees of freedom than the least necessary to perform a given task, thus it has the features of overcoming conventional industrial robot's limitation such as improving its kinematics and dynamic performances. Crucial as it is, effective control of hyper-redundant manipulator is difficult for its redundancy. A novel shape control technique based on the concept of variable structure regular polygon and subsystem has been proposed. This technique, using variable structure regular polygon and neural networks models, is completely capable of solving the control problem of a planar hyper-redundant manipulator with any number of links following any desired path. With regular polygon side number's variety and its shape's transformation, the manipulator's configuration changes accordingly and moves actively to perform the task form point to point or following a path. Compared with other methods to our knowledge, this technique has such superiorities as fewer control parameters, higher precision and less computation. Simulation of a six-link modularized manipulator's inspection work in a bottle-like concave has demonstrated that this control technique is available and effective.