Humanoid locomotion remains a fundamental challenge in robotics due to the high degrees of freedom, strong coupling between centroidal and joint-level dynamics, underactuation during walking phases, as well as pervasive model uncertainties and external disturbances. To address these challenges, this paper proposes an advanced whole-body planning and control framework that enables compliant posture regulation and stable locomotion through optimized centroidal motion and force planning. In the proposed framework, the full-body humanoid dynamics are first decoupled into centroidal momentum dynamics and joint-level dynamics. Based on this decomposition, a whole-body planning module formulates an optimization problem to generate dynamically consistent center-of-mass (CoM) trajectories and desired ground reaction forces (GRFs), which serve as reference inputs for posture adjustment and balance stabilization during walking. These planned centroidal quantities are then tracked by a whole-body control layer, where the desired GRFs are mapped into joint torques through a torque distribution scheme that preserves compliance and accommodates contact constraints. To cope with model uncertainties and unmodeled dynamics, an adaptive robust decentralized repetitive learning control strategy is integrated into the tracking control framework. By exploiting task repetitiveness, the proposed learning scheme iteratively refines control inputs, effectively compensating for system uncertainties without highly accurate models. A Lyapunov-based stability analysis demonstrates that the closed-loop tracking errors are uniformly ultimately bounded under bounded uncertainties. Extensive experimental results validate the effectiveness of the proposed approach, demonstrating stable posture regulation, robust walking performance, and strong disturbance rejection capabilities.

Developing non-visual sensing and intelligent recognition technologies is crucial for enhancing the manipulation performance of robots in dim or obstructed environments. Although precise object recognition has been extensively researched for rigid manipulators, the adoption of these techniques in soft robotic systems has been limited by the high modulus or low sensitivity of existing sensors (gauge factor/Young’s modulus$<$10 kPa$^{-1}$). Meanwhile, these systems necessitate advanced artificial intelligence(AI) algorithms to effectively process multiple sensing data. In this study, we utilize newly developed soft sensors and AI algorithm to establish a biomimetic perceptual soft gripper system capable of sensing and generating category object information during the grasping task. A strain/pressure bimodal sensor, mimicking the exceptional softness (Young’s modulus$<$10 kPa) and high sensitivity (gauge factor$>$2000) of human skin, has been developed and seamlessly integrated into a three-finger soft robotic gripper. A Swin Transformer network was developed to learn rules from bimodal data acquired from sensors and generate category information of the grasping objects. The perceptual gripper system exhibited superior recognition accuracy compared to previously reported systems, achieving an impressive 94.9% accuracy in categorizing 18 objects of varying shapes and sizes. We believe this advancement unlocks soft machines’ potential for automated applications.Note to Practitioners—Non-visual sensing and recognition techniques for soft robotic grippers are valuable for operation in dimly lit or obstructed environments. Traditional rigid sensor have limitations when applied to soft robotic systems due to their high modulus. This research developed novel ultra-soft and high sensitive strain/pressure bimodal sensors integrated into a three-fingered soft robotic gripper. Meanwhile, an AI algorithm applicable to the soft gripper system was developed to process the sensing data and generate the corresponding object category information when grasping the object. With the aid of the developed soft sensor and AI algorithm, the perceptual soft gripper can achieve high-precision object recognition, giving it great potential for automating tasks such as item picking, assembly, and handling in energy-efficient warehouses and logistics centers. The developed soft sensor and AI algorithm co-pilot object recognition strategy can contribute to the advancement of soft machine automation capability technology.

Human-gaze-target prediction aims to predict the target point or object that humans are looking at in images. However, existing methods predominantly rely on vision-only features, which often struggle to capture the semantic context of small or occluded objects and lack explicit priors for precise head direction regression, leading to slow convergence and suboptimal performance. Therefore, we introduce VL-HTR, a novel vision-language learning method for human-target representation, which integrates multimodal knowledge from vision-language models (VLMs) to construct robust human-target relationships. Unlike traditional approaches, extracting multimodal features via pretrained VLMs enhances the model's grasp of human-target knowledge through the learnable target class and direction context. Then, a language-guided query alignment (LQA) module is introduced to improve the semantic-aware object representation capability through vision-language query alignment. Finally, to accelerate the gaze point regression learning process, we design a language-guided direction prediction (LDP) module to introduce multimodal human gaze direction priors, thereby facilitating the human-target relationship construction. Extensive validations across two distinct tasks, i.e., gaze object prediction (GOP) and gaze target estimation, involving five challenging benchmarks, demonstrating that VL-HTR achieves superior performance and much faster training convergence.

Wearable flexible lower-limb exoskeletons, commonly referred to as soft exosuits, have emerged as a promising technology for enhancing mobility and assisting gait rehabilitation. However, most conventional designs provide assistance failing to adapt to variations in walking cadence or gait phase subdivision. This study presents a planning and control frame work for a cable-driven exosuit that delivers personalised ankle assistance in the triadic interaction encompassing the human, the robot, and the environment. The framework enables automatic gait phase detection and adaptive assistance, dynamically responding to changes in gait, environmental conditions, and metabolic demands. Inspired by the periodic nature of human walking, a gait cycle is divided into eight phases using a designed transformer classification model that processes foot force and inertial measurement unit (IMU) data. Heart rate is incorporated to provide feedback on metabolic changes. A human-in-the-loop control strategy based on an event-triggered mechanism utilizing fine gait classification and heart rate is proposed to achieve ankle joint assistance adaptation. When the deviations in gait phases or metabolic conditions exceed thresholds, the ankle joint assistive trajectory is replanned online via optimization with safety constraints. An adaptive fuzzy controller ensures stable tracking under uncertain dynamics and external disturbances. The stability of the control system is analytically verified using Lyapunov theory, and experimental results demonstrate the effectiveness of the proposed approach.

Visual impairments significantly affect individuals’ ability to perform essential tasks such as communication, object search, and navigation. Traditional wearable augmented vision devices rely on modular designs that separate functions like perception and path planning, leading to cumulative errors and inefficiencies in real-world applications. To address these challenges, we propose an end-to-end multimodal large model framework, UniANS, specifically designed for wearable augmented vision devices. UniANS integrates visual perception, speech interaction, and path planning into a unified framework. Such integration improves task coordination, reduces error propagation, and enhances overall performance. We also propose a prompt design strategy with a mixture of cluster-conditional low-rank adaptation experts architectures and dual-branch encoders, combined with advanced preprocessing techniques for visual and speech modules. The framework has been validated through ablation studies, showing superior performance in accuracy and task effectiveness compared to existing methods. We further showcase its capabilities in addressing challenges related to communication, object search, and indoor navigation tasks. The design of UniANS enhances mobility and quality of life for visually impaired individuals.

This paper proposes a novel navigation and optimization approach for mobile robots operating in intelligent manufacturing (IM) scenarios with unknown obstacles. First, we introduce a set of evaluation functions that simultaneously consider multiple metrics, including speed, security, sampling step, and obstacle avoidance ability, for scenarios with unknown dynamic obstacles. We then adopt a knee-guided multi-objective evolutionary algorithm capable of balancing parameter size and performance to optimize these conflicting objectives. Finally, we present a local navigation framework that integrates navigation and multi-objective optimization to enable obstacle avoidance in unstructured environments. The proposed algorithm is validated through simulations and practical scenarios, demonstrating its effectiveness in both static and dynamic scenarios.Note to Practitioners—The Human-cyber-physical System (HCPS) is the basic principle of the new generation of intelligent manufacturing (IM). It presents new requirements for the production method, and robot operate completely autonomously is expected to be the future solution for some complex tasks. However, traditional navigation methods often fail in these scenarios due to environmental changes and restricted workspaces. To address this challenge, this paper proposes a multi-objective optimization-based local navigation approach (MONA) that transforms the local navigation problem into a multi-objective optimization problem. The approach uses the knee-guided multi-objective evolutionary algorithm to optimize the control input for the mobile robot, ensuring safe and fast navigation in complex environments. The proposed approach has been validated through simulation and real scenario experiments, demonstrating its effectiveness.

Collaborative robots are predicted to interact physically with humans in human-cyber-physical systems (HCPSs). Robotic hands are able to record force and temperature simultaneously through soft and conformable sensors applied over finger surfaces and conforming to the complex curved geometries of automatic machines with tactile perception. Here, skin-inspired triple tactile (SITT) sensors are integrated into robotic fingers to enable precise bimanual grasping. The SITT sensor has skin-inspired multilayer microstructures, which integrate three sensors, namely, an interdigital electrode sensor, a flexible force sensor, and a temperature sensor. The SITT sensor can simultaneously or independently measure a material’s dielectric property, tactile force and temperature. An HCPS based on SITT sensors, a data acquisition board, bimanual robotic hands, and human-robot interaction software is developed for efficient bimanual manipulation. Through 3C assembly experiments, the designed HCPS is demonstrated to execute complex tasks. This research presents a novel methodology for constructing robust tactile sensors for robotic fingers in a bimanual manipulation system, and it presents significant potential across various aspects of intelligent production, including sensitive object handling, adaptive manipulation, and interactive robotics applications.Note to Practitioners—This work aims to overcome the challenge of skin-inspired triple tactile (SITT) sensors developed for robotic fingers in a human-robot collaborative assembly scenario, which can also be used in many other similar human-robot/machine collaborations (e.g., wearable prosthetics with tactile feedback) with practical value. Its capability to accurately measure the touched objects’ information (e.g., material dielectric property, force, temperature) is crucial for the bimanual robot to successfully interact with human operators. HCPS is valuable for its potential to achieve complicated interactions among humans, cyber systems, and physical resources. Collaborative robots with tactile sensors are expected to interact physically with humans in the HCPS. In this paper, we report SITT sensors integrated on robotic fingers to enable precise bimanual grasping. They can simultaneously or independently measure material dielectric properties and tactile forces and record temperature. By combining tactile perception information and a related control method, our smart bimanual robotic hands with the developed HCPS demonstrates the capability to execute 3C assembly in intelligent manufacturing. In the future, additional application scenarios will be designed for the HCPS platform, and some traditional tasks (such as a single arm for collaborative assembly) will be replaced by our hardware and software systems.

The enhancement of underwater images has emerged as a significant technological challenge in advancing marine research and exploration tasks. Due to the scattering of suspended particles and absorption of light in underwater environments, underwater images tend to present blurriness and predominantly color distortion. In this study, we propose a novel approach utilizing denoising diffusion models to improve underwater degraded images. After training the noise estimation network of the denoising diffusion models, we accelerate the deterministic sampling process with denoising diffusion implicit models. We also propose a patch-based method by implementing average sampling between overlapping image patches at each sampling step, enabling the generation of images at arbitrary resolution while preserving their natural appearance and details. Through benchmark experiments, we illustrate that our method outperforms or closely approaches state-of-the-art techniques in terms of effectiveness and performance. We demonstrate that our approach reduces the interference of underwater environments with the semantic information of the images by salient object detection experiments.

Underwater areas have harsh environments with poor light, limited visibility, and high levels of noise. Humans have a weak perception of position, surroundings, and exterior information when staying underwater, which makes it difficult for humans to carry out complex underwater tasks, such as rescue, observation, and construction. Wearable devices have shown good results in enhancing human sensory function on land, thus they could potentially play a role in enhancing human underwater perception ability. This perspective aims to analyze the state-of-the-art of underwater wearable systems for human perception enhancement. This work discusses the core technology and challenges of human underwater perceptual enhancement, including wearable underwater navigation, underwater environment reconstruction, and underwater sensorial information delivery. Future research could focus on designing waterproof flexible human-machine interfaces for sensing and feedback, exploiting advanced sensors and fusion algorithms for wearable underwater positioning, and studying multimodal information interaction strategies of wearable systems.

For leg prosthesis user, the soft tissue and skin under the stump of are not accustomed to weight bearing, excessive continuous contact pressure can lead to the risk of degenerative tissue ulceration. This article presents a novel human-robot collaborative control scheme that achieves control weight self-adjustment for robotic prostheses to minimize interaction torque. To establish the human-robot interaction relationship, we regard the contact pressure between human residual limb and the prosthetic receiving cavity as the interaction force. We aim at reducing the interaction force under the premise of minimally changing the original motion trajectory of the robotic prosthesis. The control scheme mainly includes trajectory optimization based on a dual-agent game control scheme under a cooperative relationship, and a fuzzy logic system for improving the control accuracy of trajectory tracking of robotic prostheses with unknown dynamic parameters. Experiments were carried out on two amputee participants to verify the proposed human-robot interactive control scheme in a robotic prosthesis. The results show that the interaction torque could be reduced while maintaining minimal trajectory tracking error. The proposed control scheme could potentially facilitate the dexterous manipulation of leg prostheses, thus benefiting amputees.

Gaze object prediction aims to predict the location and category of the object that is watched by a human. Previous gaze object prediction works use CNN-based object detectors to predict the object's location. However, we find that Transformer-based object detectors can predict more accurate object location for dense objects in retail scenarios. Moreover, the long-distance modeling capability of the Transformer can help to build relationships between the human head and the gaze object, which is important for the GOP task. To this end, this paper introduces Transformer into the fields of gaze object prediction and proposes an end-to-end Transformer-based gaze object prediction method named TransGOP. Specifically, TransGOP uses an off-the-shelf Transformer-based object detector to detect the location of objects and designs a Transformer-based gaze autoencoder in the gaze regressor to establish long-distance gaze relationships. Moreover, to improve gaze heatmap regression, we propose an object-to-gaze cross-attention mechanism to let the queries of the gaze autoencoder learn the global-memory position knowledge from the object detector. Finally, to make the whole framework end-to-end trained, we propose a Gaze Box loss to jointly optimize the object detector and gaze regressor by enhancing the gaze heatmap energy in the box of the gaze object. Extensive experiments on the GOO-Synth and GOO-Real datasets demonstrate that our TransGOP achieves state-of-the-art performance on all tracks, i.e., object detection, gaze estimation, and gaze object prediction. Our code will be available at https://github.com/chenxi-Guo/TransGOP.git.

This article presents U 2PNet, a novel unsupervised underwater image restoration network using polarization for improving signal-to-noise ratio and image quality in underwater imaging environments. Traditional methods for underwater image restoration using polarization require specific cues or pairs of underwater polarization datasets, which limit their practical applications. Our proposed method requires only one mosaicked polarized image of the scene and does not require datasets for pretraining or specific cues. We design two subnetworks (T-net and B textsubscript ∞ -net) to accurately estimate the transmission map and background light, and unique nonreference loss functions to ensure effective restoration. Our experiments are based on an indoor polarization simulated dataset and a real polarization image dataset constructed from our underwater robotic platform equipped with polarization cameras. Experiment results demonstrate that our proposed method achieves state-of-the-art performance on both simulated and real underwater polarization images. The code and datasets will be available at https://github.com/polwork/U-2Pnet.

Surface electromyography (sEMG) signal has been used in upper limb rehabilitation robots (ULRR). However, existing ULRR based on myoelectric controllers suffers from limited generalization ability in estimating three-dimensional (3-D) motion intention. This article proposes a muscle-synergy-inspired approach to enhance the generalization ability of the myoelectric controller of a cable-driven ULRR. Low-dimensional commands are extracted from sEMG signals based on an EMG-to-muscle activation model and non-negative matrix factorization. The extracted commands are used to estimate the 3-D human force. Two different trajectory tracking tasks are selected to test the generalization ability. The system is trained based on training sets where participants perform one task. Then the system is tested using testing sets where participants perform the other task. Finally, the system is verified on real-time robotic control experiment. Results show that the proposed controller achieves better force estimating accuracy, better trajectory tracking accuracy, and lower interaction force than the myoelectric controller without considering muscle synergies, which means the proposed controller yields better generalization performance.

Navigating independently is a challenge for visually impaired vision due to the demand of obstacles avoiding, recognizing desired objects, and wayfinding in complicated environments. In this paper, we present an augmented wearable E-Glasses with a set of sensors, where an object detection neural network based on visual-auditory fusion method is employed to search desired targets, thus addressing navigation challenges and improving the mobility and independence of the visually impaired. We demonstrate advanced navigation capabilities: indoor wayfinding, recognizing and steering the users to desired goals, and a sequence of indoor challenges. The fusion network adopts a feature-level fusion strategy, which is capable to align two modalities automatically and effectively integrate visual features and audio features. Across all experiments, the developed fusion algorithm has a 94.67% success rate. The wearable E-Glasses supply a platform that helps to improve the mobility and quality of life of people with impaired vision.

Soft exo-suit could facilitate walking assistance activities (such as level walking, upslope, and downslope) for unimpaired individuals. In this article, a novel human-in-the-loop adaptive control scheme is presented for a soft exo-suit, which provides ankle plantarflexion assistance with unknown human-exosuit dynamic model parameters. First, the human-exosuit coupled dynamic model is formulated to express the mathematical relationship between the exo-suit actuation system and the human ankle joint. Then, a gait detection approach, including plantarflexion assistance timing and planning, is proposed. Inspired by the control strategy that is used by the human central nervous system (CNS) to handle interaction tasks, a human-in-the-loop adaptive controller is proposed to adapt the unknown exo-suit actuator dynamics and human ankle impedance. The proposed controller can emulate human CNS behaviors which adapt feedforward force and environment impedance in interaction tasks. The resulting adaptation of actuator dynamics and ankle impedance is demonstrated with five unimpaired subjects and implemented on a developed soft exo-suit. The human-like adaptivity is performed by the exo-suit in several human walking speeds and illustrates the promising potential of the novel controller.

The integration and interaction of proprioception and exteroception in the human multisensory network facilitate high-level cognitive functionalities, such as cross-modal integration, recognition, and imagination for accurate evaluation and comprehensive understanding of the multimodal world. In this article, we propose a novel cross-modal integration framework for the upper limb prosthesis based on type-2 fuzzy logic system (FLS), which can facilitate the high-level cognitive and dexterous manipulation of the prosthesis by combing human's surface electromyography (sEMG) with the computer vision. First, a transfer learning approach is proposed to improve the decoding of human's intent and enhance the effectiveness of skill transition. Then the sEMG signals and image information are integrated to jointly determine the grasp posture of the bionic hand based on fuzzy decision strategy. Fusing multisensory data and using cross-modal integration, the system is capable of crossmodally recognizing multimodal information. In order to realize the prosthesis automatically reaching the target position under the guidance of computer vision, an interval type-2 fuzzy logic controller considering uncertain dynamic parameters and disturbance is designed. Experiments are performed in some typical 3C assembly scenarios, and results show that our proposed strategy can obviously improve the accuracy of grasp posture selection and trajectory tracking effect, which brings more potential job opportunities with hope to amputees, and provides a promising approach toward robotic sensing and perception.

Many previous studies of soft exosuits improved human locomotion performance. However, there is no example to control a soft exosuit using human ankle impedance adaption in assistance tasks compliantly. In this article, the human–environment interaction information is exploited into the exosuit control. A novel fuzzy-based optimization and control method of soft exosuit is proposed to provide plantarflexion assistance for human walking by changing the human–robot interaction. In particular, a fuzzy neurodynamics optimization is developed to learn the unknown human ankle impedance parameters automatically. A fuzzy approximation technique is applied to improve the control performance of the exosuit when a human is walking with unknown human–robot interaction model parameters. This control scheme guarantees that the human–robot dynamics follows a target human ankle impedance model to obtain the compliant interaction performance. Experiments on different participants verify the effectiveness of the control scheme. Results show that a compliant human–robot interaction is achieved by learning the human–environment interaction parameters, i.e., the human ankle parameters. It indicates that our proposed method can facilitate exosuit control to achieve compliant robot–human–environment interaction.