This article presents a model predictive control (MPC) method for humanoid dual-arm robots (DARs) to realize hybrid force-velocity control in the task space. The proposed method, named hybrid force-velocity MPC (HFV-MPC), enables simultaneous tracking of velocity and external force in the task space, as well as internal force regulation. First, a synchronous decoupling model of internal force, external force, and velocity is developed to address the limitations of conventional methods that decouple internal and external forces solely in the end-effector space and fail to meet the independent regulation requirements of external force and velocity in the task space. Furthermore, a generalized velocity model that integrates task-space velocity, external force, and internal force was established, and a HFV-MPC framework is constructed to enhance system robustness and achieve precise dual-arm force–velocity tracking. Subsequently, a comprehensive MPC cost function is designed, incorporating a dual-arm motion coordination coefficient to suppress undesired redundant joint motions and improve whole-body motion stability. In addition, a feedforward-linearized incremental system is derived, where the incremental model and force prediction jointly construct a generalized state-space model for MPC, to balance modeling accuracy and computational efficiency. Finally, the proposed method is validated in terms of effectiveness and robustness through both simulation and physical robotic experiments.

Grippers serve as essential end-effectors, pivotal for facilitating interaction between robots and their external environments. However, existing grippers emerge with primary issues, such as inadequate adaptability, limited grasping range, and narrow functionality. To address these issues, we present a novel under-actuated three-finger gripper driven by a single motor. This gripper accomplishes adaptive, wide-range, and multi-modal grasping through the mechanism retraction and reconfiguration, thereby offering significant advancement in this field. Firstly, inspired by the variable contact area observed in grasps for human hands, the fingers are designed with a variable-length structure that incorporates a guide rail-slider mechanism. This mechanism realizes the automatic adaptive grasping of the gripper. Secondly, a single-motor-driven mechanism capable of both power distribution and reconfiguration has been devised. This mechanism simplifies the control complexity by enabling simultaneous grasping and reconfiguration. By changing the position and posture of the fingers, the gripper can perform multiple grasp modes. Finally, the performance of the gripper is experimentally validated. In particular, a series of grasping trials are undertaken with objects exhibiting a wide range of weights, shapes, and materials. The outcomes illustrate the gripper’s versatility across various grasping cases, highlighting its potential for broad application.

Visual odometry (VO) is a critical component of autonomous robot systems, enabling precise pose estimation from visual inputs. Learning-based VO methods are increasingly recognized for their robustness in challenging scenarios, including dynamic environments, motion blur, and low-light conditions. However, their performance is constrained by both the diversity of the data and its utilization rate. To overcome these limitations, we propose an end-to-end monocular VO system incorporating a novel learning-based end-to-end VO framework and multiple analogy augmentation strategies. We introduce the Context Attention Uncertainty-aware VO Network (CUVO), which prioritizes semantically rich regions and mitigating interference from high-uncertainty areas to enhance attentional focus and pose estimation accuracy. Furthermore, our analogy augmentation methods—temporal reversal, random rotation, and geometric mirroring—enhance image pairs and compute corresponding true pose transformations, significantly increasing training data quantity and diversity. Simultaneously, an analogous loss is applied to ensure consistency between the original and augmented data. Extensive experiments demonstrate that CUVO significantly enhances VO performance, outperforming previous end-to-end VO methods on TartanAir and KITTI datasets. By leveraging analogy augmentation strategy to expand training data under limited data conditions (27k), zero-shot capability of CUVO degrades by up to 29.5% on TartanAir and 23.3% on KITTI. Our work introduces the first image-to-pose data augmentation method tailored for VO and establishes CUVO as a robust system for advancing learning-based visual odometry.

Infrared (IR) optical gas imaging (OGI) is widely adopted in industrial environments for detecting fugitive gas emissions. However, conventional IR OGI systems rely heavily on manual inspection, lacking capabilities for active leak localization and in-depth analysis, which increases labor costs and risks of human error. To address these challenges, we present LeakHunter, an intelligent multitask framework designed for industrial gas leak monitoring and decision support. LeakHunter integrates seamlessly with IR cameras and can be deployed on edge computing devices, enabling real-time, on-site leak detection in harsh industrial settings. At the core of LeakHunter is a novel keypoint detection paradigm tailored for IR OGI, capable of localizing both leak sources and diffusion endpoints to enable effective spatiotemporal trend analysis. The framework also estimates critical leak attributes, including plume morphology and flow rate, supporting rapid and informed response. To further enhance detection accuracy, we introduce a biomimetic attention module that improves gas-background separation under complex thermal conditions, and a collaborative multitask head for efficient cross-task feature sharing. In addition, two benchmark datasets are proposed, one of which is a field-test set collected in real industrial scenarios. Experiments demonstrate that LeakHunter achieves state-of-the-art performance across multiple tasks, with an F2 score of 94.8% for gas segmentation and 97.9% for leak keypoint localization, while running at 28.4 FPS on a portable IR OGI device. These results highlight its potential as a deployable, intelligent solution for enhancing industrial safety and automation.

Sketches, as a new solution in multimedia systems that can replace natural language, are characterized by sparse visual cues such as simple strokes that differ significantly from natural images containing complex elements such as background, foreground, and texture. This misalignment poses substantial challenges for zero-shot sketch-based image retrieval (ZS-SBIR). Prior approaches match sketches to full images and tend to overlook redundant elements in natural images, leading to model distraction and semantic ambiguity. To address this issue, we introduce a distraction-agnostic framework, purified cross-domain matching (PuXIM), which operates on a straightforward principle: masking and matching. We devise a visual-cross-linguistic (VxL) sampler that generates linguistic masks based on semantic labels to obscure semantically irrelevant image features. Our novel contribution is the concept of purified masked matching (PMM), which comprises two processes: (1)reconstruction, which compels the image encoder to reconstruct the masked image feature, and (2)interaction, which involves a transformer decoder that processes both sketch and masked image features to investigate cross-domain relationships for effective matching. Evaluated on the TU-Berlin, Sketchy, and QuickDraw datasets, PuXIM sets new benchmarks in terms of performance. Importantly, the distraction-agnostic nature of the matching process renders PuXIM more conducive to training, enabling efficient adaptation to zero-shot scenarios with reduced data requirements and low data quality.

Predicting 3D Semantic Scene Graphs (3DSSG) is vital for understanding complex scenes by constructing structured representations. Current methods struggle with significant granularity discrepancies among instances, often relying on features at a single scale, which hampers their ability to perceive and interact with differently sized instances. To tackle this challenge, we introduce GranSSG, a novel approach that integrates volumetric granular awareness into 3DSSG prediction. Central to GranSSG is the Volumetric Pooling block, which aggregates features from multiple instance volumes, enhancing the representation of instance patterns across different granularities. Complementing this, the Granularity Transformer block dynamically directs attention to instance features across various network layers, ensuring precise perception of instances regardless of their granularity. Furthermore, the Cross-Granularity Correlation Transformer block mitigates performance degradation in instance pair relationship prediction by adaptively fusing hybrid features from different granularities, providing a comprehensive representation of instance pairs. Extensive evaluations on the challenging 3DSSG benchmark demonstrate that GranSSG significantly enhances prediction performance, setting a new state-of-the-art in 3DSSG prediction.

Sketching is a quick ideation and multimedia tool for effectively expressing design intent. By translating simple strokes into CAD models, it allows non-expert users to create editable designs, reducing the learning curve associated with traditional CAD software. However, current sketch-based CAD modeling methods are often limited to basic shapes and require structured inputs, making them less robust when dealing with varied sketch styles. To overcome these challenges, we propose a novel sketch-based modeling framework DAFU-CAD, that is both efficient and robust. Our approach features a Depth-Assisted and Feature-Unraveling sketch classification module that categorizes sketches into corresponding modeling operations, independent of their drawing style. A parameter regression and optimization module then estimates the modeling parameters, ensuring consistent and stable model reconstruction across different sketch inputs. To support this, we compile a diverse sketch dataset with a range of modeling categories and abstraction levels. Experimental results show that our method outperforms existing approaches in terms of both robustness and versatility.

3D Visual Grounding (3DVG) faces persistent challenges due to coarse scene-level observations and logically inconsistent annotations, which introduce ambiguities that compromise data quality and hinder effective model supervision. To address these challenges, we introduce Refer-Judge, a novel framework that harnesses the reasoning capabilities of Multimodal Large Language Models (MLLMs) to identify and mitigate toxic data. At the core of Refer-Judge is a Jury-and-Judge Chain-of-Thought paradigm, inspired by the deliberative process of the judicial system. This framework targets the root causes of annotation noise: jurors collaboratively assess 3DVG samples from diverse perspectives, providing structured, multi-faceted evaluations. Judges then consolidate these insights using a Corroborative Refinement strategy, which adaptively reorganizes information to correct ambiguities arising from biased or incomplete observations. Through this two-stage deliberation, Refer-Judge significantly enhances the reliability of data judgments. Extensive experiments demonstrate that our framework not only achieves human-level discrimination at the scene level but also improves the performance of baseline algorithms via data purification. Code is available at https://github.com/Hermione-HKX/Refer_Judge.

Recovering a user-special and controllable human model from a single RGB image is a nontrivial challenge. Existing methods usually generate static results with an image consistent subject's pose. Our work aspires to achieve pose-controllable human reconstruction from a single image by learning a dynamic (multi-pose) implicit field. We first construct a feature-embedded human model (FEHM) as a bridge to propagate image features to different pose spaces. Based on FEHM, we then encode three pose-decoupled features. Global image features represent user-specific shapes in images and replace widely used pixel-aligned ways to avoid unwanted shape-pose entanglement. Spatial color features propagate FEHM-embedded image cues into 3D pose space to provide spatial high-frequency guidance. Spatial geometry features improve reconstruction robustness by using the surface shape of the FEHM as the prior. Finally, new implicit functions are designed to predict the dynamic human implicit fields. For effective supervision, a realistic human avatar dataset, SimuSCAN, with 1000+ models is constructed using a low-cost hierarchical mesh registration method. Extensive experiments demonstrate that our method achieves the state-of-the-art reconstruction level.

Single-view clothing reconstruction usually relies on topologically fixed clothing templates to reduce the problem complexity, but this strategy also makes the reconstructed clothing shape contours simple and lack diversity. In this article, we propose a novel clothing reconstruction method to generate complex shape contours and open clothing mesh from a single image. At the heart of our work is an implicit unsigned distance field condition on clothing-oriented and pose-stable spatial shape features to represent the clothing from the image. This feature can provide spatially aligned clothing shape priors to improve the pose robustness. It is based on a type-generic clothing template derived from the mainstream clothing generative model to avoid tedious template design and switching. To output open clothing mesh results from noisy clothing unsigned distance fields, we develop a two-stage clothing mesh extraction method. It takes the point clouds as an intermediate representation and produces smooth, plausible and editable clothing mesh results. To provide effective supervision, we construct a pose-rich and shape-complete clothing scan dataset by enhancing clothing pose diversity and complementing missing clothing geometry caused by occlusion. Extensive experiments demonstrate that our method achieves state-of-the-art levels. More importantly, we provide a simple but effective, and low-cost way to reconstruct complex shape contours clothing from a single image.

Sketch-based modeling studies reconstructing models from sketches automatically, allowing users visualize design concepts rapidly. Generating CAD models based on user sketches helps reduce the learning curve for novice users, which promotes the everyday use of CAD software, and expands its reach to non-professional groups. While various algorithms study automatically generating models from single sketch or line drawing, they often produce non-editable models or editable models limited to simple extrusion operations. To improve this issue, we propose a novel sketch-based modeling system, Sketch2Seq, which generates complex, semantic, and editable CAD models. Our system eliminates the need for additional annotations from users and produces models that support subsequent application in commercial software. The core of our method lies in understanding users' design intent from CAD sketches. We design a novel sketch segmentation network for identifying diverse operation features in CAD sketches, which utilizes geometric features of strokes and different levels of topological connections. Additionally, to tackle the segmentation task, a dataset for CAD sketch segmentation is introduced. Comparative experiments and ablation evaluations prove the effectiveness of the proposed method. Based on segmentation result, coarse CAD sequences are generated and progressively executed. Meanwhile, the orders and parameters of the CAD sequences are optimized with context models and input sketches. All algorithms are integrated into a user interface. Experiments and evaluations validate the feasibility and superiority of our entire system which is able to reconstruct more complex features and achieve better results for longer sequence.

Under-actuated robot grippers, as a pervasive tool of robots, have become a considerable research focus. Despite their simplicity of mechanical design and control strategy, they suffer from poor versatility and weak adaptability, making widespread applications limited. To better address relevant research gaps, we present a novel 3-finger linkage-based gripper that realizes retractable and reconfigurable multi-mode grasps driven by a single motor. Firstly, inspired by the changes occurred in the contact surface with a human finger moving, we artfully design a slider-slide rail mechanism as the phalanx to achieve retraction of each finger, allowing for better performance in the enveloping grasping mode. Secondly, a reconfigurable structure is constructed to broaden the grasping range of objects’ dimensions for the proposed gripper. By adjusting the configuration and gesture of each finger, the gripper can achieve five grasping modes. Thirdly, the proposed gripper is solely actuated by a single motor, yet it can be capable of grasping and reconfiguring simultaneously. Finally, various experiments on grasps of slender, thin, and large-volume objects are implemented to evaluate the performance of the proposed gripper in practical scenarios, which demonstrates the excellent grasping capabilities of the gripper.

Collaboration with robots through physical contact offers a more intuitive, natural, and engaging operational experience, showcasing vast potential in the field of human-robot interaction. However, current physical interaction devices, such as collaborative robots and haptic feedback mechanisms, are limited by their singular modes of motion and feedback, hindering enhancements in interaction experiences. Herein, we present a multistable soft actuator capable of driving multimodal shape changes and passively conforming to user touch. This actuator can memorize and maintains any deformation with zero power consumption. Its structural mechanical properties can be dynamically adjusted to produce rich haptic feedback for the user, including changes in shape, elasticity, stiffness, and even sensations of rupture and weightlessness. Structurally, the mechanism consists of a network of pneumatic bistable units in series and parallel configurations, which can switch states under air pressure or external force, achieving extension, contraction, and omnidirectional bending. The input of air pressure can either impede or assist deformation, altering structural stiffness and resulting in varied loading curves. With its high safety in physical interactions, robust operability, and rich mechanical tactile feedback, the multistable soft actuator promises new design directions for physical human-robot interaction devices.

This paper addresses the significant challenges in 3D Semantic Scene Graph (3DSSG) prediction, essential for understanding complex 3D environments. Traditional approaches, primarily using PointNet and Graph Convolutional Networks , struggle with effectively extracting multi-grained features from intricate 3D scenes , largely due to a focus on global scene processing and single-scale feature extraction. To overcome these limitations, we introduce Granular3D, a novel approach that shifts the focus towards multi-granularity analysis by predicting relation triplets from specific sub-scenes. One key is the Adaptive Instance Enveloping Method (AIEM), which establishes an approximate envelope structure around irregular instances, providing shape-adaptive local point cloud sampling, thereby comprehensively covering the contextual environments of instances. Moreover, Granular3D incorporates a Hierarchical Dual-Stage Network (HDSN), which differentiates and processes features of instances and their pairs at varying scales, leading to a targeted prediction of instance categories and their relationships. To advance the perception of sub-scene in HDSN, we design a Gather Point Transformer structure (GaPT) that enables the combinatorial interaction of local information from multiple point cloud sets, achieving a more comprehensive local contextual feature extraction. Extensive evaluations on the challenging 3DSSG benchmark demonstrate that our methods provide substantial improvements, establishing a new state-of-the-art in 3DSSG prediction, boosting the top-50 triplet accuracy by ＋2.8%.

We propose a robust and highly realistic clothing modeling method to generate a 3D clothing model with visually consistent clothing style and wrinkles distribution from a single RGB image. Notably, this entire process only takes a few seconds. Our high-quality clothing results benefit from the idea of combining learning and optimization, making it highly robust. First, we use the neural networks to predict the normal map, a clothing mask, and a learning-based clothing model from input images. The predicted normal map can effectively capture high-frequency clothing deformation from image observations. Then, by introducing a normal-guided clothing fitting optimization, the normal maps are used to guide the clothing model to generate realistic wrinkles details. Finally, we utilize a clothing collar adjustment strategy to stylize clothing results using predicted clothing masks. An extended multi-view version of the clothing fitting is naturally developed, which can further improve the realism of the clothing without tedious effort. Extensive experiments have proven that our method achieves state-of-the-art clothing geometric accuracy and visual realism. More importantly, it is highly adaptable and robust to in-the-wild images. Further, our method can be easily extended to multi-view inputs to improve realism. In summary, our method can provide a low-cost and user-friendly solution to achieve realistic clothing modeling.

The Roller-Quadrotor is a novel quadrotor that combines the maneuverability of aerial drones with the endurance of ground vehicles. This work focuses on the design, modeling, and experimental validation of the Roller-Quadrotor. Flight capabilities are achieved through a quadrotor config-uration, with four thrust-providing actuators. Additionally, rolling motion is facilitated by a unicycle-driven and rotor-assisted turning structure. By utilizing terrestrial locomotion, the vehicle can overcome rolling and turning resistance, thereby conserving energy compared to its flight mode. This innovative approach not only tackles the inherent challenges of traditional rotorcraft but also enables the vehicle to roll through narrow gaps and overcome obstacles by taking advantage of its aerial mobility. We develop comprehensive models and controllers for the Roller-Quadrotor and validate their performance through experiments. The results demonstrate its seamless transition between aerial and terrestrial locomotion, as well as its ability to safely roll through gaps half the size of its diameter. Moreover, the terrestrial range of the vehicle is approximately 2.8 times greater, while the operating time is about 41.2 times longer compared to its aerial capabilities. These findings underscore the feasibility and effectiveness of the proposed structure and control mechanisms for efficient rolling through challenging terrains while conserving energy.

In many robot operation scenarios, the end-effector’s attitude constraints of movement are indispensable for the task process, such as robotic welding, spraying, handling, and stacking. Meanwhile, the inverse kinematics, collision detection, and space search are involved in the path planning procedure under attitude constraints, making it difficult to achieve satisfactory efficiency and effectiveness in practice. To address these problems, we propose a distributed variable density path planning method with attitude constraints (DVDP-AC) for industrial robots. First, a position–attitude constraints reconstruction (PACR) approach is proposed in the inverse kinematic solution. Then, the distributed signed-distance-field (DSDF) model with single-step safety sphere (SSS) is designed to improve the efficiency of collision detection. Based on this, the variable density path search method is adopted in the Cartesian space. Furthermore, a novel forward sequential path simplification (FSPS) approach is proposed to adaptively eliminate redundant path points considering path accessibility. Finally, experimental results verify the performance and effectiveness of the proposed DVDP-AC method under end-effector’s attitude constraints, and its characteristics and advantages are demonstrated by comparison with current mainstream path planning methods.

We present a novel indirect adaptive fuzzy-regulated optimal control scheme for continuous-time nonlinear systems with unknown dynamics, mismatches, and disturbances. Initially, the Hamilton-Jacobi-Bellman (HJB) equation associated with its performance function is derived for the original nonlinear systems. Unlike existing adaptive dynamic programming (ADP) approaches, this scheme uses a special non-quadratic variable performance function as the reinforcement medium in the actor-critic architecture. An adaptive fuzzy-regulated critic structure is correspondingly constructed to configure the weighting matrix of the performance function for the purpose of approximating and balancing the HJB equation. A concurrent self-organizing learning technique is designed to adaptively update the critic weights. Based on this particular critic, an adaptive optimal feedback controller is developed as the actor with a new form of augmented Riccati equation to optimize the fuzzy-regulated variable performance function in real time. The result is an online indirect adaptive optimal control mechanism implemented as an actor-critic structure, which involves continuous-time adaptation of both the optimal cost and the optimal control policy. The convergence and closed-loop stability of the proposed system are proved and guaranteed. Simulation examples and comparisons show the effectiveness and advantages of the proposed method.

Abstract Three‐dimensional human model reconstruction has wide applications due to the rapid development of computer vision. The appearance of cheap depth camera, such as Kinect, opens up new horizons for home‐oriented 3D human reconstructions. However, the resolution of Kinect is relatively low, making it difficult to build accurate human models. In this paper, we improve the accuracy of human model reconstruction from two aspects. First, we improve the depth data quality by registering the depth images captured from multi‐views with a single Kinect. The part‐wise registration method and implicit‐surface‐based de‐noising method are proposed. Second, we utilize a statistical human model to iteratively augment and complete the human body information by fitting the statistical human model to the registered depth image. Experimental results and several applications demonstrate the applicability and quality of our system, which can be potentially used in virtual try‐on systems. Copyright © 2015 John Wiley & Sons, Ltd.