Dynamic object grasping in human-robot shared workspaces presents significant challenges in ensuring grasping success, efficiency, and safety. To address these challenges, we propose a novel Actor-Critic reinforcement learning (RL) framework integrated with a differentiable nonlinear least squares (DNLS) optimization module. The RL component generates target poses and adaptively tunes cost function weights for DNLS, while the DNLS module refines motion trajectories to satisfy kinodynamic and safety constraints. The framework supports end-to-end training and enhances grasping performance and training speed, thanks to the GPU-accelerated DNLS optimization. We evaluate the proposed method in both simulation and real-world environments, demonstrating superior performance over baseline method in terms of success rate, efficiency, safety, and trajectory smoothness. These results highlight the effectiveness of our approach and its potential for enabling safe and reliable human-robot collaboration in dynamic, safety-critical environments.

In this article, a novel simulation-to-real (sim2real) multimodal learning framework is proposed for adaptive dexterous grasping and grasp status prediction. A two-stage approach is built upon the Isaac Gym and several proposed pluggable modules, which can effectively simulate dexterous grasps with multimodal sensing data, including RGB-D images of grasping scenarios, joint angles, 3-D tactile forces of soft fingertips, etc. Over 500K multimodal synthetic grasping scenarios are collected for neural network training. An adaptive dexterous grasping neural network (ADG-Net) is trained to learn dexterous grasp principles and predict grasp parameters, employing an attention mechanism and a graph convolutional neural network module to fuse multimodal information. The proposed adaptive dexterous grasping method can detect feasible grasp parameters from an RGB-D image of a grasp scene and then optimize grasp parameters based on multimodal sensing data when the dexterous hand touches a target object. Various experiments in both simulation and physical grasps indicate that our ADG-Net grasping method outperforms state-of-the-art grasping methods, achieving an average success rate of 92% for grasping isolated unseen objects and 83% for stacked objects. Code and video demos are available at https://github.com/huikul/adgnet.

In this article, a six-step approach is proposed to simulate the grasp and evaluate the grasp quality for a versatile vacuum gripper by tracking the deformation and force-torque wrench of the gripping pad. Over 100 K synthetic grasps are generated for neural network training. Furthermore, a gripping attention convolutional neural network (GA-CNN) is developed to predict the grasp quality for real-world grasp, running by 15 Hz closed-loop control with the real-time robotic observation and force-torque feedback. Various experiments in both the simulation and physical grasps indicate that our GA-CNN can focus on the crucial region of the soft gripping pad to predict grasp qualities and perform a lower average error compared with a same-scale traditional CNN. In addition, the complexity of grasping clutters is defined from Level 1 to Level 9. The proposed grasping method achieves an average success rate of 90.2% for static clutters at Level 1 to Level 8 and an average success rate of >80.0% for dynamic grasping at Level 1 to Level 7, which outperforms state-of-the-art grasping methods.

In the existing work on active face detection and tracking, it is usually required that the face has to appear in the field-of-view. However, this may not be practical in some challenging scenarios. In this paper, we formulate the problem of active face finding as a Markov Decision Process and resort to the deep Q-learning to solve it in an end-to-end manner. Under the proposed framework, the agent is able to learn how to adjust the control parameters of a camera in order to find the face. Even if the captured image contains only some parts of the person, the PTZ camera can still adjust its pose until the face is found. Extensive experimental validations are performed to show the effectiveness of the developed system.

In this paper, we describe the deep learning method to reduce the dimension of the data samples under the framework Non-negative Matrix Factorization (NMF). That is to say, we try to find the good representation of the data samples for the task of NMF. To this end, a nonlinear NMF optimization model is constructed and the optimization algorithm is developed. The experimental results on some benchmark dataset show the nonlinear dimension reduction helps the NMF to improve the clustering performance.

In this paper, we demonstrate nonlinear features extracted by deep neural network have better results in the task of dictionary learning. A nonlinear dictionary learning model is constructed and the optimization algorithm is developed. In the learning algorithm, we use the deep neural network to convey raw samples to feature space and learn a nonlinear dictionary. The extensive experimental results of classification on some benchmark dataset show the proposed nonlinear dictionary significantly improves the classification performance.