Robotic hand tool use has garnered significant attention from robotics researchers, because it enhances dexterity beyond the limitations imposed by manipulators with fixed tool configurations and human-involved manual tool changes. Despite extensive research, current methodologies predominantly focus on imitating human hand trajectories, often neglecting the pivotal role of tool-environment interaction. This study addresses this gap by exploring the task of cucumber peeling as a case study to implement contact-based demonstration strategies in robotic tool use. Our approach concentrates on the subtle tool contact behaviors that manifest through contact dynamics. Specifically, we select appropriate tool stiffness for the peeling tasks, which is captured via a handheld teaching device equipped with optical tactile sensors. Subsequently, object-level stiffness control strategies are employed to emulate these behaviors using a three-fingered robotic hand. Experimental results from real-world cucumber peeling trials substantiate our methodology, illustrating that the robotic hand can adjust contact through finger movements, thereby achieving humanlike peeling efficiency without necessitating alterations to the tool structure. This study not only demonstrates the feasibility of sophisticated tool use by robotic hands, but also highlights the critical importance of integrating tactile feedback to refine interaction with the environment.

Tactile servoing is an effective approach to enabling robots to safely interact with unknown environments. One of the core problems in tactile servoing is to robustly converge the contact features to the desired ones via a dedicated controller. This paper proposes a Data-Driven Model Predictive Controller (DDMPC) to compute the motion command given the previous interaction experience and feature deviations in tactile space. Compared with the manually designed PID-based controller, the proposed controller depends on the sound control theory and its convergence is guaranteed from a computational perspective. It is applied to the balancing control of a rolling bottle on a robotic forearm covered by a custom tactile sensor array. The real experiment demonstrates the superior robustness of the proposed approach and shows its great potential for other tactile servoing scenarios with measurement noise, which is inevitable for current tactile sensors.

Anthropomorphic robotic arms, mimicking the structure and function of human arms, show great potential for helping people in various tedious and repetitive household tasks. However, such arms mostly consist of multiple serial links controlled independently by actuators at joints with high reduction ratios, posing challenges in household services in terms of load capacity, responsiveness, and safety. In this paper, we propose a high-performance anthropomorphic arm called TRX-Arm based on differential cable transmission, characterized by features of high dynamics, high load capacity, and inherent compliance. TRX-Arm is composed of three deferential cable-driven coupling joints and one independent roll joint. Thanks to the cable differential transmission, the joints are capable of achieving doubled torque and stiffness without replacing motors. To enhance safety in human-robot interaction, the actuators including motors, reducer, belt, and pulley are mounted at the shoulder near the base and drive the joints remotely using cables, thereby minimizing the inertia of the whole arm. The workspace of TRX-Arm has a volume of 1.56 m3, much larger than that of the human arm. Real experiments show its capabilities including high repeatability and load capacity as well as high dynamic behavior of a dual-arm robot platform built with TRX-Arms.

Objects of daily life are designed to suit the human hand. Without major modifications to these objects and our environments, robots will need end-effectors with human hand-like configuration and dexterity to efficiently operate on them. Tight integration of tactile and proprioceptive sensors are also critical to ensure robust execution of manipulation policies without sacrificing range-of-motion. Reliability is also key, and a mechanically robust, easy to repair end-effector is important to minimize downtime. To meet these challenges, we designed a 13 degree-of-freedom anthropomorphic hand with over 1000 tactile sensing elements, named TRX-Hand5. Also embedded within are positional encoders and cable tension sensors to provide proprioceptive perception. TRX-Hand5 has a novel biomimetic topology with six small posture motors in the palm to replicate the function of intrinsic hand muscles and five large power motors in the forearm to play the role of forearm flexor muscles. The whole hand weighs 2.6 kg with its dimensions comparable to those of an adult male’s hand and is capable of actuating its fingertips at over 200°/s while exerting up to 22 N of force. The system can be disassembled in modules for easy maintenance.

Video-based visible-infrared person re-identification (VVI-ReID) aims to match the identity of a person captured in video sequences from both visible and infrared cameras. The VVI-ReID task requires considering both the spatial relationship between body parts within each frame and the temporal change of appearance between successive frames. Existing VVI Re-ID methods employ Convolutional Neural Networks to extract local spatial features and Long Short-Term Memory to form temporal associations. However, these methods can not effectively capture the global spatial feature and the long-range temporal dependencies in ultra-long sequences. In this paper, we propose a Cross-modality Spatial-temporal Transformer (CST) including a Cross-frame Tube Transformer Module (CTTM) and a Multi-frame Transformer Fusion Module (MTFM) to address these challenges. Firstly, CTTM tokenizes a video clip into multiple 3D tubes, each encapsulating local spatial-temporal information of pedestrians, and then obtains global spatial-temporal representations by establishing the relationship between tubes. Secondly, we design MTFM to exchange information between multiple frames using message tokens, thus modeling the long-range temporal dependencies of features of pedestrians. In addition, to prevent the potential representation collapse caused by triplet-based loss functions, we propose a diversity-consistency (DC) loss function to preserve the diversity and consistency of cross-modality feature representations by imposing variance, invariance, and covariance constraints in feature representations. Extensive benchmark experiments demonstrate that our approach outperforms the state-of-the-art methods with large margins.

With the rapid growth of various images and video data on the network platform, multimodal emotional analysis and recognition have become an increasingly popular research field. Inspired by the human emotional judgment and cross information between different characteristics, this paper proposed a multimodal video emotional analysis network of time features alignment and information auxiliary learning(METI). This network not only fully consider the time alignment between modals to obtain the preliminary information but also pay attention to the auxiliary information in the video and make emotional judgments. METI network is mainly composed of the following three parts: Time alignment network pays attention to time alignment information between different modes; Prepose feedback module uses the auxiliary information in the video to simulate the emotional judgment process; Multimodal gate control module pays attention to the emotional correlation of the context. The comparison of experimental results shows that the performance of the METI model on the dataset exceeds the current most advanced emotional analysis model. At the same time, it has been proved from ablation experiments that the modules and methods proposed in this paper can improve the accuracy of emotion recognition by nearly 4%, and has been respectively proved in the seven classification experiment of emotions and three classification experiment of sentiment.

For primary school teachers, revising arithmetical exercises is labor-intensive and time-consuming task. To reduce their burdens, we propose a lightweight arithmetic evaluation system that can automatically assess arithmetic exercises. The designed system divides into two branches: detection and recognition. In the detection branch, a coordinate attention module is added to the feature pyramid infrastructure of the YOLOv5 target detection network to improve the multi-scale target recognition ability. Then, the complete IoU (CIoU) loss function leads to faster convergence and better performance on the detection results. The experimental results show that the detection accuracy of the proposed method improves [email protected](%) by 5.4% compared to the original algorithm. Moreover, the number of parameters and FLOPS decrease to 1.23M and 3.8G. In the recognition branch, we employ the improved-CRNN network to recognize arithmetical exercises, achieving an accuracy being up to 97.1 %.

Visible-infrared person re-identification (VI-ReID) aims to match person images between the visible and near-infrared modalities. Previous VI-ReID methods are based on holistic pedestrian images and achieve excellent performance. However, in real-world scenarios, images captured by visible and near-infrared cameras usually contain occlusions. The performance of these methods degrades significantly due to the loss of information of discriminative features from the occlusion of the images. We define visible-infrared person re-identification in this occlusion scene as Occluded VI-ReID, where only partial content information of pedestrian images can be used to match images of different modalities from different cameras. In this paper, we propose a matching framework for occlusion scenes, which contains a local feature enhance module (LFEM) and a modality information fusion module (MIFM). LFEM adopts Transformer to learn features of each modality, and adjusts the importance of patches to enhance the representation ability of local features of the non-occluded areas. MIFM utilizes a co-attention mechanism to infer the correlation between each image for reducing the difference between modalities. We construct two occluded VI-ReID datasets, namely Occluded-SYSU-MM01 and Occluded-RegDB datasets. Our approach outperforms existing state-of-the-art methods on two occlusion datasets, while remains top performance on two holistic datasets.

A cable-driven segmented manipulator (CDSM) has superior dexterity for operations in confined space due to its light-slender body and redundant degree of freedoms (DOFs). However, its inverse kinematics resolving and configuration planning are very challenging due to the complex structure and strict constraints. In this article, we propose a two-layer geometric iteration (TLGI) method for inverse kinematics resolving and configuration-constrained Cartesian path planning. The computation efficiency is largely improved and singularities are avoided. First, the end-effector attitude is decomposed into a direction vector and a rotation angle. The former and the end-effector position are combined into state variables of the inner layer, and the latter is treated separately as the state variable of the outer layer. Then, the TLGI method enables to rapidly reach the desired 6-DOF pose by two-layer iterations, i.e., the inner and outer loop iteration. Second, during the inner loop iteration, the CDSM is modeled as an equivalent articulated arm whose end-effector position and direction is the same as that of CDSM, but its links length and joint angles depend on the current configuration of CDSM. Then, the efficient forward and backward reaching inverse kinematics (FABRIKs) method is extended to apply on CDSM so that it can fast reach the inner state variables. During the outer loop iteration, three different rotation cases, i.e., the rotating around the end, root, and both end and root, are designed to switch automatically to reach the outer state variable iteratively. Moreover, by parameterizing geometric constraints of the environment, a TLGI-based configuration-pose simultaneous planning method is also put forward to efficiently achieve additional configuration constraints for operations of CDSM in confined space. Finally, the proposed method is verified by both the simulations and experiments.

With the rapid development of social media, visual sentiment analysis from image or video has become a hot spot in visual understanding researches. In this work, we propose an effective approach using visual and textual fusion for sentiment analysis of short GIF videos with textual descriptions. We extract both sequence-level and frame-level visual features for each given GIF video. Next, we build a visual sentiment classifier by using the extracted features. We also define a mapping function, which converts the sentiment probability from the classifier to a sentiment score used in our fusion function. At the same time, for the accompanied textual annotations, we employ the Synset forest to extract the sets of the meaningful sentiment words and utilize the SentiWordNet3.0 model to obtain the textual sentiment score. Then, we design a joint visual-textual sentiment score function weighted with visual sentiment component and textual sentiment one. To make the function more robust, we introduce a noticeable difference threshold to further process the fused sentiment score. Finally, we adopt a grid search technique to obtain relevant model hyper-parameters by optimizing a sentiment aware score function. Experimental results and analysis extensively demonstrate the effectiveness of the proposed sentiment recognition scheme on three benchmark datasets including T-GIF dataset, GSO-2016 dataset and Adjusted-GIFGIF dataset.

With many degrees of freedom (DOFs), a hyper-redundant manipulator has superior dexterity and flexible manipulation ability. However, its inverse kinematics and configuration planning are very challenging. With the increase in the number of DOFs, the corresponding computation load or training set will be much larger for traditional methods (such as the generalized inverse method and the artificial neural network method). In this paper, a segmented geometry method is proposed for a spatial hyper-redundant manipulator to solve the above problems. Similar to the human arm, the hyper-redundant manipulator is segmented into three sections from geometry, i.e., shoulder, elbow, and wrist. Then, its kinematics can be solved separately according to the segmentation, which reduces the complexity of the solution and simplifies the computation of the inverse kinematics. Furthermore, the configuration is parameterized by several parameters, i.e., the arm-angle, space arc parameters, and desired direction vector. The shoulder has proximal four DOFs, which is redundant for positioning the elbow and avoiding the joint limit. The arm-angle parameter is defined to solve the redundancy. The wrist consists of the distal two DOFs, and its joints are determined to match the desired direction vector of the end-effector. All the other joints (except for the joints belonging to shoulder and wrist) compose the elbow. These joint angles are solved by using space arc-based method. The configuration planning for avoiding joint limit, obstacles, and inspecting narrow pipeline are detailed for practical applications. Finally, circular trajectory tracking and pipeline inspection are, respectively, simulated and experimented on a 20-DOFs hyper-redundant manipulator. The results show that the proposed method can give solutions of the three-dimensional-pose-determining problem and the configuration-planning problem. The computation of the inverse kinematics is simplified for real-time control. It can also be applied to other spatial hyper-redundant manipulators with similar serial configurations.

Cable-driven segmented manipulators (CDSMs) featured by superior dexterity, light and slender body are excellent candidates for operations in confined environments. However, the stiffness and load capacity of such manipulators have been a challenge due to their structural elasticity. In this paper, we propose an improved mechanism design based on the preliminary work to enhance the linkage accuracy and arm continuity without sacrificing the dexterity, high stiffness and load capacity of CDSM. The manipulator is composed of 4 improved hybrid active-passive linkage segments. Its short and long linkage cables with pretension mechanism are designed to keep equal angles of adjacent joints. An improved separable small driving control box is also designed with both quick release and load mechanism and stroke amplification mechanism. Then the size of control box can still remain small, even the number of segments and the joint limit angles increase. Considering the improved in-segment linkage characteristic, traditional kinematic equations and Jacobian matrix are greatly simplified with Denavit-Hartenberg (D-H) method. Further trajectory tracking planning based on the simplified kinematics solved the Cartesian space planning for task design. Finally, a prototype system is developed to perform the linkage accuracy and comprehensive obstacle avoidance experiments. Experimental results show that the developed hybrid active and passive CDSM has relatively high accuracy and super dexterity.

With a light and slender body, a cable-driven redundant spatial manipulator (CRSM) has flexible manipulability and high maneuverability in confined environment. However, compared with revolute rigid manipulators, such type of manipulators generally has low stiffness and weak load capacity. In this paper, we propose a new mechanism design to improve the stiffness and load capacity without sacrificing the manipulator dexterity and the end-effector accuracy. The manipulator is composed of 3 active-passive-linkage segments and 1 active tool end-effector. Each active-passive segment has 2 degrees of freedom (DOFs) driven by three evenly distributed cables. Pretension mechanism and linkage cables are designed to keep strict equal angles of adjacent joints. A separable control box, which contains all the motors and cable transmission mechanisms is also designed with a quick release-and-lock mechanism. Therefore, the robotic arm can be easily removed and installed. Based on the equal angle characteristic, kinematic equations of manipulator are established with Denavit-Hartenberg (D-H) method and the Jacobian matrix is also simplified. Further analysis of the workspace supplies the guidance for the task design and motion planning. Finally, a prototype system is developed to perform the stiffness and load capacity experiments. Experimental results show that the developed CRSM has relatively high stiffness and load capacity.

A snake-like robot is a hyper-redundant robot. It has flexible movement ability and high stability with low center of gravity. It is very suitable for environment detection in the rugged road or narrow space. In this paper, a 16-DOFs snake robot is composed. It has ten 2-DOFs modular spherical-shape joints. The joints are arranged as the structure of "(Roll-Pitch)-(Roll-Pitch)-", where "(Roll-Pitch)" denotes a modularized 2-DOF joint, which can rotate along the roll and pitch axis. The exterior frame of each joint is designed as a spherical structure, which is connected with the motor through two stage reduction mechanism. Therefore, the drive torques are largely increased. Many small passive wheels are mounted along a circle of the exterior surface. Such design largely decrease the friction between the robot and the road. It also has more movement modes than the traditional design. We also develop the embedded controller based on the ARM processor and uc/os-ii real-time operation system. The gait planning algorithms are programmed using C language and realized in the embedded processor. At last, typical cases are experimented. The experiment results show that the developed robot has high mobility and flexibility.

The existing quaternion-type moments (QTMs) are based on the quaternion representation (QR) of color images. However, this representation creates redundancy when using four-dimensional quaternions to represent color images with three components. In this paper, for RGB-D images, the QR is improved by combining both color and depth information, which is invariant to lighting and color variations. The improved QR fully utilizes the four-dimensional quaternion domain. The new QTMs (NQTMs) are defined using the improved QR. They are combined with the quaternion back-propagation neural network (QBPNN) for RGB-D object recognition. The experimental results demonstrate that the NQTMs outperform our previous QTMs considering only color information.

The linear-chain CRFs is one of the most popular discriminative models for human action recognition, as it can achieve good prediction performance in temporal sequential labeling by capturing the one-or few-timestep interactions of the target states. However, existing CRFs formulations have limited capabilities to capture deeper intermediate representations within the target states and higher order dependence between the given states, which are potentially useful and significant in the modeling of complex action recognition scenarios. To address these issues, we formulate a deep recursive and hierarchical conditional random fields (DR-HCRFs) model in an infinite-order dependencies framework. The DR-HCRFs model is able to capture richer contextual information in the target states, and infinite-order temporal-dependencies between the given states. Moreover, we derive a mean-field-like approximation of the model marginal likelihood to efficiently facilitate the model inference. The parameters of the predefined model are learnt with the block-coordinate primal-dual Frank-Wolfe algorithm in a structured support vector machine framework. Experimental results on the CAD-120 benchmark dataset demonstrate that the proposed approach can achieve high scalability and perform better than other state-of-the-art methods in terms of the evaluation criteria.

We propose a parallel training framework of convolutional neural networks (CNNs) for small sample learning. In the framework we model the feature filter process and show Sadowsky energy distribution exists in the model. Using Sadowsky energy distribution, the weights in convolutional kernels can be rearranged after each update according to special cases. With this rearrangement, each CNNs in the framework has different predicted probability especially for easily misclassified samples, which avoids the situation of a low predicted probability traditional CNNs may have. The class that gets the maximum predicted probability among the CNNs would be chosen as the result of prediction. Our CNNs framework gives better hand-written digit classification for small samples than one-stage CNNs, and has a faster convergence rate than multiplestages CNNs.