• We propose a smart semantic enhancement collaborative reasoning framework called CORE-CLIP, which is well driven by the openness and robustness of pre-training model CLIP. This framework effectively utilizes the semantic priors of visual-language pre-trained models to construct a text guided visual cross-modal alignment strategy to achieve fine-grained understanding of human-object interaction. The CORE-CLIP can achieve a good tradeoff between the model size and inference speed. • Two novel modules combining text features are elaborated for fine-grained interaction recognition and interaction classification. Specifically, the TDA module uses a dual fusion method to aggregate interactive text information and visual context in the structure to achieve accurate recognition of interaction categories, while the SEC module focuses on three special types of text information to explicitly and efficiently classify objects, actions, and interactions, respectively. • Extensive experiments demonstrate excellent performance on the two benchmarks and self-constructed validating set TableTennis. In particular, the proposed CORE-CLIP improves 1.36 mAP on V-COCO and 1.01 mAP on HICODET compared to GEN-VLKT. Zero-shot experimental results outperform the existing state-of-the-art methods such as GEN-VLKT and HOICLIP. The inference speed of CORE-CLIP surprisingly attains 38.3 FPS while its total parameters are only 52.9M. Ablation studies validate the contribution of each module for efficient and accurate interaction understanding. Human-object interaction (HOI) detection has attracted more and more attention due to its wide potential applications. Recently, Contrastive Language-Image Pre-training (CLIP) achieves promising results in 2D/3D zero-shot and few-shot learning. To overcome current heavy reliance on the large collection of annotated HOI data and often failure to recognize such unseen HOI relationships in training datasets in existing methods, we propose a smart collaborative reasoning framework named CORE-CLIP based on semantic enhancement driven by the openness and robustness of CLIP. Specifically, the text guided dual fusion attention module (TDA) captures precise interaction patterns by progressively integrating CLIP text embeddings, CLIP visual features, and global contextual features. The semantically enhanced explicit interaction classification module (SEC) is initialized by leveraging semantic information of objects, actions, and interactions generated through CLIP text embeddings, to ensure alignment between visual features and linguistic semantics. Extensive experiments demonstrate that CORE-CLIP has state-of-the-art performance on two benchmark datasets. In particular, compared to GEN-VLKT, the proposed CORE-CLIP improves 1.36 mAP on V-COCO and 1.01 mAP on HICO-DET. Zero-shot experimental results outperform state-of-the-art methods GEN-VLKT by a signifigant margin of 34.4 % in UO type setting and HOICLIP by 50.9 % in UO type setting, respectively. The inference speed of CORE-CLIP surprisingly attains 38.3 FPS while its total parameters are only 52.9M.

Category-level 6D object pose estimation has gained increasing attention in applications of robotic manipulation, augmented reality, and scene understanding, due to its ability to generalize to unseen instances within the same category. However, existing methods struggle with handling the intra-class shape variations as they either adopt mean shape as priors, or build the associations among different instances without explicit category-shared information. To address this problem, a novel category-level object pose estimation method based on diffusion model is proposed, which utilize the generative ability of diffusion model to refine a sparse categorical representation. In contrast to existing dense correspondence-based methods, our method employs a set of keypoints provided by learnable queries to represent object shape, enabling better categorical representation of different instances by focusing on the representative object components. The keypoints are then refined through a forward diffusion process and a reverse denoising process conditioned on category information. This allows the flexible adaption to various instances, especially for those that deviate from the mean shape within the same category. On this basis, a geometric-semantic feature fusion module is presented to enhance keypoint feature representation. By integrating the geometric information from point cloud with the high-level semantics from RGB image using a two-branch attention mechanism, the keypoint feature is enriched and deeply combined, which facilitates the subsequent pose estimation. Extensive experiments on the REAL275 dataset, the CAMERA25 dataset, and real-world complex scenarios demonstrated the effectiveness of proposed method.

LiDAR odometry has gained popularity due to the LiDAR sensor’s accurate depth measurement and robustness to varying illumination conditions. The feature-based methods achieve the advantage of efficiency through feature extraction, while the distribution-based ones attain better accuracy by modeling the point cloud as distributions. Combining the strengths of both methods is anticipated to yield superior performance. However, existing combination schemes typically extract and process features and distributions separately, and such a loosely integrated approach cannot fully leverage their complementarity. To address this problem, we propose a novel LiDAR odometry method based on surface distributed point (SDP) feature with dual feature fusion. Specifically, the SDP feature is introduced to tightly integrate features and distributions, facilitating efficient feature association and map maintenance. On this basis, the associated source and target features are then effectively integrated through dual feature fusion to form the dual-fusion (DF) associated plane. This plane serves as the basis for constructing the point-to-DF associated plane constraint for pose optimization. As a result, the local planar structure is more accurately reflected, thereby enhancing the accuracy of pose estimation. The SDP feature and the resultant constraint are employed in both scan-to-map matching and fixed-lag smoothing, which are hierarchically organized to achieve accurate pose estimation. Experiments on KITTI dataset and large-scale KITTI-360 dataset demonstrate the effectiveness of the proposed method.

Visual place recognition is crucial to accurate localization in large-scale environments. Existing methods combine convolutional neural network and deep metric learning to improve performance, however, it is still challenging to promote the adaptability of features under various environmental conditions. To address the problem, this paper proposes a hardness-aware metric learning method with cluster-guided attention. By leveraging the affinity between each local feature and the corresponding scene cluster center, the model is attracted to focus on the local features proximate to their clusters, while suppressing outlier local features that deviate from their clusters. In this way, the scene-related reliable local features are concentrated to construct the global feature of the whole image with adaptability to environmental conditions. Meanwhile, a hardness-aware metric loss is designed to train the proposed network, which determines the hardness of negative samples based on their similarity to the query images and the training iterations. Subsequently, the hardness is used to reweigh each term of the loss function to promote network optimization. In addition, a condition normalization layer is also introduced to regularize the feature distributions under different environmental conditions to a canonical space, improving the feature robustness to condition variations. Our method achieves the top-10 recalls of 97.2%, 99.0%, and 94.9% on Pitts250k-test, TokyoTM-val, and Tokyo 24/7 datasets, respectively. Extensive experiments demonstrate that the proposed method learns robust global features with the adaptability to various environmental conditions.

Few-shot object detection (FSOD) aims to detect novel objects with limited annotated examples. Mainstream methods suffer from the data scarcity of novel classes with insufficient intra-class variations, which makes the trained model biased to base classes. Actually, there are massive unlabeled novel instances in the base dataset and their adequate utilization will enhance the discriminability of model to novel classes. This paper proposes a semi-supervised few-shot object detection method, which utilizes a teacher model and a pre-trained few-shot object detector to guide the learning of a student model through adaptive pseudo labeling. In particular, a class-adaptive threshold filtering (CATF) strategy is designed to deal with the class-imbalance problem of pseudo labels. And for each novel class, the threshold to select valuable pseudo labels is determined by quantile statistics of the confidence score distribution of pseudo labels. Furthermore, the pre-trained detector and the teacher model are associated with the preliminary CATF and in-depth CATF, respectively, and then the pseudo labels from the two-stream CATF are fused to provide supervisions. In this way, the knowledge of these two models is exploited, which improves the quality of pseudo labels. Under these supervisions, the student model is trained and the teacher model is correspondingly updated through parameters sharing, thus forming a positive feedback to improve the performance of both models. Besides, an attention module is integrated to the teacher and student models to enhance the feature representation of novel instances. The validations on PASCAL VOC and MS COCO show the effectiveness of the proposed method.

The high precision measurement of binocular camera depends on the calibration accuracy. Mainstream methods focus on the calibration of intrinsic parameters and the pose relationship of two cameras in sequence. However, the transmission of error will decrease the calibration quality. It is still challenging especially in the case of an unknown imaging model due to the intervention of baffle and multimedia refraction. In this article, a general calibration method for a binocular vision system with an unknown imaging model is proposed. Based on the one-to-one mapping between 4-D image coordinates and 3-D Cartesian coordinates of spatial points, the intersection line of two calibration planes is extracted. All calibration planes are organized in the form of a three-plane group, and in each group, the poses are solved by the intersection lines. These poses are further optimized to acquire accurate 3-D coordinates of points in calibration planes. Combined with the correspondences of image pixels and points in a calibration plane, the line-plane intersection point between the projection line and this plane is calculated. Finally, projection lines are fitted based on corresponding line-plane intersection points. By modeling the relationship between image pixels and object-end projection lines, the uncertainty of the imaging model is solved. The proposed method is universal and stable for different imaging models including refraction, and the results of simulation and experiment show its effectiveness.

The grasp detection is crucial to high-quality robotic grasping. Typically, the mainstream encoder-decoder regression solution is attractive due to its high accuracy and efficiency, however, it is still challenging to solve the checkerboard artifacts from the uneven overlap of convolution results in decoder, and features from the encoder also need to be further refined. In this paper, a novel pixel-wise grasp detection network is proposed, which is composed of an encoder, a multi-dimensional attention bottleneck, and a decoder based on twin deconvolution. The proposed decoder introduces a twin branch upon the original transposed convolution branch. Through the overlap degree matrix provided by the twin branch, the original branch is re-weighted and then the checkerboard artifacts of the original branch are eliminated. Besides, to deeply explore the intrinsic relationship of features and strengthen feature discrimination, residual multi-head self-attention, cross-amplitude attention, and channel attention are integrated together. As a result, adaptive feature refinement is achieved. The effectiveness of the proposed method is verified by experiments.

Dynamic object perception is an important yet challenging direction in the field of robot navigation. Without any prior knowledge about motion and objects, a novel dynamic rigid bodies mining and motion estimation method based on monocular camera is proposed in this article. Different from the existing works based on sampling that associate feature points to motion hypotheses according to the reprojection errors, our work endeavors to find the intrinsic relevance among motion hypotheses. To represent this relevance, the concept of the probabilistic field on the Lie group Sim(3) manifold is introduced, which is established using random sampling. It provides a computable way for the regions on the manifold where rigid bodies possibly appear. The probability of a motion hypothesis falling on a region is expressed by its confidence. The regions with large confidences in the probabilistic field are selected as potential rigid bodies, whose corresponding feature points are further sampled for pose calculation. As a result, the randomness of sampling is reduced and the inliers for possible rigid bodies are enhanced, which guarantees the accuracy of motion estimation. On this basis, the tracking of rigid bodies is achieved. The proposed method distinguishes the feature points of dynamic objects with 3-D motion from those in the static background, thus enabling simultaneous localization and mapping (SLAM) to be initialized in dynamic environments. The experimental results on the KITTI, Hopkins 155, and MTPV62 datasets demonstrate the effectiveness. Comparison experiments indicate that our method outperforms the other methods in sensitivity of dynamic objects perception.

In this brief, we define a self-limiting control term, which has the function of guaranteeing the boundedness of variables. Then, we apply it to a finite-time stability control problem. For nonstrict feedback nonlinear systems, a finite-time adaptive control scheme, which contains a piecewise differentiable function, is proposed. This scheme can eliminate the singularity of derivative of a fractional exponential function. By adding a self-limiting term to the controller and the virtual control law of each subsystem, the boundedness of the overall system state is guaranteed. Then the unknown continuous functions are estimated by neural networks (NNs). The output of the closed-loop system tracks the desired trajectory, and the tracking error converges to a small neighborhood of the equilibrium point in finite time. The theoretical results are illustrated by a simulation example.

Category-level 6D object pose estimation has gained popularity and it is still challenging due to the diversity of different instances within the same category. In this paper, a novel category-level 6D object pose estimation framework with structure encoder and reasoning attention is proposed. A structure autoencoder is introduced to mine the shared structure features in the color images within the same category, via a distinct learning strategy that recovers the image of another instance but with the most similar pose to the input. On this basis, a reasoning attention decoder and full connected layers are stacked to form a rotation prediction network, where the structure features and 3D shape features are integrated and projected to a semantic space. The semantic space includes observed patterns and learnable patterns, which are better learned by adding a shortcut connection branch parallel to reasoning attention decoder with gradient decouple. Further reasoning based on these patterns endows the decoder with powerful feature representation. Without 3D object models, the proposed method models the attributes of category implicitly in the semantic space and better performance of 6D object pose estimation is guaranteed by reasoning on this space. The effectiveness of the proposed method is verified by the results on public datasets and actual experiments.

The manipulating robots receive much attention by offering better services, where object grasping is still challenging especially under background interferences. In this article, a novel two-stream grasping convolutional neural network (CNN) with simultaneous detection and segmentation is proposed. The proposed method is cascaded by an improved simultaneous detection and segmentation network BlitzNet and a two-stream grasping CNN TsGNet. The improved BlitzNet introduces the channel-based attention mechanism, and achieves an improvement of detection accuracy and segmentation accuracy with the combination of the learning of multitask loss weightings and background suppression. Based on the obtained bounding box and the segmentation mask of the target object, the target object is separated from the background, and the corresponding depth map and grayscale map are sent to TsGNet. By adopting depthwise separable convolution and designed global deconvolution network, TsGNet achieves the best grasp detection with only a small amount of network parameters. This best grasp in the pixel coordinate system is converted to a desired 6-D pose for the robot, which drives the manipulator to execute grasping. The proposed method combines a grasping CNN with simultaneous detection and segmentation to achieve the best grasp with a good adaptability to background. With the Cornell grasping dataset, the image-wise accuracy and object-wise accuracy of the proposed TsGNet are 93.13% and 92.99%, respectively. The effectiveness of the proposed method is verified by the experiments.

The quadruped robot imitates the motions of four-legged animals with a superior flexibility and adaptability to complex terrains, compared with the wheeled and tracked robots. Its leader-following ability is unique to help a human to accomplish complex tasks in a more convenient way. However, long-term following is severely obstructed due to the high-frequency vibration of the quadruped robot and the unevenness of terrains. To solve this problem, a visual approach under a novel T-D-R framework is proposed. The proposed T-D-R framework is composed of a visual tracker based on correlation filter, a person detector with deep learning, and a person re-identification (re-ID) module. The result of the tracker is verified by the detector to improve tracking performance. Especially, the re-ID module is introduced to handle distractions and occlusion caused by other persons, where the convolutional correlation filter (CCF) is employed to discriminate the leader among multiple persons through recording the appearance information in the long run. By comparing the results of the tracker and the detector as well as their similarity scores with the leader identified by the re-ID module, a stable and real-time tracking of the leader can be guaranteed. Experiments reveal that our approach is effective in handling distractions, appearance changes, and illumination variations. A long-distance experiment on a quadruped robot indicates the validity of the proposed approach.

In this correspondence paper, an image dynamics-based visual servoing for quadrotors is proposed to realize stable hovering and tracking. Four perspective image moments are adopted as visual features to control all the independent degrees of freedom of a quadrotor. The complicated interaction matrix is simplified by projecting original image to virtual image plane. On this basis, the dynamics of the system is determined by considering the dynamics of image features and the quadrotor simultaneously. Backstepping controllers are then designed to stabilize the visual servoing system of the quadrotor. In reality, it is unrealistic to have exact prior knowledge about the trajectory parameters of an unpredictable moving target. To solve this problem, a trajectory observer based on nonlinear tracking-differentiator to estimate trajectory parameters of the target is firstly integrated into the quadrotor with image dynamics, which guarantees a satisfactory performance. The effectiveness of the proposed approach is verified by simulations.

In this paper, a novel method is proposed for curve detection in images using a feature-related searching control model. It is composed of three parts: 1) prediction; 2) searching; and 3) updating. First, curve related features are modeled to a three order array. Then, equations of the prediction, searching, and curve parameter updating are deduced. Third, an optimal model for curve parameter estimation during iterations is given. Based on the proposed model, a curve detection algorithm is designed. Experiments on thousands of images demonstrate the effectiveness and advantages of the proposed method. Comparison experiments with state-of-the-art methods show that the proposed method outperforms the existing methods on most indexes. Our method can describe the contents of original images more completely with fewer curves. The contour evaluation framework and the Berkeley segmentation dataset are used to evaluate the performances of different curve detection methods. The proposed method can also detect curves in the order relates to their importance, which has been validated in experiments.

This paper presents a navigation framework for mobile robots. In order to traverse in different environments, mobile robots have to avoid obstacles it encountered and make a suitable decision with the consideration of global destination. On the basis of grid map provided by the 3D LiDAR, this framework is related to the SLAMs with motion planning methods, where different SLAMs and different planning solutions can be combined to achieve the navigation in dynamic environments. Concretely, the ORB SLAM2 with a slightly improved VFH+ provides a solution, and a navigation method based on Bezier curve with Cartographer SLAM is presented, specifically, we consider the influence area caused by the dynamic obstacle with probabilistic descriptions. The effectiveness is verified by the experiments.

In this brief, sufficient conditions are proposed for the existence of the compact sets in the neural network controls. First, we point out that the existence of the compact set in a classical neural network control scheme is unsolved and its result is incomplete. Next, as a simple case, we derive the sufficient condition of the existence of the compact set for the neural network control of first-order systems. Finally, we propose the sufficient condition of the existence of the compact set for the neural-network-based backstepping control of high-order nonlinear systems. The theoretic result is illustrated through a simulation example.

This correspondence paper proposes a fast orientation estimation approach of natural images without the help of semantic information. Different from traditional low-level features, our low-level features are extracted inspired by the biological simple cells of the visual cortex. Two approximated receptive fields to mimic the biological cells are presented, and a local rotation operator is introduced to determine the optimal output and local orientation corresponding to an image position, which serve as the low-level feature employed in this paper. To generate the low-level features, a bisection method is applied to the first derivative of the model of receptive fields. Moreover, the feature screener is introduced to eliminate too much useless low-level features, which will speed up the processing time. After all the valuable low-level features are combined, the overall image orientation is estimated. The proposed approach possesses several features suitable for real-time applications. First, it avoids the tedious training procedure of some conventional methods. Second, no specific reference such as the horizon is assumed and no a priori knowledge of image is required. The proposed approach achieves a real-time orientation estimation of natural images using only low-level features with a satisfactory resolution. The effectiveness of our proposed approach is verified on real images with complex scenes and strong noises.

This paper proposes a novel general framework for line segment perception, which is motivated by a biological visual cortex, and requires no parameter tuning. In this framework, we design a model to approximate receptive fields of simple cells. More importantly, the structure of biological orientation columns is imitated by organizing artificial complex and hypercomplex cells with the same orientation into independent arrays. Besides, an interaction mechanism is implemented by a set of self-organization rules. Enlightened by the visual topological theory, the outputs of these artificial cells are integrated to generate line segments that can describe nonlocal structural information of images. Each line segment is evaluated quantitatively by its significance. The computation complexity is also analyzed. The proposed method is tested and compared to state-of-the-art algorithms on real images with complex scenes and strong noises. The experiments demonstrate that our method outperforms the existing methods in the balance between conciseness and completeness.

In this paper, the firing rate of the neuron based on temporal coincidence coding is estimated for correlation detection. Two cases are considered: the independent inputs are stochastic spike trains that are modeled by the homogeneous Poisson process or the renew process. The situation that the inputs are correlated is also considered and the conditions for the neuron to detect the correlation among inputs are discussed. The results are demonstrated by the simulations.

This paper proposes a kinematic modeling method for a bio-inspired robotic fish based on single joint. Lagrangian function of freely swimming robotic fish is built based on a simplified geometric model. In order to build the kinematic model, the fluid force acting on the robotic fish is divided into three parts: the pressure on links, the approach stream pressure and the frictional force. By solving Lagrange's equation of the second kind and the fluid force, the movement of robotic fish is obtained. The robotic fish's motion, such as propelling and turning are simulated, and experiments are taken to verify the model.

This paper proposes a new forward passageway (FP) based real-time collision-free target tracking approach for a mobile robot with local sensing. After the position of the target is estimated and localized in robot coordinate system through the combination of vision system and encoder, the sonar information and the target position are converted to a uniform environment model framework called decision-making space. Based on the space, a FP based decision-making is given to endow the robot with the ability to avoid possible obstacles and track the target in unknown environments. Experiment results show the validity of the proposed approach.

The swimming backward method for biomimetic carangiform robotic fish is analyzed in this paper based on the dynamic/kinematic model. The equation of Lagrange of multi-link carangiform robotic fish and simplified fluid force are inducted to calculate the dynamic and kinematic characteristics of the motions. A specific gait is calculated to make the profile of the carangiform robotic fishpsilas undulation fit the characteristics of European eelpsilas swimming backward, which is summarized from the motion sequence of European eel. The simulated and experimental data is given to verify the method.

With the development of multicast service in the Internet, much attention has been drawn to multicast congestion control and analysis. Multicast traffic poses new challenges to the design of Internet congestion control protocols and system stability analysis. The rate control problem of feedback-based sessions on the coexistence of both unicast and multirate multicast traffic architecture networks is focused upon in this paper. First, a fairness problem is discussed in detail, and a reasonable consumption strategy is proposed. In the reasonable consumption strategy, scaling functions are adaptively adjusted based on a relationship between the session rates. Second, contraposing the case that available link capacities are changing with time for these feedback-based unicast and multicast sessions, stability analysis of a closed-loop rate control system under the modified rate mechanism is made based on Lyapunov stable theory. Finally, the simulations illustrate the effectiveness and goodness of the reasonable consumption strategy

In this paper, a method for the posture control of a biomimetic robot fish TPF-I is proposed. In this method, the position of the robot fish's gravity centre can be changed by a barycenter-adjustor, which leads to the pitching angle changing. Propelled by coordinating a multi-link body and a tail, the robot fish can complete the posture control and 3-D Locomotion. The 3-D locomotion and posture control are implemented by synthesizing three basic control methods speed control, orientation control and pitching control, which are described in detail respectively. Finally, the experimental results of the robot fish's motion control are given and the performance is analyzed

This paper proposes a distributed control approach called local interactions with local coordinate systems (LILCS)to multirobot hunting tasks in unknown environments, where a team of mobile robots hunts a target called evader, which will actively try to escape with a safety strategy. This robust approach can cope with accumulative errors of wheels and imperfect communication networks. Computer simulations show the validity of the proposed approach.

This paper explores the application of the behavior-based approach to path planning for multiple mobile robots performing a formation control task in unknown environments. To predict the positions of moving obstacles for the purpose of collision avoidance, parabola prediction model whose parameters are estimated by the recurrence least square algorithm with restricted scale is adopted. Then, on the basis of the task and environment, we adopt five primitive behaviors and design a series of generation functions to generate control parameters for behaviors' combination. Furthermore, as the outputs of these functions can be adjusted according to the current situation, thus robots can achieve a motion strategy by reasonably combining behaviors and the adaptability to the environment is improved. We illustrate the validity of the approach by the simulations.