Generating 3D human motions from textual descriptions is an important research problem with broad applications in video games, virtual reality, and augmented reality. Recent methods align the textual description with human motion at the sequence level, neglecting the internal semantic structure of modalities. However, both motion descriptions and motion sequences can be naturally decomposed into smaller and semantically coherent segments, which can serve as atomic alignment units to achieve finer-grained correspondence. Motivated by this, we propose SegMo, a novel Segment-aligned text-conditioned human Motion generation framework to achieve fine-grained text–motion alignment. Our framework consists of three modules: (1) Text Segment Extraction, which decomposes complex textual descriptions into temporally ordered phrases, each representing a simple atomic action; (2) Motion Segment Extraction, which partitions complete motion sequences into corresponding motion segments; and (3) Fine-grained Text–Motion Alignment, which aligns text and motion segments with contrastive learning. Extensive experiments demonstrate that SegMo improves the strong baseline on two widely used datasets, achieving an improved TOP 1 score of 0.553 on the HumanML3D test set. Moreover, thanks to the learned shared embedding space for text and motion segments, SegMo can also be applied to retrieval-style tasks such as motion grounding and motion-to-text retrieval.

Infrared small target segmentation technology plays an important role in fields such as missile warning, maritime rescue, and military reconnaissance. However, CNN methods based on convolution tend to lose information regarding infrared small targets, resulting in poor segmentation performance. On the other hand, methods based on transformers, lacking convolution-induced biases, also struggle to achieve good results. To address this issue, this article proposes a model called Multiscale Feature Fusion Spatial-channel Attention Network (MFFSANet) for the segmentation of infrared small targets. The MFFSANet model consists of three blocks: the Multi-scale Convolution Fusion Attention (MCFA) block, the Hierarchical Guided Channel Attention (HGCA) block, and the Atrous Residual U-Block (ARU). The MCFA block leverages multi-scale atrous convolutions and self-attention mechanisms to obtain both local and global information about the image, learning the difference between target features and background noise features, thus enabling the model to suppress background noise in infrared images. The HGCA block leverages coarser information to guide the learning of finer features, assigning weights to decisive channels, and reducing redundant information. This reduces background noise in infrared images, making small targets stand out more clearly against the background. The ARU facilitates interaction between feature maps of different layers and scales, enabling the model to recognize the characteristics of small infrared targets in a more detailed and comprehensive manner. Extensive experiments conducted on four publicly available datasets, namely SIRST, IRSTD-1k, NUDT-SIRST, and SIRST-Aug, demonstrate the effectiveness and superiority of the proposed MFFSANet method compared to several SOTA infrared small target segmentation methods. The source code is available athttps://github.com/change68/MFFSANet.

In this paper, we present a sequential joint dependency aware model for monocular 2D-to-3D human pose estimation. While existing estimators leverage the (bi)directional joint dependency with graph convolutions and attention, we further propose to exploit the sequential dependency between joints with state space model (SSM). Our sequential dependency takes into consideration the information of kinematic chain, joint hierarchy and the body part. We design a sequential dependency aware representation to transform the pose data into sequential data for our pose SSM module. We tailor the SSM layer in the pose SSM module for pose estimation by learning joint-dependent parameters and introducing pose aware hidden state initialization. Extensive experiments are conducted on two datasets to validate the effectiveness of our proposed SSM module, and the results demonstrate that our pose estimator can deliver impressive performance.

Accurate 2D human pose estimation from images is vital for understanding human actions. However, deploying the latest models, e.g., regression-based models, on resource-limited devices remains challenging due to their high computational requirements. In this paper, we address the resolution dilemma in regression-based multiperson pose estimation, where low-resolution inputs cause performance degradation, while high-resolution inputs drastically increase computational costs. To achieve a lightweight regression approach, it becomes crucial to enhance the model's capabilities in low-resolution scenarios. We propose the staggered alignment self-distillation (SASD) method and a corresponding network architecture. Our approach involves training two twin networks with shared weights: a high-resolution network and a low-resolution network. The high-resolution network serves as a teacher, guiding the learning process of the low-resolution network through feature map staggered alignment. The knowledge from the high-resolution network enhances the performance of the low-resolution network during low-resolution inference. Additionally, we employ a normalized skeleton loss to capture the loss of bone-related structure during training. Through extensive experiments on the MS-COCO and CrowdPose datasets, we demonstrate the superiority of our proposed method over state-of-the-art, lightweight multiperson pose estimation techniques, achieving much better performance with lower computational costs. Furthermore, our method achieves comparable performance to recent advanced regression-based pose estimation methods but with only 1/4 of the computational cost.

3D pose transfer is a challenging generation task that aims to transfer the pose of a source geometry onto a target geometry with the target identity preserved. Many prior methods require keypoint annotations to find correspondence between the source and target. Current pose transfer methods allow end-to-end correspondence learning but require the desired final output as ground truth for supervision. Unsupervised methods have been proposed for graph convolutional models but they require ground truth correspondence between the source and target inputs. We present a novel self-supervised framework for 3D pose transfer which can be trained in unsupervised, semi-supervised, or fully supervised settings without any correspondence labels. We introduce two contrastive learning constraints in the latent space: a mesh-level loss for disentangling global patterns including pose and identity, and a point-level loss for discriminating local semantics. We demonstrate quantitatively and qualitatively that our method achieves state-of-the-art results in supervised 3D pose transfer, with comparable results in unsupervised and semi-supervised settings. Our method is also generalisable to unseen human and animal data with complex topologies†.

Visual anomaly detection methods with localization are critically important for industrial manufacturing quality control. Because of the rarity of anomalies and the irregular variation of anomaly patterns, unsupervised methods have been widely explored. For the anomaly detection and localization tasks, the image reconstruction-based approaches have shown competitive performance. However, reconstruction results of those methods are coarse and visually blurred, which leads to a high rate of false detection and false pixel-level localization. To address those issues, we propose a framework called Dual-Path Re-construction Guided Segmentation Network (DRGS-Net), which determines the abnormal regions by segmenting the anomaly image with its reconstructed result as the reference template. DRGS-Net mainly consists of a novel dual-path reconstruction sub-network and a specifically designed anomaly segmentation sub-network. They are jointly trained end-to-end, with the former fusing texture repair and image reconstruction information to obtain fine-grained reconstruction results and the latter learning a decision boundary between normal and anomalous regions based on reconstruction results. On the standard benchmark dataset MVTecAD and an additional dataset DAGM, DRGS-Net shows competitive performance in image-level detection and achieves outstanding improvement in pixel-level localization. Further experiments with the few samples setting demonstrate that DRGS-Net retains strong performance with only few anomaly-free training images.

In autonomous driving, monocular 3D object detection is an important but challenging task. Towards accurate monocular 3D object detection, some recent methods recover the distance of objects from the physical height and visual height of objects. Such decomposition framework can introduce explicit constraints on the distance prediction, thus improving its accuracy and robustness. However, the inaccurate physical height and visual height prediction still may exacerbate the inaccuracy of the distance prediction. In this paper, we improve the framework by multivariate probabilistic modeling. We explicitly model the joint probability distribution of the physical height and visual height. This is achieved by learning a full covariance matrix of the physical height and visual height during training, with the guide of a multivariate likelihood. Such explicit joint probability distribution modeling not only leads to robust distance prediction when both the predicted physical height and visual height are inaccurate, but also brings learned covariance matrices with expected behaviors. The experimental results on the challenging Waymo Open and KITTI datasets show the effectiveness of our framework1.

Unsupervised Domain Adaptation (UDA) can transfer knowledge from labeled source data to unlabeled target data of the same categories. However, UDA for first-person video action recognition is an under-explored problem, with a lack of benchmark datasets and limited consideration of first-person video characteristics. Existing benchmark datasets provide videos with a single activity scene, e.g. kitchen, and similar global video statistics. However, multiple activity scenes and different global video statistics are still essential for developing robust UDA networks for real-world applications. To this end, we first introduce two first-person video domain adaptation datasets: ADL-7 and GTEA_KITCHEN-6. To the best of our knowledge, they are the first to provide multi-scene and cross-site settings for UDA problem on first-person video action recognition, promoting diversity. They provide five more domains based on the original three from existing datasets, enriching data for this area. They are also compatible with existing datasets, ensuring scalability. First-person videos have unique challenges, i.e. actions tend to occur in hand-object interaction areas. Therefore, networks paying more attention to such areas can benefit common feature learning in UDA. Attention mechanisms can endow networks with the ability to allocate resources adaptively for the important parts of the inputs and fade out the rest. Hence, we introduce channel-temporal attention modules to capture the channel-wise and temporal-wise relationships and model their inter-dependencies important to this characteristic. Moreover, we propose a Channel-Temporal Attention Network (CTAN) to integrate these modules into existing architectures. CTAN outperforms baselines on the new datasets and one existing dataset, EPIC-8.

3D morphable models are widely used for the shape representation of an object class in computer vision and graphics applications. In this work, we focus on deep 3D morphable models that directly apply deep learning on 3D mesh data with a hierarchical structure to capture information at multiple scales. While great efforts have been made to design the convolution operator, how to best aggregate vertex features across hierarchical levels deserves further attention. In contrast to resorting to mesh decimation, we propose an attention based module to learn mapping matrices for better feature aggregation across hierarchical levels. Specifically, the mapping matrices are generated by a compatibility function of the keys and queries. The keys and queries are trainable variables, learned by optimizing the target objective, and shared by all data samples of the same object class. Our proposed module can be used as a train-only drop-in replacement for the feature aggregation in existing architectures for both downsampling and upsampling. Our experiments show that through the end-to-end training of the mapping matrices, we achieve state-of-the-art results on a variety of 3D shape datasets in comparison to existing morphable models.

Monocular 3D object detection is of great significance for autonomous driving but remains challenging. The core challenge is to predict the distance of objects in the absence of explicit depth information. Unlike regressing the distance as a single variable in most existing methods, we propose a novel geometry-based distance decomposition to recover the distance by its factors. The decomposition factors the distance of objects into the most representative and stable variables, i.e. the physical height and the projected visual height in the image plane. Moreover, the decomposition maintains the self-consistency between the two heights, leading to robust distance prediction when both predicted heights are inaccurate. The decomposition also enables us to trace the causes of the distance uncertainty for different scenarios. Such decomposition makes the distance prediction interpretable, accurate, and robust. Our method directly predicts 3D bounding boxes from RGB images with a compact architecture, making the training and inference simple and efficient. The experimental results show that our method achieves the state-of-the-art performance on the monocular 3D Object Detection and Bird’s Eye View tasks of the KITTI dataset, and can generalize to images with different camera intrinsics1.

Most existing unsupervised video hashing methods generate binary codes by using RNNs in a deterministic manner, which fails to capture the dominant latent variation of videos. In addition, RNN-based video hashing methods suffer the content forgetting of early input frames due to the sequential processing inherency of RNNs, which is detrimental to global information capturing. In this work, we propose an unsupervised variational video hashing (UVVH) method for scalable video retrieval. Our UVVH method aims to capture the salient and global information in a video. Specifically, we introduce a variational autoencoder to learn a probabilistic latent representation of the salient factors of video variations. To better exploit the global information of videos, we design a 1D-CNN-LSTM model. The 1D-CNN-LSTM model processes long frame sequences in a parallel and hierarchical way, and exploits the correlations between frames to reconstruct the frame-level features. As a consequence, the learned hash functions can produce reliable binary codes for video retrieval. We conduct extensive experiments on three widely used benchmark datasets, FCVID, ActivityNet and YFCC to validate the effectiveness of our proposed approach.

In this paper, we propose a Neighborhood Preserving Hashing (NPH) method for scalable video retrieval in an unsupervised manner. Unlike most existing deep video hashing methods which indiscriminately compress an entire video into a binary code, we embed the spatial-temporal neighborhood information into the encoding network such that the neighborhood-relevant visual content of a video can be preferentially encoded into a binary code under the guidance of the neighborhood information. Specifically, we propose a neighborhood attention mechanism which focuses on partial useful content of each input frame conditioned on the neighborhood information. We then integrate the neighborhood attention mechanism into an RNN-based reconstruction scheme to encourage the binary codes to capture the spatial-temporal structure in a video which is consistent with that in the neighborhood. As a consequence, the learned hashing functions can map similar videos to similar binary codes. Extensive experiments on three widely-used benchmark datasets validate the effectiveness of our proposed approach.

This paper presents a discrepancy minimizing model to address the discrete optimization problem in hashing learning. The discrete optimization introduced by binary constraint is an NP-hard mixed integer programming problem. It is usually addressed by relaxing the binary variables into continuous variables to adapt to the gradient based learning of hashing functions, especially the training of deep neural networks. To deal with the objective discrepancy caused by relaxation, we transform the original binary optimization into differentiable optimization problem over hash functions through series expansion. This transformation decouples the binary constraint and the similarity preserving hashing function optimization. The transformed objective is optimized in a tractable alternating optimization framework with gradual discrepancy minimization. Extensive experimental results on three benchmark datasets validate the efficacy of the proposed discrepancy minimizing hashing.

In this paper, we present a deep hashing method for multi-label image retrieval, which uses a rank list to provide global supervision information. Unlike most existing approaches using shallow models to learn hash functions for multi-label images, we deepen the DNN structure to extract powerful features. In addition, we apply a rank-consistency objective function to align the similarity orders in the hamming space and the ones from the original space. Compared with conventional contrast loss and triplet loss, our listwise ranking can capture sufficient global information. We also propose a multi-label softmax cross-entropy loss to strengthen the discriminative power. Specifically, we consider the number of common labels of multi-label images as the metric of similarity in the original space with the hamming distance between binary codes as the distance metric in the hamming space. Experimental results on MIRFLICKR-25K and IAPRTC12 are presented to show the effectiveness of our proposed approach.

In this paper, we propose a nonlinear structural hashing approach to learn compact binary codes for scalable video search. Unlike most existing video hashing methods which consider image frames within a video separately for binary code learning, we develop a multi-layer neural network to learn compact and discriminative binary codes by exploiting both the structural information between different frames within a video and the nonlinear relationship between video samples. To be specific, we learn these binary codes under two different constraints at the output of our network: 1) the distance between the learned binary codes for frames within the same scene is minimized and 2) the distance between the learned binary matrices for a video pair with the same label is less than a threshold and that for a video pair with different labels is larger than a threshold. To better measure the structural information of the scenes from videos, we employ a subspace clustering method to cluster frames into different scenes. Moreover, we design multiple hierarchical nonlinear transformations to preserve the nonlinear relationship between videos. Experimental results on three video data sets show that our method outperforms state-of-the-art hashing approaches on the scalable video search task.

In this paper, we propose a reconstruction-based supervised hashing (RSH) method to learn compact binary codes with holistic structure preservation for large scale image search. Unlike most existing hashing methods which consider pair-wise similarity, our method exploits the structural information of samples by employing a reconstruction-based criterion. Moreover, the label information of samples is also utilized to enhance the discriminative power of the teamed hash codes. Specifically, our method minimizes the distance between each point and the selected generated-structure with the same class label and maximizes the distance between each point and the selected generated-structure with different class labels. Experimental results on two widely used image datasets demonstrate the effectiveness of the proposed method.

In this paper, we propose a nonlinear discrete hashing approach to learn compact binary codes for scalable image search. Instead of seeking a single linear projection in most existing hashing methods, we pursue a multilayer network with nonlinear transformations to capture the local structure of data samples. Unlike most existing hashing methods that adopt an error-prone relaxation to learn the transformations, we directly solve the discrete optimization problem to eliminate the quantization error accumulation. Specifically, to leverage the similarity relationships between data samples and exploit the semantic affinities of manual labels, the binary codes are learned with the objective to: 1) minimize the quantization error between the original data samples and the learned binary codes; 2) preserve the similarity relationships in the learned binary codes; 3) maximize the information content with independent bits; and 4) maximize the accuracy of the predicted labels based on the binary codes. With an alternating optimization, the nonlinear transformation and the discrete quantization are jointly optimized in the hashing learning framework. Experimental results on four datasets including CIFAR10, MNIST, SUN397, and ILSVRC2012 demonstrate that the proposed approach is superior to several state-of-the-art hashing methods.

To facilitate fast similarity search, this paper proposes to encode the nonlinear similarity and image structure as compact binary codes. Rather than adopting single matrix as projection in the literature, we employ a nonlinear transformation in the form of multilayer neural network to generate binary codes to capture the local structure between data samples. Specifically, we train the network such that the quantization loss is minimized and the variance over all bits is maximized. In addition, we capture the salient structure of image samples at the abstract level with sparsity constraint and inherit the generalization power to unseen samples. Furthermore, we incorporate the supervisory label information into the learning procedure to take advantage of the manual label. To obtain the desired binary codes and the parameterized nonlinear transformation, we optimize the formulated objective problem over each variable with an iterative alternating method. To validate the efficacy of the proposed hashing approach, we conduct experiments on three widely used datasets, namely CIFAR10, MNIST, and SUN397, by comparing with several recent proposed hashing methods.

Multi-core homogeneous processors have been widely used to deal with computation-intensive embedded applications. However, with the continuous down scaling of CMOS technology, within-die variations in the manufacturing process lead to a significant spread in the operating speeds of cores within homogeneous multi-core processors. Task scheduling approaches, which do not consider such heterogeneity caused by within-die variations, can lead to an overly pessimistic result in terms of performance. To realize an optimal performance according to the actual maximum clock frequencies at which cores can run, we present a heterogeneity-aware schedule refining (HASR) scheme by fully exploiting the heterogeneities of homogeneous multi-core processors in embedded domains. We analyze and show how the actual maximum frequencies of cores are used to guide the scheduling. In the scheme, representative chip operating points are selected and the corresponding optimal schedules are generated as candidate schedules. During the booting of each chip, according to the actual maximum clock frequencies of cores, one of the candidate schedules is bound to the chip to maximize the performance. A set of applications are designed to evaluate the proposed scheme. Experimental results show that the proposed scheme can improve the performance by an average value of 22.2%, compared with the baseline schedule based on the worst case timing analysis. Compared with the conventional task scheduling approach based on the actual maximum clock frequencies, the proposed scheme also improves the performance by up to 12%.