While gait recognition based on Large Vision Models (LVMs) benefits from powerful spatial representations, existing paradigms often reduce video sequences to unordered image sets, inherently neglecting the inter-frame temporal causality. To bridge the modal gap between static image perception and dynamic video understanding, we propose the Layer-wise Spatio-Temporal Activation (LTA) framework and the LTA-Gait model. Leveraging the linear complexity of State Space Models (SSMs), our method alleviates the efficiency challenge of introducing temporal modeling into deep LVMs, enabling full-level dense temporal adaptation in a practical manner. Addressing the challenges of recursive noise sensitivity and spatio-temporal granularity misalignment specific to gait video, we design an Anatomy-Aware Temporal Adapter (AATA). By introducing an input-output synergistic dual-gating mechanism, we apply a binary mask-based input constraint to suppress background noise before temporal recursion, while using adaptive soft gating at the output to stabilize feature injection during fine-tuning. Furthermore, we construct a progressive temporal receptive field strategy to better match temporal modeling granularity with the hierarchical spatial features of the LVM. Extensive experiments demonstrate that LTA-Gait achieves State-of-the-Art (SOTA) performance on benchmarks such as CCPG, CCGR, and CASIA-B*. Notably, significant performance gains in challenging clothing-change scenarios validate the core value of robust ordered modeling in extracting intrinsic gait dynamics. All implementation code is fully available at our project repository: https://github.com/zjucgl/LTA-Gait.

As a crucial research direction in the field of computer vision, fine-grained visual classification (FGVC) aims to distinguish different subcategories belonging to the same base category. Its core challenge lies in effectively addressing subtle inter-class differences and significant intra-class differences. Existing methods usually have shortcomings in the collaboration among detail preservation, feature selection, and long-range dependency modeling. To this end, this paper proposes a novel multi-branch collaborative network named MBCNet. This network adopts a three-branch structure design: The backbone path integrates a hierarchical separated attention mechanism, which applies spatial attention and channel attention in the shallow and deep layers respectively to achieve accurate feature selection; the gating branch uses LSTM units to model spatial long-range dependencies; the detail preservation branch retains high-frequency texture features through the combination of dilated convolutions. Through an innovative forward-backward cross-attention mechanism, the model realizes bidirectional information interaction between the gating branch and the backbone path, forming interactive feedback of spatial features and temporal features. Experiments on multiple standard finegrained datasets and a self-built pig face recognition dataset demonstrate that MBCNet significantly improves classification accuracy while maintaining a reasonable computational complexity, which effectively verifies the superiority of the proposed method.

Microsoft's Kinect sensors are receiving an increasing amount of interests by security researchers since they are cost-effective and can provide both visual and depth modality data at the same time. Unfortunately, depth or RGB modalities are unavailable in training or testing procedures in some realistic scenarios. Therefore, we explore a new problem focusing on the arbitrary absence of modality, which is completely different from the conventional action recognition. The new problem in action recognition aims to deal with cross modality data (e.g., RGB training and depth testing data), “missing” modality data (e.g., RGB training and RGB-D test data), and single-modality data (e.g., RGB/depth in both phases). Accordingly, our method aims to borrow some information (e.g., correlation between two modalities) from the well-established RGB-D dataset and apply it to the existing dataset to recover some latent information to improve the performance of recognition. For instance, a cross-modality regularizer is used to preserve the correlation of RGB and depth modalities. The “missing” knowledge is considered as latent information, which is recovered by low-rank learning in our model. In the real world, the target data are usually sparsely labeled or completely unlabeled; however, we could exploit the pseudolabels of the target as prior knowledge for “supervised” learning in the target domain. Accordingly, we propose a semi-supervised model for transfer learning. The experiments on three widely used RGB-D action datasets show that our method performs better than that of the state-of-the-art transfer learning methods in most cases in terms of accuracy and time efficiency.

Style classification (e.g., Baroque and Gothic architecture style) is grabbing increasing attention in many fields such as fashion, architecture, and manga. Most existing methods focus on extracting discriminative features from local patches or patterns. However, the spread out phenomenon in style classification has not been recognized yet. It means that visually less representative images in a style class are usually very diverse and easily getting misclassified. We name them weak style images. Another issue when employing multiple visual features towards effective weak style classification is lack of consensus among different features. That is, weights for different visual features in the local patch should have been allocated similar values. To address these issues, we propose a Consensus Style Centralizing Auto-Encoder (CSCAE) for learning robust style features representation, especially for weak style classification. First, we propose a Style Centralizing Auto-Encoder (SCAE) which centralizes weak style features in a progressive way. Then, based on SCAE, we propose both the non-linear and linear version CSCAE which adaptively allocate weights for different features during the progressive centralization process. Consensus constraints are added based on the assumption that the weights of different features of the same patch should be similar. Specifically, the proposed linear counterpart of CSCAE motivated by the "shared weights" idea as well as group sparsity improves both efficacy and efficiency. For evaluations, we experiment extensively on fashion, manga and architecture style classification problems. In addition, we collect a new dataset-Online Shopping, for fashion style classification, which will be publicly available for vision based fashion style research. Experiments demonstrate the effectiveness of the SCAE and CSCAE on both public and newly collected datasets when compared with the most recent state-of-the-art works.

Spatially or temporally corrupted action videos are impractical for recognition via vision or learning models. It usually happens when streaming data are captured from unintended moving cameras, which bring occlusion or camera vibration and accordingly result in arbitrary loss of spatiotemporal information. In reality, it is intractable to deal with both spatial and temporal corruptions at the same time. In this paper, we propose a coupled stacked denoising tensor auto-encoder (CSDTAE) model, which approaches this corruption problem in a divide-and-conquer fashion by jointing both the spatial and temporal schemes together. In particular, each scheme is a SDTAE designed to handle either spatial or temporal corruption, respectively. SDTAE is composed of several blocks, each of which is a denoising tensor auto-encoder (DTAE). Therefore, CSDTAE is designed based on several DTAE building blocks to solve the spatiotemporal corruption problem simultaneously. In one DTAE, the video features are represented as a high-order tensor to preserve the spatiotemporal structure of data, where the temporal and spatial information are processed separately in different hidden layers via tensor unfolding. In summary, DTAE explores the spatial and temporal structure of the tensor representation, and SDTAE handles different corrupted ratios progressively to extract more discriminative features. CSDTAE couples the temporal and spatial corruptions of the same data through a thorough step-by-step procedure based on canonical correlation analysis, which integrates the two sub-problems into one problem. The key point is solving the spatiotemporal corruption in one model by considering them as noises in either spatial or temporal direction. Extensive experiments on three action data sets demonstrate the effectiveness of our model, especially when large volumes of corruption in the video.

In this paper we solve three problems in action recognition: sub-action, multi-subject, and multi-modality, by reducing the diversity of intra-class samples. The main stage contains canonical temporal alignment and key frames selection. As we know, temporal alignment aims to reduce the diversity of intra-class samples, however, dense frames may yield misalignment or overlapped alignment and decrease recognition performance. To overcome this problem, we propose a Sparse Canonical Temporal Alignment (SCTA) method which selects and aligns key frames from two sequences to reduce diversity. To extract better features from the key frames, we propose a Deep Non-negative Tensor Factorization (DNTF) method to find a tensor subspace integrated with SCTA scheme. First we model an action sequence as a third-order tensor with spatiotemporal structure. Then we design a DNTF scheme to find a tensor subspace in both spatial and temporal directions. Particularly, in the first layer the original tensor is decomposed into two lowrank tensors by Non-negative Tensor Factorization (NTF), and in the second layer each low-rank tensor is further decomposed by Tensor-Train (TT) for time efficiency. Finally, our framework composed of SCTA and DNTF could solve the three problems and extract effective features for action recognition. Experiments on synthetic data, MSRDailyActivity3D and MSRActionPairs datasets show that our method works better than competitive methods in terms of accuracy.

Style classification (e.g., architectural, music, fashion) attracts an increasing attention in both research and industrial fields. Most existing works focused on low-level visual features composition for style representation. However, little effort has been devoted to automatic mid-level or high-level style features learning by reorganizing low-level descriptors. Moreover, styles are usually spread out and not easy to differentiate from one to another. In this paper, we call these less representative images as weak style images. To address these issues, we propose a consensus style centralizing auto-encoder (CSCAE) to extract robust style features to facilitate weak style classification. CSCAE is the ensemble of several style centralizing auto-encoders (SCAEs) with consensus constraint. Each SCAE centralizes each feature of certain category in a progressive way. We apply our method in fashion style classification and manga style classification as two example applications. In addition, we collect a new dataset, Online Shopping, for fashion style classification evaluation, which will be publicly available for vision based fashion style research. Experiments demonstrate the effectiveness of SCAE and CSCAE on both public and newly collected datasets when compared with the most recent state-of-the-art works.

RGB-D action streams have aroused impressive attentions for recognition task, for its geometric characteristic and less influence of illumination. However, there exists large divergences of intra-class actions performed between sub-action, multi-subject and multi-modality, which may affect the result of action recognition. In order to solve these three problems, we propose a Sparse alignment guided Non-negative Tensor Factorization (SaNTF) framework with non-negative tensor factorization (NTF) for subspace learning. SaNTF selects the key frames from two intra-class action sequences by sparse regression, and aligns them to mitigate the diversity in the new tensor subspace. In this paper, the high-dimensional RGB-D action sequence is represented as a third-order tensor to preserve the original spatiotemporal structure, and we employ NTF to find a common tensor subspace for realistic action recognition. The experiments on MSRDailyActivity3D action and MSRPair3D action datasets show the higher accuracy compared with the state-of-the-art temporal alignment methods.

RGB-D action data inherently equip with extra depth information to improve performance of action recognition compared with RGB data, and many works represent the RGB-D data as a third-order tensor containing a spatiotemporal structure and find a subspace with lower dimension. However, there are two main challenges of these methods. First, the dimension of subspace is usually fixed manually, which may not describe the samples well in the subspace. Second, preserving local information by finding the intra-class and inter-class neighbors from a manifold is highly time-consuming. In this paper, we learn a tensor subspace, whose dimension is learned automatically by low-rank learning, for RGB-D action recognition. Particularly, the tensor samples are factorized to obtain three projection matrices (PMs) by Tucker Decomposition, where all the PMs are performed by nuclear norm in a close-form to obtain the tensor ranks, which are used as tensor subspace dimension. In addition, we extract the discriminant and local information from a manifold using a graph constraint. This graph preserves the local knowledge inherently, which is faster than the previous way of calculating both the intra-class and inter-class neighbors of each sample. We evaluate the proposed method on four widely used RGB-D action datasets including MSRDailyActivity3D, MSRActionPairs, MSRActionPairs skeleton, and UTKinect-Action3D datasets, and the experimental results show higher accuracy and efficiency of the proposed method.

Tensor completion is an important topic in the area of image processing and computer vision research, which is generally built on extraction of the intrinsic structure of the tensor data. Drawing on this fact, action classification, relying heavily on the extracted features of high-dimensional tensors, may indeed benefit from tensor completion techniques. In this paper, we propose a low-rank tensor completion method for action classification, as well as image recovery. Since there may exist distortion and corruption in the tensor representations of video sequences, we project the tensors into a subspace, which contains the invariant structure of the tensors. In order to integrate useful supervisory information for classification, we adopt a discriminant analysis criterion to learn the projection matrices. The resulting multi-variate optimization problem can be effectively solved using the augmented Lagrange multiplier (ALM) algorithm. Experiments demonstrate that our method results with better accuracy compared with some other state-of-the-art low-rank tensor representation learning approaches on the MSR Hand Gesture 3D database and the MSR Action 3D database. By denoising the Multi-PIE face database, our experimental setup testifies the proposed method can also be employed to recover images.

This paper proposes a method to compensate RGB-D images from the original target RGB images by transferring the depth knowledge of source data. Conventional RGB databases (e.g., UT-Interaction database) do not contain depth information since they are captured by the RGB cameras. Therefore, the methods designed for {RGB} databases cannot take advantage of depth information, which proves useful for simplifying intra-class variations and background subtraction. In this paper, we present a novel transfer learning method that can transfer the knowledge from depth information to the RGB database, and use the additional source information to recognize human actions in RGB videos. Our method takes full advantage of 3D geometric information contained within the learned depth data, thus, can further improve action recognition performance. We treat action data as a fourth-order tensor (row, column, frame and sample), and apply latent low-rank transfer learning to learn shared subspaces of the source and target databases. Moreover, we introduce a novel cross-modality regularizer that plays an important role in finding the correlation between RGB and depth modalities, and then more depth information from the source database can be transferred to that of the target. Our method is extensively evaluated on public by available databases. Results of two action datasets show that our method outperforms existing methods.

This letter addresses the problem of analyzing spatio-temporal patterns for action recognition. In this letter we organize the whole training set in a single tensor, with each mode indicating one factor which influences the result of recognition, e.g., various view points. A novel method is proposed for tensor decomposition by discriminant analysis of multiscale features which represent the motion details on different scales. In addition, the nearest neighbor classifier (NNC) is employed for action classification. Experiments on the self-manufactured action database under ideal conditions showed that the proposed method was better than state-of-the-art methods under various view angles in terms of accuracy. Experiments on the commonly used KTH database also showed that the proposed method had low time complexity and was robust against changing view points.