Despite the recent success of 3D action recognition using depth sensor, most existing works target how to improve the action recognition performance, rather than understanding how different types of actions are performed. In this paper, we propose to discover discriminative spatio-temporal patterns for 3D action recognition. Discovering these patterns can not only help to improve the action recognition performance but also help us to understand and differentiate between the action category. Our proposed method takes the spatio-temporal structure of 3D action into consideration and can discover essential spatio-temporal patterns that play key roles in action recognition. Instead of relying on an end-to-end network to learn the 3D action representation and perform classification, we simply present each 3D action as a series of temporal stages composed by 3D poses. Then, we rely on nearest neighbor matching and bilinear classifiers to simultaneously identify both critical temporal stages and spatial joints for each action class. Despite using raw action representation and a linear classifier, experiments on five benchmark data sets show that the proposed spatio-temporal naïve Bayes mutual information maximization can achieve a competitive performance compared with the state-of-the-art methods that use sophisticated end-to-end learning, and has the advantage of finding discriminative spatio-temporal action patterns.

Motivated by previous success of using non-parametric methods to recognize objects, e.g., NBNN [2], we extend it to recognize actions using skeletons. Each 3D action is presented by a sequence of 3D poses. Similar to NBNN, our proposed Spatio-Temporal-NBNN applies stage-to-class distance to classify actions. However, ST-NBNN takes the spatio-temporal structure of 3D actions into consideration and relaxes the Naive Bayes assumption of NBNN. Specifically, ST-NBNN adopts bilinear classifiers [19] to identify both key temporal stages as well as spatial joints for action classification. Although only using a linear classifier, experiments on three benchmark datasets show that by combining the strength of both non-parametric and parametric models, ST-NBNN can achieve competitive performance compared with state-of-the-art results using sophisticated models such as deep learning. Moreover, by identifying key skeleton joints and temporal stages for each action class, our ST-NBNN can capture the essential spatio-temporal patterns that play key roles of recognizing actions, which is not always achievable by using end-to-end models.

We propose a novel formulation to find representatives in data samples via learning with structured sparsity . To find representatives with both diversity and representativeness , we formulate the problem as a structurally-regularized learning where the objective function consists of a reconstruction error and three structured regularizers: (1) group sparsity regularizer, (2) diversity regularizer, and (3) locality-sensitivity regularizer. For the optimization of the objective, we propose an accelerated proximal gradient algorithm, combined with the proximal-Dykstra method and the calculation of parametric maximum flows. Experiments on image and video data validate the effectiveness of our method in finding exemplars with diversity and representativeness and demonstrate its robustness to outliers.

Scene image is a spatial composition of objects and background contexts and finding discriminative spatial layouts is critical for scene recognition. In this letter, we propose an ℓ1-regularized max-margin formulation to discover class-specific spatial layouts by jointly learning the image classifier and the class-specific spatial layouts for scene recognition. Unlike previous methods that classify images into different categories either without considering the spatial layouts explicitly or only using class generic spatial layout, our proposed method can discover a sparse combination of class-specific spatial layouts for different scenes and boost the recognition performance. Experiments on scene-15, landuse-21, and MIT indoor-67 datasets validate the advantages of our proposed algorithm.

This paper is concerned with a system for detecting and tracking multiple 3D bounding boxes based on information from multiple sensors. Our framework is built around an inference engine similar to the probability hypothesis density (PHD) filter, where the state space consists of stochastic bounding boxes with constant velocity dynamics. We outline measurement equations for two modalities (vision and radar). The result is a flexible inference system suitable for use on autonomous vehicles.

The co-occurrence features are the composition of base features that have more discriminative power than individual base features. Although they show promising performance in visual recognition applications such as object, scene, and action recognition, the discovery of optimal co-occurrence features is usually a computationally demanding task. Unlike previous feature mining methods that fix the order of the co-occurrence features or rely on a two-stage frequent pattern mining to select the optimal co-occurrence feature, we propose a novel branch-and-bound search-based co-occurrence feature mining algorithm that can directly mine both optimal conjunctions (AND) and disjunctions (OR) of individual features at arbitrary orders simultaneously. This feature mining process is integrated into a multi-class boosting framework Adaboost.MH such that the weighted training error is minimized by the discovered co- occurrence features in each boosting step. Experiments on UCI benchmark datasets, the scene recognition dataset, and the action recognition dataset validate both the effectiveness and efficiency of our proposed method.

In this paper, we propose a novel formulation for multi-feature clustering using minimax optimization. To find a consensus clustering result that is agreeable to all feature modalities, our objective is to find a universal feature embedding, which not only fits each individual feature modality well, but also unifies different feature modalities by minimizing their pairwise disagreements. The loss function consists of both (1) unary embedding cost for each modality, and (2) pairwise disagreement cost for each pair of modalities, with weighting parameters automatically selected to maximize the loss. By performing minimax optimization, we can minimize the loss for the worst case with maximum disagreements, thus can better reconcile different feature modalities. To solve the minimax optimization, an iterative solution is proposed to update the universal embedding, individual embedding, and fusion weights, separately. Our minimax optimization has only one global parameter. The superior results on various multi-feature clustering tasks validate the effectiveness of our approach when compared with the state-of-the-art methods.

Local feature extraction, coding, spatial pooling, and image classification are the four typical steps for state-of-the-art visual recognition systems. Unlike previous work that treats spatial pooling and image classification as separated steps, we propose to jointly learn the geometric pooling and image classifier such that class-specific geometric information of local descriptors can be incorporated to improve classification performance. Inspired by previous work of spatial pyramid matching and receptive field learning, we also propose spatial pyramid geometric pooling, receptive field geometric pooling and random partition geometric pooling approaches to further exploit the spatial structural information to boost classification performance. Experiments on 15-scene dataset validate the advantages of our proposed algorithms.

The co-occurrence features are the composition of base features that have more discriminative power than individual base features. Although they show promising performance in visual recognition applications such as object and scene recognition, the discovery of discriminative co-occurrence features is usually a computational demanding task. Unlike previous feature mining methods that fix the order of the co-occurrence features or rely on a two-stage frequent pattern mining to select the optimal feature co-occurrence, we propose a novel branch-and-bound based co-occurrence feature mining algorithm that can directly mine both optimal conjunctions (AND) and disjunctions (OR) of individual features at arbitrary orders simultaneously. This feature mining process is integrated into a multi-class boosting framework Adaboost. MH such that the weighted error is minimized by the discovered co-occurrence features in each boosting step. Experiments on the benchmark datasets and scene recognition dataset validate the advantages of our proposed method.

We propose a novel hierarchical sparse coding algorithm with spatial pooling and multi-feature fusion, to construct the low-level visual primitives, e.g., local image patches or regions, into high-level visual phrases, e.g., image patterns. In the first layer we learn the sparse codes for the visual primitives and then pass them into the second layer by spatial pooling and multi-feature fusion. In the second layer we further learn the sparse codes for the visual phrases. In order to obtain the high-quality representations for visual phrases, our proposed algorithm iteratively optimizes over the two-layer sparse codes, as well as the two-layer codebooks. Since we have explored both the spatial and multi-feature contextual information, more representative sparse codes of the visual phrases can be obtained. The experiments on image pattern discovery, image scene clustering and image classification justify the advantages of the proposed algorithm.