Reconstructing pedestrian motion from dynamic sensors, with a focus on pedestrian intention, is crucial for advancing autonomous driving safety. However, this task is challenging due to data limitations arising from technical complexities, safety, and cost concerns. We introduce the Pedestrian Motion Reconstruction (PMR) dataset, which focuses on pedestrian intention to reconstruct behavior using multiple perspectives and modalities. PMR is developed from a mixed reality platform that combines real-world realism with the extensive, accurate labels of simulations, thereby reducing costs and risks. It captures the intricate dynamics of pedestrian interactions with objects and vehicles, using different modalities for a comprehensive understanding of human-vehicle interaction. Analyses show that PMR can naturally exhibit pedestrian intent and simulate extreme cases. PMR features a vast collection of data from 54 subjects interacting across 12 urban settings with 7 objects, encompassing 12,138 sequences with diverse weather conditions and vehicle speeds. This data provides a rich foundation for modeling pedestrian intent through multi-view and multi-modal insights. We also conduct comprehensive benchmark assessments across different modalities to thoroughly evaluate pedestrian motion reconstruction methods.

This paper presents an online human model-based framework for gait-based age and gender estimation from a sequence of monocular frames. More specifically, we fine-tune a human mesh recovery model (i.e., HMR) to estimate the shape and pose parameters of a predefined 3D human model (i.e., SMPL). We then utilize the estimated parameters to predict the age and gender of the walking subject. To make the age and gender estimation task more favorable for real-time applications, we consider estimating the corresponding probability distributions of age and gender, which preserve the prediction uncertainty. Experiments on the world’s largest multi-view gait age and gender estimation dataset showed the superiority of the proposed method compared to the existing appearance-based baseline. We implement online standalone and client-server systems based on the proposed framework to demonstrate the performance of real-time estimation. We further propose a geometric correction step to the input gait sequence for a more generalization capability of the online system.

Human Identification at a Distance (HID) is an important research area due to its importance (especially in biometrics) and inherent challenges within this domain. To mitigate some of the constraints, we have introduced the HID challenge. This paper presents an overview of the 4th International Competition on Human Identification at a Distance (HID 2023), which serves as a benchmark for evaluating various methods in the field of human identification at a distance. We have introduced a new dataset, SUSTech-Competition, engulfing a cross-domain challenge. This dataset has 859 subjects, having various variations of clothing, carrying conditions, occlusions, and view angles. With a substantial participation of 254 registered teams, HID 2023 has attracted considerable attention and yielded highly encouraging results. Notably, the top-performing teams achieved significantly good accuracies. In this paper, we provide an introduction to the competition, encompassing the dataset, experimental settings, and competition organization, as well as an analysis of the results obtained by the top teams. Additionally, we delve into the methodologies employed by these leading teams. The progress demonstrated in this competition offers an optimistic outlook on the advancements in gait recognition, highlighting its potential for robust real applications.

In this paper, we describe an uncertainty-aware estimation framework for gait relative attributes. We specifically design a two-stream network model that takes a pair of gait videos as input. It then outputs a corresponding pair of Gaussian distributions of gait absolute attribute scores and annotator-dependent gait relative attribute label distributions. Moreover, we propose a differentiable annotator-independent uncertainty layer to estimate the gait relative attribute score distribution from the absolute distributions then map it to a relative attribute label distribution using the computation of cumulative distribution functions. Furthermore, we propose another annotator-dependent uncertainty layer to estimate the uncertainty on the gait relative attribute labels in terms of a set of trainable transition matrices. Finally, we design a joint loss function on the relative attribute label distribution to learn the model parameters. Experiments on two gait relative attribute datasets demonstrated the effectiveness of the proposed method against baselines in quantitative and qualitative evaluations.

Partial occlusion of the human body caused by obstacles or a limited camera field of view often occurs in surveillance videos, which affects the performance of gait recognition in practice. Existing methods for gait recognition against occlusion require a bounding box or the height of a full human body as a prerequisite, which is unobserved in occlusion scenarios. In this paper, we propose an occlusion-aware model-based gait recognition method that works directly on gait videos under occlusion without the above-mentioned prerequisite. Specifically, given a gait sequence that only contains non-occluded body parts in the images, we directly fit a skinned multi-person linear (SMPL)-based human mesh model to the input images without any pre-normalization or registration of the human body. We further use the pose and shape features extracted from the estimated SMPL model for recognition purposes, and use the extracted camera parameters in the occlusion attenuation module to reduce intra-subject variation in human model fitting caused by occlusion pattern differences. Experiments on occlusion samples simulated from the OU-MVLP dataset demonstrated the effectiveness of the proposed method, which outperformed state-of-the-art gait recognition methods by about 15% rank-1 identification rate and 2% equal error rate in the identification and verification scenarios, respectively.

The paper provides a summary of the Competition on Human Identification at a Distance 2022 (HID 2022), which is the third one in a series of competitions. HID 2022 is for promoting the research in human identification at a distance by providing a benchmark to evaluate different methods. The competition attracted 112 valid registered teams. 71 teams and 51 teams submitted their results in the first phase and the second phase, respectively. Very encouraging results have been achieved, and the accuracies of the top teams are much higher than those achieved in the previous two competitions. In this paper, we introduce the competition including the dataset, experimental settings, competition organization, results from the top teams and their analysis. The methods used by the top teams are also presented in the paper. The progress of this competition can give us an optimistic view on gait recognition.

Deep neural networks have been shown to be vulnerable against specifically-crafted perturbations designed to affect their predictive performance. Such perturbations, formally termed ‘adversarial attacks’ have been designed for various domains in the literature, most prominently in computer vision and more recently, in time series classification. Therefore there is a need to derive robust strategies to defend deep networks from such attacks. In this work we propose to establish axioms of robustness against adversarial attacks in time series classification. We subsequently design a suitable experimental methodology and empirically validate the hypotheses put forth. Results obtained from our investigations confirm the proposed hypotheses, and provide a strong empirical baseline with a view to mitigating the effects of adversarial attacks in deep time series classification.

This paper describes a method for estimating gait relative attribute distributions. Existing datasets for gait relative attributes have only three-grade annotations, which cannot be represented in the form of distributions. Thus, we first create a dataset with seven-grade annotations for five gait relative attributes (i.e., beautiful, graceful, cheerful, imposing, and relaxed). Second, we design a deep neural network to handle gait relative attribute distributions. Although the ground-truth (i.e., annotation) is given in a relative (or pairwise) manner with some degree of uncertainty (i.e., inconsistency among multiple annotators), it is desirable for the system to output an absolute attribute distribution for each gait input. Therefore, we develop a model that converts a pair of absolute attribute distributions into a relative attribute distribution. More specifically, we formulate the conversion as a transportation process from one absolute attribute distribution to the other, then derive a differentiable model that determines the trade-off between optimal transport and uniform transport. Finally, we learn the network parameters by minimizing the dissimilarity between the estimated and ground-truth distributions through the Kullback–Leibler divergence and the expectation dissimilarity. Experimental results show that the proposed method successfully estimates both absolute and relative attribute distributions.

The Competition on Human Identification at a Distance 2021 (HID 2021) is to promote the research in human identification at a distance and to provide a benchmark to evaluate different methods. HID 2021 is the second follow-up from the first one, HID 2020. The dataset size and the evaluation protocal are the same with the previous competition, but the data in the test set has been changed. The paper firstly introduces the dataset and the evaluation protocol, then describes the methods from the top teams and their results. The methods show how to achieve state-of-the-art performance on gait recognition. The results in HID 2021 are better than those in HID 2020. From the comparisons and analysis, some useful conclusions can be drawn. We hope more improvements can be achieved by better followup competitions.

In this paper, we propose a unified real-time framework for gait-based age estimation and gender classification that uses just a single image, which reduces the latency in video capturing compared with the existing methods based on a gait cycle. To cope with the problem of lacking motion information in the input single image, we first reconstruct a gait cycle of a silhouette sequence from the input image via a gait cycle reconstruction network. The reconstructed gait cycle is then fed into a state-of-the-art gait recognition network for feature representation learning, which is further used to obtain the class of the gender and the estimated probability distribution of integer age labels. Unlike the existing methods focusing on the gait sequences captured from the side view, the proposed method is applicable to the gait images from an arbitrary view with a single trained model, which is more suitable for real-world application scenarios (e.g., automatic access control). Stand-alone and client-server online systems were implemented based on the proposed method, which validates the real-time/online property in actual scenes. The experiments on the world's largest multi-view gait dataset demonstrate the effectiveness of the proposed method, which achieves performance improvement compared with the benchmark algorithms.

Action recognition is a very widely explored research area in computer vision and related fields. We propose Kinematics Posture Feature (KPF) extraction from 3D joint positions based on skeleton data for improving the performance of action recognition. In this approach, we consider the skeleton 3D joints as kinematics sensors. We propose Linear Joint Position Feature (LJPF) and Angular Joint Position Feature (AJPF) based on 3D linear joint positions and angles between bone segments. We then combine these two kinematics features for each video frame for each action to create the KPF feature sets. These feature sets encode the variation of motion in the temporal domain as if each body joint represents kinematics position and orientation sensors. In the next stage, we process the extracted KPF feature descriptor by using a low pass filter, and segment them by using sliding windows with optimized length. This concept resembles the approach of processing kinematics sensor data. From the segmented windows, we compute the Position-based Statistical Feature (PSF). These features consist of temporal domain statistical features (e.g., mean, standard deviation, variance, etc.). These statistical features encode the variation of postures (i.e., joint positions and angles) across the video frames. For performing classification, we explore Support Vector Machine (Linear), RNN, CNNRNN, and ConvRNN model. The proposed PSF feature sets demonstrate prominent performance in both statistical machine learning- and deep learning-based models. For evaluation, we explore five benchmark datasets namely UTKinect-Action3D, Kinect Activity Recognition Dataset (KARD), MSR 3D Action Pairs, Florence 3D, and Office Activity Dataset (OAD). To prevent overfitting, we consider the leave-one-subject-out framework as the experimental setup and perform 10-fold cross-validation. Our approach outperforms several existing methods in these benchmark datasets and achieves very promising classification performance.

In this paper, we propose a pairwise spatial transformer network (PSTN) for cross-view gait recognition, which reduces unwanted feature mis-alignment due to view differences before a recognition step for better performance. The proposed PSTN is a unified CNN architecture that consists of a pairwise spatial transformer (PST) and subsequent recognition network (RN). More specifically, given a matching pair of gait features from different source and target views, the PST estimates a non-rigid deformation field to register the features in the matching pair into their intermediate view, which mitigates distortion by registration compared with the case of direct deformation from the source view to target view. The registered matching pair is then fed into the RN to output a dissimilarity score. Although registration may reduce not only intra-subject variations but also inter-subject variations, we can still achieve a good trade-off between them using a loss function designed to optimize recognition accuracy. Experiments on three publicly available gait datasets demonstrate that the proposed method yields superior performance for both verification and identification scenarios by combining any gait recognition network benchmarks with the PST.

Existing gait recognition approaches typically focus on learning identity features that are invariant to covariates (e.g., the carrying status, clothing, walking speed, and viewing angle) and seldom involve learning features from the covariate aspect, which may lead to failure modes when variations due to the covariate overwhelm those due to the identity. We therefore propose a method of gait recognition via disentangled representation learning that considers both identity and covariate features. Specifically, we first encode an input gait template to get the disentangled identity and covariate features, and then decode the features to simultaneously reconstruct the input gait template and the canonical version of the same subject with no covariates in a semi-supervised manner to ensure successful disentanglement. We finally feed the disentangled identity features into a contrastive/triplet loss function for a verification/identification task. Moreover, we find that new gait templates can be synthesized by transferring the covariate feature from one subject to another. Experimental results on three publicly available gait data sets demonstrate the effectiveness of the proposed method compared with other state-of-the-art methods.

In recent times, deep neural networks have seen increased adoption in highly critical tasks. They are also susceptible to adversarial attacks, which are specifically crafted changes made to input samples which lead to erroneous output from such models. Such attacks have been shown to affect different types of data such as images and more recently, time-series data. Such susceptibility could have catastrophic consequences, depending on the domain.We propose a method for detecting Fast Gradient Sign Method (FGSM) and Basic Iterative Method (BIM) adversarial attacks as adapted for time-series data. We frame the problem as an instance of outlier detection and construct a normalcy model based on information and chaos-theoretic measures, which can then be used to determine whether unseen samples are normal or adversarial. Our approach shows promising performance on several datasets from the 2015 UCR Time Series Archive, reaching up to 97% detection accuracy in the best case.

Gait-based age estimation is one of key techniques for many applications (e.g., finding lost children/aged wanders). It is well known that the age estimation uncertainty is highly dependent on ages (i.e., it is generally small for children while is large for adults/the elderly), and it is important to know the uncertainty for the above-mentioned applications. We therefore propose a method of uncertainty-aware gait-based age estimation by introducing a label distribution learning framework. More specifically, we design a network which takes an appearance-based gait feature as an input and outputs discrete label distributions in the integer age domain. Experiments with the world-largest gait database OULP-Age show that the proposed method can successfully represent the uncertainty of age estimation and also outperforms or is comparable to the state-of-the-art methods.

In this paper, we propose a unified convolutional neural network (CNN) framework for robust gait recognition against posture changes (e.g., those induced by walking speed changes). In order to mitigate the posture changes, we first register an input matching pair of gait features with different postures by a deformable registration network, which estimates a deformation field to transform the input pair both into their intermediate posture. The pair of the registered features is then fed into a recognition network. Furthermore, ways of the deformation (i.e., deformation patterns) can differ between the same subject pairs (e.g., only posture deformation) and different subject pairs (e.g., not only posture deformation but also body shape deformation), which implies the deformation pattern can be another cue to distinguish the same subject pairs from the different subject pairs. We therefore introduce another recognition network whose input is the deformation pattern. Finally, the deformable registration network, and the two recognition networks for the registered features and the deformation patterns, constitute the whole framework, named DeformGait, and they are trained in an end-to-end manner by minimizing a loss function which is appropriately designed for each of verification and identification scenario. Experiments on the publicly available dataset containing the largest speed variations demonstrate that the proposed method achieves the state-of-the-art performance in both identification and verification scenarios.

This paper presents a deep learning-based method to estimate gait attributes (e.g., stately, cool, relax, etc.). Similarly to the existing studies on relative attribute, human perception-based annotations on the gait attributes are given to pairs of gait videos (i.e., the first one is better, tie, and the second one is better), and the relative annotations are utilized to train a ranking model of the gait attribute. More specifically, we design a Siamese (i.e., two-stream) network which takes a pair of gait inputs and output gait attribute score for each. We then introduce a suitable loss function called a signed contrastive loss to train the network parameters with the relative annotation. Unlike the existing loss functions for learning to rank does not inherit a nice property of a quadratic contrastive loss, the proposed signed quadratic contrastive loss function inherits the nice property. The quantitative evaluation results reveal that the proposed method shows better or comparable accuracies of relative attribute prediction against the baseline methods.

A plethora of techniques have been proposed in human action recognition fields, and particularly deep learning-based methods such as convolutional neural networks (CNNs) have achieved impressive results. Usually, there is need to tune hyper-parameters in the deep neural network (e.g., filter size, stride) to achieve reasonable results. Such hyper-parameter tuning is, however, extremely time and resource-intensive even for small models. In this paper, we posit that the inclusion of an adaptive pooling in CNNs used for human action recognition largely eliminates the need for hyper-parameter tuning. Specifically, we demonstrated our idea for human action recognition using inertial sensor data (i.e., a temporal sequence) with a one-dimensional adaptive pooling. We compared the adaptive pooling to conventional CNNs with randomly chosen hyper-parameters using a publicly available data set for human action recognition. Experimental results showed that the adaptive pooling achieved better accuracy than the conventional CNNs.

Gait recognition invariant to carried objects (COs) is very difficult in a real-life scene because the COs can have various shapes and sizes, in addition to unpredictable carrying locations (e.g., front, back, and side, or multiple locations). Therefore, in this paper, we propose a robust method for gait recognition against various COs by reconstructing a gait template without COs. A straightforward approach is to directly generate a gait template without COs given a gait template with COs as the input using a conventional generative adversarial network. There is, however, a potential risk of unnecessarily altering parts that were originally unaffected by COs (e.g., leg parts for a person carrying a backpack). Because we do not want to touch such unaffected parts in the original template, we first estimate a gait template without COs, and then blend it with the original template by an estimated alpha matte that indicates the blending parameters. We then create an alpha-blended template from the original template and the generated template without COs based on the estimated alpha matte. We use two independent generators to estimate the alpha matte and the generated template without COs. Finally, we feed the alpha-blended gait template into a state-of-the-art discrimination network for gait recognition. The experimental results on three publicly available gait databases with real-life COs demonstrate the state-of-the-art performance of the proposed method.

Gait is one of the most popular behavioral biometrics because it can be authenticated at a distance from a camera without subject cooperation. Speed differences between matching pairs, however, cause significant performance drops in gait recognition, and gait mode difference (i.e., walking versus running) makes gait recognition further challenging. We therefore propose a speed-invariant gait representation called single-support GEI (SSGEI), which realizes a good trade-off between speed invariance and stability by aggregating multiple frames around single-support phases. In addition, to mitigate the pose differences between walking and running modes at single-support phases, we morph walking and running SSGEIs into intermediate SSGEIs between walking and running mode, where we exploit a free-form deformation field from the walking or running modes to the intermediate mode obtained by training data. We finally apply Gabor filtering and spatial metric learning as postprocessing for further accuracy improvement. Experiments on two publicly available datasets, the OU-ISIR Treadmill Dataset A and the CASIA-C Dataset demonstrate that the proposed method yields the state-of-the-art accuracies in both identification and verification scenarios with a low computational cost.

Gait is believed to be an advanced behavioral biometric that can be perceived at a large distance from a camera without subject cooperation and hence is favorable for many applications in surveillance and forensics. However, appearance differences caused by human aging may significantly reduce the performance of gait recognition. Modeling the aging process on gait features is one of the possible solutions to this problem, and it may inspire more potential applications, such as finding lost children and examining health status. To the best of our knowledge, this topic has not been studied in the literature. Motivated by the fact that aging effects are mainly reflected in the shape and appearance deformations of the gait feature, we propose a baseline algorithm for gait-based age progression and regression using a generic geometric transformation between different age groups, in conjunction with the gait energy image, which is an appearance-based gait feature frequently used in the gait analysis community, to render gait aging and reverse aging effects simultaneously. Various evaluations were conducted through gait-based age group classification and cross-age gait identification to validate the performance of the proposed method, in addition to providing several insights for future research on the subject.

In this paper, we discuss input/output architectures for convolutional neural network (CNN)-based cross-view gait recognition. For this purpose, we consider two aspects: verification versus identification and the tradeoff between spatial displacements caused by subject difference and view difference. More specifically, we use the Siamese network with a pair of inputs and contrastive loss for verification and a triplet network with a triplet of inputs and triplet ranking loss for identification. The aforementioned CNN architectures are insensitive to spatial displacement, because the difference between a matching pair is calculated at the last layer after passing through the convolution and max pooling layers; hence, they are expected to work relatively well under large view differences. By contrast, because it is better to use the spatial displacement to its best advantage because of the subject difference under small view differences, we also use CNN architectures where the difference between a matching pair is calculated at the input level to make them more sensitive to spatial displacement. We conducted experiments for cross-view gait recognition and confirmed that the proposed architectures outperformed the state-of-the-art benchmarks in accordance with their suitable situations of verification/identification tasks and view differences.

Clothing and carrying status variations are the two key factors that affect the performance of gait recognition because people usually wear various clothes and carry all kinds of objects, while walking in their daily life. These covariates substantially affect the intensities within conventional gait representations such as gait energy images. Hence, to properly compare a pair of input gait features, an appropriate metric for joint intensity is needed in addition to the conventional spatial metric. We therefore propose a unified joint intensity transformer network for gait recognition that is robust against various clothing and carrying statuses. Specifically, the joint intensity transformer network is a unified deep learning-based architecture containing three parts: a joint intensity metric estimation net, a joint intensity transformer, and a discrimination network. First, the joint intensity metric estimation net uses a well-designed encoder-decoder network to estimate a sample-dependent joint intensity metric for a pair of input gait energy images. Subsequently, a joint intensity transformer module outputs the spatial dissimilarity of two gait energy images using the metric learned by the joint intensity metric estimation net. Third, the discrimination network is a generic convolution neural network for gait recognition. In addition, the joint intensity transformer network is designed with different loss functions depending on the gait recognition task (i.e., a contrastive loss function for the verification task and a triplet loss function for the identification task). The experiments on the world's largest datasets containing various clothing and carrying statuses demonstrate the state-of-the-art performance of the proposed method.

Human age estimation from gait is expected to be an important technology for a variety of applications such as automatic customer counting for marketing research or automatic age-based access control restriction for a specific area because the gait can be observable at a distance from a camera (e.g., CCTV). Although the aging process of gait significantly differs among age groups (e.g., children, adults, and the elderly), previous studies on gait-based human age estimation employ a single age group-independent estimation model that suffers from large estimation errors when the age variation increases. We therefore propose an age group-dependent gait-based human age estimation method for better accuracy. Specifically, in the training phase, we first compose age groups that are well-separated from each other by clustering gait features along with their age labels. We then learn a classifier that classifies the gait features for multiple age groups using a directed acyclic graph support vector machine. Next, we learn an age regression model for each age group using support vector regression with a Gaussian kernel in conjunction with a manifold learning technique, i.e., orthogonal locality preserving projection, to better characterize the gait feature. In the test phase, given a gait feature, it is first classified into an age group and then its age is estimated with the age regression model of the classified age group. Experimental results on a gait database that has the world’s largest population of participants ranging from 2 to 90 years old demonstrate the state-of-the-art performance of the proposed method.

This paper describes a joint intensity metric learning method to improve the robustness of gait recognition with silhouette-based descriptors such as gait energy images. Because existing methods often use the difference of image intensities between a matching pair (e.g., the absolute difference of gait energies for the l1-norm) to measure a dissimilarity, large intrasubject differences derived from covariate conditions (e.g., large gait energies caused by carried objects vs. small gait energies caused by the background), may wash out subtle intersubject differences (e.g., the difference of middle-level gait energies derived from motion differences). We therefore introduce a metric on joint intensity to mitigate the large intrasubject differences as well as leverage the subtle intersubject differences. More specifically, we formulate the joint intensity and spatial metric learning in a unified framework and alternately optimize it by linear or ranking support vector machines. Experiments using the OU-ISIR treadmill data set B with the largest clothing variation and large population data set with bag, β version containing carrying status in the wild demonstrate the effectiveness of the proposed method.

Cross-view gait recognition authenticates a person using a pair of gait image sequences with different observation views. View difference causes degradation of gait recognition accuracy, and so several solutions have been proposed to suppress this degradation. One useful solution is to apply a view transformation model (VTM) that encodes a joint subspace of multiview gait features trained with auxiliary data from multiple training subjects, who are different from test subjects (recognition targets). In the VTM framework, a gait feature with a destination view is generated from that with a source view by estimating a vector on the trained joint subspace, and gait features with the same destination view are compared for recognition. Although this framework improves recognition accuracy as a whole, the fit of the VTM depends on a given gait feature pair, and causes an inhomogeneously biased dissimilarity score. Because it is well known that normalization of such inhomogeneously biased scores improves recognition accuracy in general, we therefore propose a VTM incorporating a score normalization framework with quality measures that encode the degree of the bias. From a pair of gait features, we calculate two quality measures, and use them to calculate the posterior probability that both gait features originate from the same subjects together with the biased dissimilarity score. The proposed method was evaluated against two gait datasets, a large population gait dataset of over-ground walking (course dataset) and a treadmill gait dataset. The experimental results show that incorporating the quality measures contributes to accuracy improvement in many cross-view settings.

Gait recognition is a useful biometric trait for person authentication because it is usable even with low image resolution. One challenge is robustness to a view change (cross-view matching); view transformation models (VTMs) have been proposed to solve this. The VTMs work well if the target views are the same as their discrete training views. However, the gait traits are observed from an arbitrary view in a real situation. Thus, the target views may not coincide with discrete training views, resulting in recognition accuracy degradation. We propose an arbitrary VTM (AVTM) that accurately matches a pair of gait traits from an arbitrary view. To realize an AVTM, we first construct 3D gait volume sequences of training subjects, disjoint from the test subjects in the target scene. We then generate 2D gait silhouette sequences of the training subjects by projecting the 3D gait volume sequences onto the same views as the target views, and train the AVTM with gait features extracted from the 2D sequences. In addition, we extend our AVTM by incorporating a part-dependent view selection scheme (AVTM_PdVS), which divides the gait feature into several parts, and sets part-dependent destination views for transformation. Because appropriate destination views may differ for different body parts, the part-dependent destination view selection can suppress transformation errors, leading to increased recognition accuracy. Experiments using data sets collected in different settings show that the AVTM improves the accuracy of cross-view matching and that the AVTM_PdVS further improves the accuracy in many cases, in particular, verification scenarios.

This paper describes a method of gait recognition from image sequences wherein a subject is accelerating or decelerating. As a speed change occurs due to a change of pitch (the first-order derivative of a phase, namely, a gait stance) and/or stride, we model this speed change using a cylindrical manifold whose azimuth and height corresponds to the phase and the stride, respectively. A radial basis function (RBF) interpolation framework is used to learn subject specific mapping matrices for mapping from manifold to image space. Given an input image sequence of speed transited gait of a test subject, we estimate the mapping matrix of the test subject as well as the phase and stride sequence using an energy minimization framework considering the following three points: (1) fitness of the synthesized images to the input image sequence as well as to an eigenspace constructed by exemplars of training subjects, (2) smoothness of the phase and the stride sequence, and (3) pitch and stride fitness to the pitch-stride preference model. Using the estimated mapping matrix, we synthesize a constant-speed gait image sequence, and extract a conventional period-based gait feature from it for matching. We conducted experiments using real speed transited gait image sequences with 179 subjects and demonstrated the effectiveness of the proposed method.

This paper describes a method for score-level fusion in multi-cue two-class classification problems. Fusion based on the probability density function (PDF) of multiple scores given for each class is a promising approach because it guarantees optimality as long as the estimated PDFs are correct. Instead of lattice-type control points used in previous non-parametric density-based approaches, floating control points (FCPs) are introduced to improve scalability and the whole posterior distribution is represented by interpolation or extrapolation using generalized Delaunay triangulation. Given a set of FCPs obtained by k-means, posteriors on the FCPs are estimated by an energy minimization framework using training samples. The experiments, using both simulation data as well as several types of real data from three publicly available score databases for multi-cue biometric authentication, demonstrate the effectiveness of the proposed method.

Gait is a unique and promising behavioral biometrics which allows to authenticate a person even at a distance from the camera. Since a matching pair of gait features are often drawn from different views due to differences in camera position/attitude and walking directions in the real world, it is important to cope with cross-view gait recognition. In this paper, we propose a discriminative approach to cross-view gait recognition using view-dependent projection matrices, unlike the existing discriminant approaches which utilize only a single common projection matrix for different views. We demonstrated the effectiveness of the proposed method through cross-view gait recognition experiments with two publicly available gait datasets. In addition, since the success of the discriminant analysis relies on the training sample size, we show the effect of transfer learning across two gait datasets as well as provide the rigorous sensitivity analysis of the proposed method against the number of training subjects ranging from 10 to approximately 1,000 subjects.

Human gait modeling (e.g., for person identification) largely relies on image-based representations that muddle gait with body shape. Silhouettes, for instance, inherently entangle body shape and gait. For gait analysis and recognition, decoupling these two factors is desirable. Most important, once decoupled, they can be combined for the task at hand, but not if left entangled in the first place. In this paper, we introduce Two-Point Gait, a gait representation that encodes the limb motions regardless of the body shape. Two-Point Gait is directly computed on the image sequence based on the two point statistics of optical flow fields. We demonstrate its use for exploring the space of human gait and gait recognition under large clothing variation. The results show that we can achieve state-of-the-art person recognition accuracy on a challenging dataset.

Pose-based approaches for human action recognition are attractive owing to their accurate use of human motion information. Traditionally, such approaches used kinematic features for classification. However, in addition to having high dimensions and a small interclass variation, kinematic features do not consider the interaction of the environment on human motion. In this paper, we propose a method for action recognition using dynamic features, derived by applying inverse dynamics to a physics-based representation of the human body. The physics-based model is articulated and actuated with muscles and consists of joints with variable stiffness. Dynamic features under consideration include the torques from the knee and hip joints of both legs and, implicitly, gravity, ground reaction forces, and the pose of the remaining body parts. These features are more discriminative than kinematic features, resulting in a low-dimensional representation for human actions, which preserves much of the information of the original high-dimensional pose. This low-dimensional feature achieves good classification performance even with a relatively small training data set in a simple classification framework such as a hidden Markov model. The effectiveness of the proposed method is demonstrated through experiments on the Carnegie Mellon University motion capture data set and Osaka University Kinect action data set with various actions.

In this paper, we propose a temporal super resolution approach for quasi-periodic image sequence such as human gait. The proposed method effectively combines example-based and reconstruction-based temporal super resolution approaches. A periodic image sequence is expressed as a manifold parameterized by a phase and a standard manifold is learned from multiple high frame-rate sequences in the training stage. In the test stage, an initial phase for each frame of an input low frame-rate image sequence is estimated based on the standard manifold at first, and the manifold reconstruction and the phase estimation are then iterated to generate better high frame-rate images in the energy minimization framework that ensures the fitness to both the input images and the standard manifold. The proposed method is applied to low frame-rate gait recognition and experiments with real data of 100 subjects demonstrate a significant improvement by the proposed method, particularly for quite low frame-rate videos (e.g., 1 fps).

This paper proposes a method of gait recognition using not only shape feature but also motion feature from silhouette image sequences. The inner silhouette motion called pseudo motion is constructed by dividing the silhouette shape into small clusters and by computing many-to-many correspondence via earth mover's morphing framework. The raw pseudo motion, however, tends to be locally fluctuated in the spatio-temporal domain, and hence the spatio-temporal regularization is imposed to provide the smooth pseudo motion. The smooth pseudo motion image sequences are further partitioned into images with eight different orientations, and then averaged over each gait period to produce images of oriented smooth pseudo motion. Both shape and motion cues are integrated in score-level fusion framework based on linear logistic regression and the single-dimensional fused distance is returned by the learned optimal weights. The experiments with the publicly available gait database show the effectiveness of the proposed method compared with the case where the shape information is used alone.

This paper describes the world's largest gait database-the “OU-ISIR Gait Database, Large Population Dataset”-and its application to a statistically reliable performance evaluation of vision-based gait recognition. Whereas existing gait databases include at most 185 subjects, we construct a larger gait database that includes 4007 subjects (2135 males and 1872 females) with ages ranging from 1 to 94 years. The dataset allows us to determine statistically significant performance differences between currently proposed gait features. In addition, the dependences of gait-recognition performance on gender and age group are investigated and the results provide several novel insights, such as the gradual change in recognition performance with human growth.

Recently, many gait recognition algorithms are proposed, and the optimal camera arrangement is necessary to maximize the performance. In this paper, we propose the optimal camera arrangement by using a performance model that considers observation conditions comprehensively. We select silhouette resolution, observation view, and its local and global changes as the observation conditions affecting the performance. Then, training sets composed of pairs of the observation conditions and the performance is obtained by gait recognition experiments under several camera arrangements. A performance model is constructed by applying Gaussian Processes Regression to the training set. The optimal arrangement is determined by estimating the performance for each camera arrangement with the performance model. The effectiveness of the proposed method is demonstrated by experiments of performance estimation with a training set including 17 subjects and the optimal camera arrangement.

This paper describes a method of gait recognition where both a gallery and a probe are based on low frame-rate videos. The sparsity of phases (stances) per gait period makes it much harder to match the gait using existing gait recognition algorithms. Consequently, we introduce a super resolution technique to generate a high frame-rate periodic image sequence as a preprocess to matching. First, the initial phase for each frame is estimated based on an exemplar of a high frame-rate gait image sequence. Images between a pair of adjacent frames sorted by the estimated phases are then filled using a morphing technique to avoid ghosting effects. Next, a manifold of the periodic gait image sequence is reconstructed based on the estimated phase and morphed images. Finally, the phase estimation and manifold reconstruction are iterated to generate better high frame-rate images in the energy minimization framework. Experiments with real data on 100 subjects demonstrate the effectiveness of the proposed method particularly for low frame-rate videos of less than 5 fps.

This paper describes a method of score-level fusion to optimize a Receiver Operating Characteristic (ROC) curve for multimodal biometrics. When the Probability Density Functions (PDFs) of the multimodal scores for each client and imposter are obtained from the training samples, it is well known that the isolines of a function of probabilistic densities, such as the likelihood ratio, posterior, or Bayes error gradient, give the optimal ROC curve. The success of the probability density-based methods depends on the PDF estimation for each client and imposter, which still remains a challenging problem. Therefore, we introduce a frame work of direct estimation of the Bayes error gradient that bypasses the troublesome PDF estimation for each client and imposter. The lattice-type control points are allocated in a multiple score space, and the Bayes error gradients on the control points are then estimated in a comprehensive manner in the energy minimization framework including not only the data fitness of the training samples but also the boundary conditions and monotonic increase constraints to suppress the over-training. The experimental results for both simulation and real public data show the effectiveness of the proposed method.

This paper addresses gait-based age estimation using a large-scale whole-generation gait database. Previous work on gait-based age estimation evaluated their methods using databases that included only 170 subjects at most with a limited age variation, which was insufficient to statistically demonstrate the possibility of gait-based age estimation. Therefore, we first constructed a much larger whole generation gait database which includes 1,728 subjects with ages ranging from 2 to 94 years. We then provided a base line algorithm for gait-based age estimation implemented by Gaussian process regression, which has achieved successes in the face-based age estimation field, in conjunction with silhouette-based gait features such as an averaged silhouette (or Gait Energy Image) which has been used extensively in many gait recognition algorithms. Finally, experiments using the whole-generation gait database demonstrated the viability of gait-based age estimation.

In this paper, we propose a method of inertial sensor-based gait authentication by inter-period phase registration of an owner's gallery. In spite of the importance for gait authentication of constructing a gallery of phase-registered gait patterns, previous implementations just relied on simple methods of period detection based on heuristic knowledge such as local peaks/valleys or local auto-correlation of the gait signals. Consequently, we propose to improve a gait gallery by incorporating a phase registration technique which globally optimizes inter-period phase consistency in an energy minimization framework. However, the previous phase registration technique suffers from a phase distortion problem due to ambiguities in the combination of a periodic signal function and a phase evolution function. We present a linear phase evolution prior to constructing an undistorted gait signal for better matching performance. Experiments using real gait signals from 32 subjects show that the proposed methods outperform the latest methods in the field.

In this paper, we propose a method of action recognition using dynamics features based on physics model. The dynamics features are composed of torques from knee and hip joints of both legs and implicitly include the gravity, ground reaction forces, and the pose of the remaining body parts. These features are more discriminative than the kinematics features, and they result in a low dimensional representation of a human action which preserves much information of the original high dimensional pose. This low dimensional feature allows us to achieve a good classification performance even with a relatively small training data in a simple classification framework such as HMM. The effectiveness of the proposed method is demonstrated through experiments on the CMU motion capture dataset with various actions.

We propose a method of gait silhouette transformation from one speed to another to cope with walking speed changes in gait identification. When a person changes his/her walking speed, dynamic features (e.g. stride and joint angle) are changed while static features (e.g. thigh and shin lengths) are unchanged. Based on the fact, firstly, static and dynamic features are separated from gait silhouettes by fitting a human model. Secondly, a factorization-based speed transformation model for the dynamic features is created using a training set for multiple persons on multiple speeds. This model can transform the dynamic features from a reference speed to another arbitrary speed. Finally, silhouettes are restored by combining the unchanged static features and the transformed dynamic features. Evaluation by gait identification using silhouette-based frequency-domain features shows the effectiveness of the proposed method.

In the biometric verification, authentication is given when a distance of biometric signatures between enrollment and test phases is less than an acceptance threshold, and the performance is usually evaluated by a so-called Receiver Operating Characteristics (ROC) curve expressing a trade off between False Rejection Rate (FRR) and False Acceptance Rate (FAR). On the other hand, it is also well known that the performance is significantly affected by the situation differences between enrollment and test phases. This paper describes a method to adaptively control an acceptance threshold with quality measures derived from situation differences so as to optimize the ROC curve. We show that the optimal evolution of the adaptive threshold in the domain of the distance and quality measure is equivalent to a constant evolution in the domain of the error gradient defined as a ratio of a total error rate to a total acceptance rate. An experiment with simulation data demonstrates that the proposed method outperforms the previous methods, particularly under a lower FAR or FRR tolerance condition.

Pattern recognition problems often suffer from the larger intra-class variation due to situation variations such as pose, walking speed, and clothing variations in gait recognition. This paper describes a method of discriminant subspace analysis focused on situation cluster pair. In training phase, both a situation cluster discriminant subspace and class discriminant subspaces for the situation cluster pair by using training samples of non recognition-target classes. In testing phase, given a matching pair of patterns of recognition-target classes, posterior of situation cluster pairs is estimated at first, and then the distance is calculated in the corresponding cluster-pairwise class discriminant subspace. The experiments both with simulation data and real data show the effectiveness of the proposed method.

This paper proposes a method for period-based gait trajectory matching in the eigenspace using phase synchronization for low frame-rate videos. First, a gait period is detected by maximizing the normalized autocorrelation of the gait silhouette sequence for the temporal axis. Next, a gait silhouette sequence is expressed as a trajectory in the eigenspace and the gait phase is synchronized by time stretching and time shifting of the trajectory based on the detected period. In addition, multiple period-based matching results are integrated via statistical procedures for more robust matching in the presence of fluctuations among gait sequences. Results of experiments conducted with 185 subjects to evaluate the performance of the gait verification with various spatial and temporal resolutions, demonstrate the effectiveness of the proposed method.

This paper describes a method of part-based gait identification under substantial clothes variations. When clothes types between a gallery and a probe are different, silhouettes fairly change for some parts and subject discrimination capability decrease for those parts. Therefore, we exploit the discrimination capability as a matching weight for each part and control the weights adaptively based on a distribution of distances between a probe and all the galleries. As a result of experiments with our clothes-variation gait dataset, the proposedmethod achievedmuch better performance than a whole-based approach.

We propose an iterative scheme of spatio-temporal local color transformation of background and graph-cut segmentation for silhouette extraction. Given an initial background subtraction, spatio-temporal background color transformation is processed for fitting modeled background colors to input background ones under a different illumination condition. After foreground colors are modeled based on the fit background, spatio-temporal graph-cut algorithm is applied to acquire a foreground/background segmentation result. Because these two processes need well-segmented background and well-fit background each other, they are iterated in turn to obtain better silhouette extraction results. Silhouette extraction experiments for a walking human on a treadmill show the effectiveness of the proposed method.

Gait recognition has recently gained attention as an effective approach to identify individuals at a distance from a camera. Most existing gait recognition algorithms assume that people have been tracked and silhouettes have been segmented successfully. Tacking and segmentation are, however, very difficult especially for articulated objects such as human beings. Therefore, we present an integrated algorithm for tracking and segmentation supported by gait recognition. After the tracking module produces initial results consisting of bounding boxes and foreground likelihood images, the gait recognition module searches for the optimal silhouette-based gait models corresponding to the results. Then, the segmentation module tries to segment people out using the provided gait silhouette sequence as shape priors. Experiments on real video sequences show the effectiveness of the proposed approach.

This paper describes a choice strategy to ease user's burdens for an interactive object recognition system when the system obtains multiple object candidates as a recognition result. First, we propose several methods to display the recognition result so as to make recognition of the candidate easy and hierarchical methods to reduce candidates per choice. We verify their effectiveness by subjective tests. Next, we propose a strategy to minimize time spent for choices. We divide the time into that for displayed candidates and that for speech dialog, and formulate each time to evaluate the strategy quantitatively. Last, we compare the strategy based on choice time with a subjective strategy