In this paper, we address the problem of recognizing group activities of human objects based on their motion trajectory analysis. In order to resolve the complexity and ambiguity problems caused by a large number of human objects, we propose a Group Interaction Zone (GIZ) to detect meaningful groups in a scene to effectively handle noisy information. Two novel features, Group Interaction Energy (GIE) feature and Attraction and Repulsion Features, are proposed to better describe group activities within a GIZ. We demonstrate the performance of our method in two ways by (i) comparing the performance of the proposed method with the previous methods and (ii) analyzing the influence of the proposed features and GIZ-based meaningful group detection on group activity recognition using public datasets.

Image retrieval has been extensively studied over the last two decades due to the increasing demands for the effective use of multimedia data. Among various approaches to image retrieval, scale space representation and local keypoint descriptors have been shown to be a promising approach. Even though the concept of scale space representation has been known for a long time, it has now gained prominence as a powerful method for image retrieval mostly due to the invention of the Scale Invariant Feature Transform (SIFT). We will review the characteristics of the scale space operation and provide an extended method of scale space operation that significantly improves the image matching accuracy in the context of image retrieval. We use an operational tattoo image database containing 1,000 near duplicate images to show the superior retrieval performance of the proposed method compared to SIFT keypoints.

Face recognition has been regarded as an effective method for subject identification at a distance because of its covert and remote sensing capability. However, face images have a low resolution when they are captured at a distance (say, larger than 5 meters) thereby degrading the face matching performance. To address this problem, we propose an imaging system consisting of static and pan-tilt-zoom (PTZ) cameras to acquire high resolution face images up to a distance of 12 meters. We propose a novel coaxial-concentric camera configuration between the static and PTZ cameras to achieve the distance invariance property using a simple calibration scheme. We also use a linear prediction model and camera motion control to mitigate delays in image processing and mechanical camera motion. Our imaging system was used to track 50 different subjects and their faces at distances ranging from 6 to 12 meters. The matching scenario consisted of these 50 subjects as probe and additional 10 000 subjects as gallery. Rank-1 identification accuracy of 91.5% was achieved compared to 0% rank-1 accuracy of the conventional camera system using a state-of-the-art matcher. The proposed camera system can operate at a larger distance (up to 50 meters) by replacing the static camera with a PTZ camera to detect a subject at a larger distance and control the second PTZ camera to capture the high-resolution face image.

Combining multiple low-level visual features is a proven and effective strategy for a range of computer vision tasks. However, limited attention has been paid to combining such features with information from other modalities, such as audio and videotext, for large scale analysis of web videos. In our work, we rigorously analyze and combine a large set of low-level features that capture appearance, color, motion, audio and audio-visual co-occurrence patterns in videos. We also evaluate the utility of high-level (i.e., semantic) visual information obtained from detecting scene, object, and action concepts. Further, we exploit multimodal information by analyzing available spoken and videotext content using state-of-the-art automatic speech recognition (ASR) and videotext recognition systems. We combine these diverse features using a two-step strategy employing multiple kernel learning (MKL) and late score level fusion methods. Based on the TRECVID MED 2011 evaluations for detecting 10 events in a large benchmark set of ~45000 videos, our system showed the best performance among the 19 international teams.

Face recognition has become a valuable and routine forensic tool used by criminal investigators. Compared to automated face recognition, forensic face recognition is more demanding because it must be able to handle facial images captured under non-ideal conditions and it has high liability for following legal procedures. This paper discusses recent developments in automated face recognition that impact the forensic face recognition community. Improvements in forensic face recognition through research in facial aging, facial marks, forensic sketch recognition, face recognition in video, near-infrared face recognition, and use of soft biometrics will be discussed. Finally, current limitations and future research directions for face recognition in forensics are suggested.

Face recognition at a distance is gaining wide attention in order to augment the surveillance systems with face recognition capability. However, face recognition at a distance in nighttime has not yet received adequate attention considering the increased security threats at nighttime. We introduce a new face image database, called Near-Infrared Face Recognition at a Distance Database (NFRAD-DB). Images in NFRAD-DB are collected at a distance of up to 60 meters with 50 different subjects using a near-infrared camera, a telescope, and near-infrared illuminator. We provide face recognition performance using FaceVACS, DoG-SIFT, and DoG-MLBP representations. The face recognition test consisted of NIR images of these 50 subjects at 60 meters as probe and visible images at 1 meter with additional mug shot images of 10,000 subjects as gallery. Rank-1 identification accuracy of 28 percent was achieved from the proposed method compared to 18 percent rank-1 accuracy of a state of the art face recognition system, FaceVACS. These recognition results are encouraging given this challenging matching problem due to the illumination pattern and insufficient brightness in NFRAD images.

Aging variation poses a serious problem to automatic face recognition systems. Most of the face recognition studies that have addressed the aging problem are focused on age estimation or aging simulation. Designing an appropriate feature representation and an effective matching framework for age invariant face recognition remains an open problem. In this paper, we propose a discriminative model to address face matching in the presence of age variation. In this framework, we first represent each face by designing a densely sampled local feature description scheme, in which scale invariant feature transform (SIFT) and multi-scale local binary patterns (MLBP) serve as the local descriptors. By densely sampling the two kinds of local descriptors from the entire facial image, sufficient discriminatory information, including the distribution of the edge direction in the face image (that is expected to be age invariant) can be extracted for further analysis. Since both SIFT-based local features and MLBP-based local features span a high-dimensional feature space, to avoid the overfitting problem, we develop an algorithm, called multi-feature discriminant analysis (MFDA) to process these two local feature spaces in a unified framework. The MFDA is an extension and improvement of the LDA using multiple features combined with two different random sampling methods in feature and sample space. By random sampling the training set as well as the feature space, multiple LDA-based classifiers are constructed and then combined to generate a robust decision via a fusion rule. Experimental results show that our approach outperforms a state-of-the-art commercial face recognition engine on two public domain face aging data sets: MORPH and FG-NET. We also compare the performance of the proposed discriminative model with a generative aging model. A fusion of discriminative and generative models further improves the face matching accuracy in the presence of aging.

In the Acknowledgment for the above paper (ibid., vol. 5, no. 4, pp. 771-780, Dec 2010), due to a production error, the corresponding author's name was spelled incorrectly. The correct spelling is Unsang Park.

The term periocular refers to the facial region in the immediate vicinity of the eye. Acquisition of the periocular biometric is expected to require less subject cooperation while permitting a larger depth of field compared to traditional ocular biometric traits (viz., iris, retina, and sclera). In this work, we study the feasibility of using the periocular region as a biometric trait. Global and local information are extracted from the periocular region using texture and point operators resulting in a feature set for representing and matching this region. A number of aspects are studied in this work, including the 1) effectiveness of incorporating the eyebrows, 2) use of side information (left or right) in matching, 3) manual versus automatic segmentation schemes, 4) local versus global feature extraction schemes, 5) fusion of face and periocular biometrics, 6) use of the periocular biometric in partially occluded face images, 7) effect of disguising the eyebrows, 8) effect of pose variation and occlusion, 9) effect of masking the iris and eye region, and 10) effect of template aging on matching performance. Experimental results show a rank-one recognition accuracy of 87.32% using 1136 probe and 1136 gallery periocular images taken from 568 different subjects (2 images/subject) in the Face Recognition Grand Challenge (version 2.0) database with the fusion of three different matchers.

Automatic illicit drug pill matching and retrieval is becoming an important problem due to an increase in the number of tablet type illicit drugs being circulated in our society. We propose an automatic method to match drug pill images based on the imprints appearing on the tablet. This will help identify the source and manufacturer of the illicit drugs. The feature vector extracted from tablet images is based on edge localization and invariant moments. Instead of storing a single template for each pill type, we generate multiple templates during the edge detection process. This circumvents the difficulties during matching due to variations in illumination and viewpoint. Experimental results using a set of real drug pill images (822 illicit drug pill images and 1,294 legal drug pill images) showed 76.74% (93.02%) rank one (rank-20) matching accuracy.

One of the challenges in automatic face recognition is to achieve temporal invariance. In other words, the goal is to come up with a representation and matching scheme that is robust to changes due to facial aging. Facial aging is a complex process that affects both the 3D shape of the face and its texture (e.g., wrinkles). These shape and texture changes degrade the performance of automatic face recognition systems. However, facial aging has not received substantial attention compared to other facial variations due to pose, lighting, and expression. We propose a 3D aging modeling technique and show how it can be used to compensate for the age variations to improve the face recognition performance. The aging modeling technique adapts view-invariant 3D face models to the given 2D face aging database. The proposed approach is evaluated on three different databases (i.g., FG-NET, MORPH, and BROWNS) using FaceVACS, a state-of-the-art commercial face recognition engine.

Most existing computer and network systems authenticate a user only at the initial login session. This could be a critical security weakness, especially for high-security systems because it enables an impostor to access the system resources until the initial user logs out. This situation is encountered when the logged in user takes a short break without logging out or an impostor coerces the valid user to allow access to the system. To address this security flaw, we propose a continuous authentication scheme that continuously monitors and authenticates the logged in user. Previous methods for continuous authentication primarily used hard biometric traits, specifically fingerprint and face to continuously authenticate the initial logged in user. However, the use of these biometric traits is not only inconvenient to the user, but is also not always feasible due to the user's posture in front of the sensor. To mitigate this problem, we propose a new framework for continuous user authentication that primarily uses soft biometric traits (e.g., color of user's clothing and facial skin). The proposed framework automatically registers (enrolls) soft biometric traits every time the user logs in and fuses soft biometric matching with the conventional authentication schemes, namely password and face biometric. The proposed scheme has high tolerance to the user's posture in front of the computer system. Experimental results show the effectiveness of the proposed method for continuous user authentication.

Soft biometric traits embedded in a face (e.g., gender and facial marks) are ancillary information and are not fully distinctive by themselves in face-recognition tasks. However, this information can be explicitly combined with face matching score to improve the overall face-recognition accuracy. Moreover, in certain application domains, e.g., visual surveillance, where a face image is occluded or is captured in off-frontal pose, soft biometric traits can provide even more valuable information for face matching or retrieval. Facial marks can also be useful to differentiate identical twins whose global facial appearances are very similar. The similarities found from soft biometrics can also be useful as a source of evidence in courts of law because they are more descriptive than the numerical matching scores generated by a traditional face matcher. We propose to utilize demographic information (e.g., gender and ethnicity) and facial marks (e.g., scars, moles, and freckles) for improving face image matching and retrieval performance. An automatic facial mark detection method has been developed that uses (1) the active appearance model for locating primary facial features (e.g., eyes, nose, and mouth), (2) the Laplacian-of-Gaussian blob detection, and (3) morphological operators. Experimental results based on the FERET database (426 images of 213 subjects) and two mugshot databases from the forensic domain (1225 images of 671 subjects and 10 000 images of 10 000 subjects, respectively) show that the use of soft biometric traits is able to improve the face-recognition performance of a state-of-the-art commercial matcher.

We propose to utilize micro features, namely facial marks (e.g., freckles, moles, and scars) to improve face recognition and retrieval performance. Facial marks can be used in three ways: i) to supplement the features in an existing face matcher, ii) to enable fast retrieval from a large database using facial mark based queries, and iii) to enable matching or retrieval from a partial or profile face image with marks. We use Active Appearance Model (AAM) to locate and segment primary facial features (e.g., eyes, nose, and mouth). Then, Laplacian-of-Gaussian (LoG) and morphological operators are used to detect facial marks. Experimental results based on FERET (426 images, 213 subjects) and Mugshot (1,225 images, 671 subjects) databases show that the use of facial marks improves the rank-1 identification accuracy of a state-of-the-art face recognition system from 92.96% to 93.90% and from 91.88% to 93.14%, respectively.

The variation caused by aging has not received adequate attention compared with pose, lighting, and expression variations. Aging is a complex process that affects both the 3D shape of the face and its texture (e.g., wrinkles). While the facial age modeling has been widely studied in computer graphics community, only a few studies have been reported in computer vision literature on age-invariant face recognition. We propose an automatic aging simulation technique that can assist any existing face recognition engine for aging-invariant face recognition. We learn the aging patterns of shape and the corresponding texture in 3D domain by adapting a 3D morphable model to the 2D aging database (public domain FG-NET). At recognition time, each probe and all gallery images are modified to compensate for the age-induced variation using an intermediate 3D model deformation and a texture modification, prior to matching. The proposed approach is evaluated on a set of age-separated probe and gallery data using a state-of-the-art commercial face recognition engine, FaceVACS. Use of 3D aging model improves the rank-1 matching accuracy on FG-NET database from 28.0% to 37.8%, on average.

Face recognition in video is being actively studied as a covert method of human identification in surveillance systems. Identifying human faces in video is a difficult problem due to the presence of large variations in facial pose and lighting, and poor image resolution. However, by taking advantage of the diversity of the information contained in video, the performance of a face recognition system can be enhanced. In this work we explore (a) the adaptive use of multiple face matchers in order to enhance the performance of face recognition in video, and (b) the possibility of appropriately populating the database (gallery) in order to succinctly capture intra class variations. To extract the dynamic information in video, the facial poses in various frames are explicitly estimated using active appearance model (AAM) and a factorization based 3D face reconstruction technique. We also estimate the motion blur using discrete cosine transformation (DCT). Our experimental results on 204 subjects in CMU's face-in-action (FIA) database show that the proposed recognition method provides consistent improvements in the matching performance using three different face matchers.