This paper presents an automatic approach to segment 3-D hand trajectories and transcribe phonemes based on them, as a step towards recognizing American sign language (ASL).We first apply a segmentation algorithm which detects minimal velocity and maximal change of directional angle to segment the hand motion trajectory of naturally signed sentences. This yields over-segmented trajectories, which are further processed by a trained naive Bayesian detector to identify true segmented points and eliminate false alarms. The above segmentation algorithm yielded 88.5% true segmented points and 11.8% false alarms on unseen ASL sentence samples. These segmentation results were refined by a simple majority voting scheme, and the final segments obtained were used to transcribe phonemes for ASL. This was based on clustering PCA-based features extracted from training sentences. We then trained hidden Markov models (HMMs) to recognize the sequence of phonemes in the sentences. On the 25 test sentences containing 157 segments, the average number of errors obtained was 15.6.

This paper presents work towards recognizing facial expressions that are used in sign language recognition. Facial features are tracked to effectively capture temporal visual cues on the signer's face during signing. A Bayesian framework is proposed as a feedback mechanism to the Kanade-Lucas-Tomasi (KLT) tracker for reliably tracking facial features in the presence of head motions and temporary occlusions by hand. This mechanism relies on a set of face shape subspaces learned by probabilistic principal component analysis with an update scheme to adapt to persons with different face shapes. The results show that the proposed tracker can track facial features with large head motions, substantial facial deformations, and temporary facial occlusions by hand. The tracked results were input to a recognition system comprising HMMs and a NN to recognize four common American sign language facial expressions.

Deaf people use facial expressions as a non-manual channel for conveying grammatical information in sign language. Tracking facial features using the Kanade-Lucas-Tomasi (KLT) algorithm is a simple and effective method toward recognizing these facial expressions, which are performed simultaneously with head motions and hand signs. To make the tracker robust under these conditions, a Bayesian framework was developed as a feedback mechanism to the KLT tracker. This mechanism relies on a set of face shape sub-spaces learned by probabilistic principal component analysis. An update scheme was utilized to modify these subspaces and adapt to persons with different face shapes. The result shows that the proposed system can track facial features with large head motions, substantial facial deformations, and temporary face occlusions by hand.

In this paper, we propose a method to reduce the false alarm rate or alternatively to improve the detection rate of a local detector for individuals within dense crowds. The detected windows from a Viola-type head detector are processed in a second pass by a cascade of boosted classifiers working with Haar-like features to improve performance. The latter classifier uses color bin images, constructed from normalized rg color histograms of detected windows. Experimental results show a reduction in false alarm rate from 35.9% obtained by the basic detector to 23.9% after the second pass with our approach. This high reduction in false alarm rate was accompanied by only a small reduction in true detections from 87.3% to 82.5%.

Recently, recovery of non-rigid structure by the factorization algorithms have received attention in the literature. The factorization algorithm decomposes the feature points over the given image sequence into motion of the camera and 3D shape bases. The non-rigid structure can be represented by the linear combination of the 3D shape bases. Although the closed-form solution of the non-rigid factorization algorithm is proven, the algorithm is sensitive to noise. In this paper, we propose a batch algorithm to recover multiple non-rigid structures from subsets of the data. Then, we introduce a set of non-linear shape constraints to optimize the recovered non-rigid structures. Synthetic data and real data were used in the experiments. The experimental results showed that the new factorization algorithm gives significant improvement than the original algorithm. With noisy data, the new algorithm is more robust and more accurate in recovering non-rigid structure. 1

This paper features two novel approaches to gait recognition; one is frontal motion analysis, using a single camera. This allows the use of other biometrics easily. Second is analysing gait using of nonlinear dynamics of time series, normally used in chaos theory, for classification. A set of point light sources attached to various points of a walking person allows the walker to be identified. Phase-space analysis of trajectories of these moving light displays (MLDs) provides sufficient information for identification of people by their gait. Using chaotic measures to identify humans by their gait sets a significant precedent

A novel similarity measure for edge-detection that is robust to varying luminance and contrast is presented. It incorporates a regularization term and employs directional FIR edge filters with hyperbolic tangent profiles to ensure improved noise performance and edge localization compared to classical methods.

For applications such as behavior recognition it is important to maintain the identity of multiple targets, while tracking them in the presence of splits and merges, or occlusion of the targets by background obstacles. Here we propose an algorithm to handle multiple splits and merges of objects based on dynamic programming and a new geometric shape matching measure. We then cooperatively combine Kalman filter-based motion and shape tracking with the efficient and novel geometric shape matching algorithm. The system is fully automatic and requires no manual input of any kind for initialization of tracking. The target track initialization problem is formulated as computation of shortest paths in a directed and attributed graph using Dijkstra's shortest path algorithm. This scheme correctly initializes multiple target tracks for tracking even in the presence of clutter and segmentation errors which may occur in detecting a target. We present results on a large number of real world image sequences, where upto 17 objects have been tracked simultaneously in real-time, despite clutter, splits, and merges in measurements of objects. The complete tracking system including segmentation of moving objects works at 25 Hz on 352times288 pixel color image sequences on a 2.8-GHz Pentium-4 workstation

In this paper, we present a new scheme to robustly estimate the head pose from either video sequence or individual images. Developed from ISOMAP, we learn a person-independent and nonlinear embedding space (we call it a 2-D feature space) for different poses. A nonlinear interpolation is proposed to map new sequences or images into the 2-D feature space. Especially for video sequences, we propose an adaptive local fitting technique to filter unreasonable mappings. By exploring the intrinsic characteristics, we further estimate the head pose of that sequence or image. Experiments reported in this paper showed robust results.

Research in automatic analysis of sign language has largely focused on recognizing the lexical (or citation) form of sign gestures as they appear in continuous signing, and developing algorithms that scale well to large vocabularies. However, successful recognition of lexical signs is not sufficient for a full understanding of sign language communication. Nonmanual signals and grammatical processes which result in systematic variations in sign appearance are integral aspects of this communication but have received comparatively little attention in the literature. In this survey, we examine data acquisition, feature extraction and classification methods employed for the analysis of sign language gestures. These are discussed with respect to issues such as modeling transitions between signs in continuous signing, modeling inflectional processes, signer independence, and adaptation. We further examine works that attempt to analyze nonmanual signals and discuss issues related to integrating these with (hand) sign gestures. We also discuss the overall progress toward a true test of sign recognition systems--dealing with natural signing by native signers. We suggest some future directions for this research and also point to contributions it can make to other fields of research. Web-based supplemental materials (appendicies) which contain several illustrative examples and videos of signing can be found at www.computer.org/publications/dlib.

Video-based surveillance systems have a wide range of applications for traffic monitoring, as they provide more information as compared to other sensors. In this paper, we present a rule-based framework for behavior and activity detection in traffic videos obtained from stationary video cameras. Moving targets are segmented from the images and tracked in real time. These are classified into different categories using a novel Bayesian network approach, which makes use of image features and image-sequence-based tracking results for robust classification. Tracking and classification results are used in a programmed context to analyze behavior. For behavior recognition, two types of interactions have mainly been considered. One is interaction between two or more mobile targets in the field of view (FoV) of the camera. The other is interaction between targets and stationary objects in the environment. The framework is based on two types of a priori information: 1) the contextual information of the camera's FoV, in terms of the different stationary objects in the scene and 2) sets of predefined behavior scenarios, which need to be analyzed in different contexts. The system can recognize behavior from videos and give a lexical output of the detected behavior. It also is capable of handling uncertainties that arise due to errors in visual signal processing. We demonstrate successful behavior recognition results for pedestrian-vehicle interaction and vehicle-checkpost interactions.

In this paper, we propose a real time system for tracking targets across blind regions of multiple cameras with non-overlapping fields of views (FOVs) using camera topology, and targets’ motion and shape information. Kalman filters are used to robustly track each target’s shape and motion in each camera view and the common ground plane view composed of all camera views. The target’s track in the blind region between cameras is obtained using Kalman filter predictions. For multi-camera correspondence matching we compute the Gaussian distributions of the tracking parameters across cameras for the target motion and position in the ground plane view. Matching of targets across camera views uses a graph based track initialization scheme, which accumulates information from occurrences of target in several consecutive frames of the video. Probabilistic matching is carried out by using the track parameters for new tracks obtained from the graph in a camera view with the parameters of the terminated tracks learnt by Kalman filters in the other camera views and ground plane view. We obtain 85% accuracy for corresponding matching while tracking vehicles observed from two cameras monitoring a highway.

This paper describes a real-time infrared-based system for eye gaze tracking. Most of the existing gaze tracking systems require a cumbersome calibration process every time an user operates the system and work well only with small head movement. Our system is simpler as it requires only one-time calibration for each user and the calibration data is stored for subsequent reuse. Besides that, our system can perform robust and accurate gaze estimation under rather significant head pose variations. This is made possible by a gaze calibration procedure which maps the pupil and glint parameters to screen coordinates using the Radial Basis Function Neural Network (RBFNN). Our system currently has a spatial gaze resolution of about 3 degrees and it operates at 25 fps with an average accuracy of 90% for gaze estimation.

This paper presents the use of a PCA based approach to segment cells from RGB light microscope images. The proposed segmentation is accurate and robust under uneven illumination, lighting variation, and noise. Principal component analysis (PCA) is first applied to the RGB color bands of the image. The image corresponding to the principal component has significantly better contrast over the original image. A set of eigenfilters is then obtained by applying PCA to local neighborhoods of this image. A pair of filters from this set, corresponding to the second and third largest eigenvalues, resembles ramp edge filters with orientations that adapt to the image. These edge filters are used to obtain the edgemap of the image. We define a criterion that enables accurate detection of valid edges of cells while suppressing noise.

Non-manual signals (NMS) are grammatical elements in sign languages. They may convey information that reinforces or is additional to the hand signing. NMS are similar to facial expressions except that, unlike spontaneous emotions, NMS are deliberate gestures. This paper explores the use of Independent Component Analysis (ICA) and Gabor wavelet networks (GWNs) for recognising 3 upper face and 3 lower face expressions related to NMS. Independent component analysis and Gabor wavelet networks were compared as representations for these facial signals. Both representations provided good recognition performance. The method of using GWNs with 116 wavelets outperformed ICA (85.3% and 93.3% for upper and lower face respectively, compared to 78.7% and 92% for ICA). However, the GWN method is computationally more expensive.

Signs produced by gestures (such as in American Sign Language) can have a basic meaning coupled with additional meanings that are like layers added to the basic meaning of the sign. These layered meanings are conveyed by Systematic temporal and spatial modification of the basic form of the gesture. The work reported in this paper seeks to recognize temporal and spatial modifiers of hand movement and integrates them with the recognition of the basic meaning of the sign. To this end, a Bayesian network framework is explored with a simulated vocabulary of 4 basic signs which give rise to 14 different combinations of basic meanings and layered meanings. In this paper we approached the problem of deciphering layered meanings by drawing analogies to the gesture parameters in Parametric HMM which represent systematic spatial modifications to gesture movement. Various Bayesian network structures were compared for recognizing the signs with layered meanings. The best performing network yielded 85.5% accuracy.

We present a real time system for detecting people and interpreting some of their postures. The system runs reliably on different people and under different lighting conditions. It uses an inexpensive PC desktop camera to acquire color monocular video sequences. Two fundamental problems of such a CCD camera for background-foreground segmentation are analyzed and a solution to them has been proposed. The system locates humans by extracting their profile using a physics based approach, which is robust in the presence of noise. Next it uses a pattern recognition algorithm to detect and interpret the posture of a human. The system can be used as a peripheral user interface, in pervasive computing applications, surveillance, monitoring people activities in an indoor environment, visual computer games, smart screen savers etc. We have already used this system for several real-world interactive video game applications.

This paper describes a modular approach to gesture recognition. The complex task of gesture recognition from image sequences was decomposed by first identifying the hand pose in individual frames. The pose information was then incorporated with hand motion to recognize gestures. Independent recognition modules were devised for different subtasks. A radial basis function (RBF) neural network was trained to recognize static hand poses. Inputs to the RBF network were feature vectors extracted from segmented 2D binary images of the hand. The pose recognition results of using Zernike moments and Fourier descriptors as the feature vectors were compared, and it was found that Fourier descriptors were superior in terms of computational speed. Combined outputs from a set of recurrent neural networks (RNN) and hidden Markov model (HMM) chains were used to recognize gestures from the temporal sequence of pose classifier outputs. The combined classifier achieved a recognition rate of 86.8%. In addition, we illustrate that the inclusion of an intermediate pose classification stage is advantageous for recognition and training speed.

We introduce a highly scalable video compression system for very low bit-rate videoconferencing and telephony applications around 10-30 kbits/s. The video codec first performs a motion-compensated three-dimensional (3-D) wavelet (packet) decomposition of a group of video frames, and then encodes the important wavelet coefficients using a new data structure called tri-zerotrees (TRI-ZTR). Together, the proposed video coding framework forms an extension of the original zero tree idea of Shapiro (1992) for still image compression. In addition, we also incorporate a high degree of video scalability into the codec by combining the layered/progressive coding strategy with the concept of embedded resolution block coding. With scalable algorithms, only one original compressed video bit stream is generated. Different subsets of the bit stream can then be selected at the decoder to support a multitude of display specifications such as bit rate, quality level, spatial resolution, frame rate, decoding hardware complexity, and end-to-end coding delay. The proposed video codec also allows precise bit rate control at both the encoder and decoder, and this can be achieved independently of the other video scaling parameters. Such a scheme is very useful for both constant and variable bit rate transmission over mobile communication channels, as well as video distribution over heterogeneous multicast networks. Finally, our simulations demonstrated comparable objective and subjective performance when compared to the ITU-T H.263 video coding standard, while providing both multirate and multiresolution video scalability.

The widespread use of block-based interframe motion estimation for video sequence compression in both MPEG and H.263 standards is due to its effectiveness and simplicity of implementation. Nevertheless, the high computational complexity of the full-search algorithm has motivated a host of suboptimal but faster search strategies. A popular example is the three-step search (TSS) algorithm. However, its uniformly spaced search pattern is not well matched to most real-world video sequences in which the motion vector distribution is nonuniformly biased toward the zero vector. Such an observation inspired the new three-step search (NTSS) which has a center-biased search pattern and supports a halfway-stop technique. It is faster on average, and gives better motion estimation as compared to the well-known TSS. Later, the four-step search (4SS) algorithm was introduced to reduce the average case from 21 to 19 search points, while maintaining a performance similar to NTSS in terms of motion compensation errors. We propose a novel unrestricted center-biased diamond search (UCBDS) algorithm which is more efficient, effective, and robust than the previous techniques. It has a best case scenario of only 13 search points and an average of 15.5 block matches. This makes UCBDS consistently faster than the other suboptimal block-matching techniques. This paper also compares the above methods in which both the processing speed and the accuracy of motion compensation are tested over a wide range of test video sequences.

The author investigated automatic extraction of left ventricular contours from cardiac magnetic resonance imaging (MRI) studies. The contour extraction algorithms were based on active contour models, or snakes. Based on cardiac MR image characteristics, the author suggested algorithms for extracting contours from these large data sets. The author specifically considered contour propagation methods to make the contours reliable enough despite noise, artifacts, and poor temporal resolution. The emphasis was on reliable contour extraction with a minimum of user interaction. Both spin echo and gradient echo studies were considered. The extracted contours were used for determining quantitative measures for the heart and could also be used for obtaining graphically rendered cardiac surfaces.

Registering volumes that have been deformed with respect to each other involves recovery of the deformation. A 3-D elastic matching algorithm has been developed to use surface information for registering volumes. Surface extraction is performed in two steps: extraction of contours in 2-D image planes using active contours, and forming triangular patch surface models from the stack of 2-D contours. One volume is modeled as being deformed with respect to another goal volume. Correspondences between surfaces in the two image volumes are used to warp the deformed volume towards its goal. This process of contour extraction, surface formation and matching, and warping is repeated a number of times, with decreasing image volume stiffness. As the iterations continue the stretched volume is refined towards its goal volume. Registration examples of deformed volumes are presented.

It is shown how multiresolution representations can be used for filter design and implementation. These representations provide a coarse frequency decomposition of the image, which forms the basis for two filtering techniques. The first method, based on image pyramids, is used for approximating the convolution of an image with a given mask. In this technique, a filter is designed using a least-squares procedure based on filters synthesized from the basic pyramid equivalent filters. The second method is an adaptive noise reduction algorithm. An optimally filtered image is synthesized from the multiresolution levels, which in this case are maintained at the original sampling density. Individual pixels of the image representation are linearly combined under a minimum mean square error criterion. This uses a local signal-to-noise ratio estimate to provide the best compromise between noise removal and resolution loss.>

In this paper, we consider application of 2-D stochastic models discussed in [1] to develop noncausal FIR filters for restoration of images degraded by additive white noise. The semicausal model of [1] is used to design masks for edge extraction from the noisy images. The results presented here indicate that good restorations and robust edge detection are possible using relatively simple algorithms.

In this paper we consider an application of spectral estimation to adaptive restoration of images degraded by additive white noise. Three adaptive techniques of restoration are compared with a non-adaptive technique. In the non-adaptive technique, the whole image is Wiener filtered by assuming a correlation model for the signal. In the adaptive methods, a) the spectral shape of the signal is kept constant, but the variance is adaptively estimated, b) the power spectrum of each block is adaptively estimated, c) the data is adaptively decorrelated in one dimension and filtered along the other.

A new method of image coding by autoregressive (AR) synthesis is presented. The physics of image formation suggests that an image may be considered as a power spectrum. Using this formulation a Cosine transform of the sampled image is shown to yield a set of autocorrelations. These are used to find an equivalent AR model whose parameters are encoded for transmission. Compared to conventional Cosine transform coding, this method is shown to give superior resolution and is shown to suppress the "block-effects" present in block-by-block transform coding methods. Distinction between this method and linear predictive coding (LPC) used for speech data compression is made. Extensions and examples for two dimensional images are given.