Multi-robot navigation in complex environments relies on inter-robot communication and mutual observation for situational awareness. This paper studies the multi-robot navigation problem in unknown environments with line-ofsight (LoS) connectivity constraints. While previous works are limited to known environment models to derive the LoS constraints between robots, this paper eliminates such requirements by directly formulating the LoS constraints from realtime LiDAR scans, adopting techniques in point cloud visibility analysis. Based on that, we propose a novel LoS-distance metric to quantify both the urgency and sensitivity of losing LoS between robots considering their potential movements. Moreover, to address the imbalanced urgency of losing LoS between two robots, we design a fusion function to capture the overall urgency while generating gradients that facilitate robots' collaborative behavior to maintain LoS. The team connectivity is guaranteed by encoding the LoS constraints into a potential function that preserves the positivity of the Fiedler eigenvalue of robots' underlying graph. Finally, we establish a LoS-constrained exploration framework integrating the proposed connectivity controller. We showcase its applications in multi-robot exploration in complex unknown environments, where robots can always maintain the LoS connectivity through distributed sensing and communication while collaboratively exploring unknown environments. Our implementations are available at https://github.com/bairuofei/LoS_constrained_navigation.

The development of unmanned aerial vehicles (UAVs) with extended maneuverability has unlocked new applications such as complex inspection tasks at height. In this work, we introduce the Dragonfly drone, a novel tilt-rotor body-morphing UAV, capable of altering its shape and orientation without compromising its position tracking. Unlike most existing UAV designs that only target at decoupling position and orientation control, Dragonfly can also perform unique body-morphing in flight, featuring all six degrees of freedom in every morphology. This enables navigation into tight gaps with irregular shapes, conforming to obstacles of varying geometries, and maintaining physical contact with uneven surfaces. Such capabilities make our design particularly effective for complex inspection tasks at height, such as pipe or bridge inspection. Our contributions include the mechanical design of the system, the modeling and control strategies employed, and the realrobot experiments with a prototype platform. See Dragonfly drone in action: https://youtu.be/YxoV_Qt_5XE.

This paper investigates theresilientmulti-robot efficient search problem (R-MuRES), which aims at coordinating multiple robots to detect a ‘non-adversarial’ moving target with the minimal expected time. One unique characteristic of R-MuRES among others is the possibility of individual robot's malfunction and withdrawal from the team during task execution, which results in avariablenumber of searchers in the deployment phase and entails that the possibility of team member failures must be considered during the planning stage, particularly in the training phase. We propose a resilient value function factorization (R-FAC) paradigm, which constructs the central value function from individual ones in a resilient manner, taking into account individual robots' failures, and ensures that the constructed central value function has the minimal mean squared temporal difference error across various team compositions. R-FAC stipulates that the individual global maximum (IGM) principle is satisfied for whichever team configuration and thus any functioning robot contributes positively to the remaining team, as long as it executes the greedy policy with respect to the factorized individual value function. Subsequently, we introduce thevariationalvalue decomposition network (V2DN) as one of the instantiated R-FAC algorithms. V2DN employs the$\log$-sum-$\exp$mechanism to construct the central value function from individual ones, enabling it to take a varying number of robots' individual value functions as inputs. Then, we explain why, specifically for the multi-robot search task, the$\log$-sum-$\exp$mechanism is superior to the brute-force summation operation used in the canonical value decomposition network (VDN), and compare V2DN with state-of-the-art MuRES solutions as well as the vanilla VDN algorithm in two canonical MuRES testing environments and show that it achieves the best resiliency score when one or several individual robots quit the team during task execution. Furthermore, we validate V2DN with a real multi-robot system in a self-constructed indoor environment as the proof of concept.

This paper considers the multi-robot active graph exploration problem, where robots need to collaboratively cover a graph environment while maintaining reliable pose estimation in collaborative Simultaneous Localization and Mapping (SLAM). Considering both objectives presents challenges for multi-robot pathfinding, as it involves the expensive covariance propagation for SLAM uncertainty evaluation, especially when considering various combinations of robots’ paths. To reduce the computational complexity, we propose an efficient two-stage strategy where exploration paths are first generated for quick coverage, and then enhanced by adding informative loop-closing actions along the paths for reliable pose estimation. We formulate the latter problem as a non-monotone submodular maximization problem by relating SLAM uncertainty with pose graph topology, which (1) facilitates a more efficient evaluation of SLAM uncertainty than covariance inference, and (2) allows the employment of approximation algorithms in submodular optimization to provide suboptimality guarantees. We further introduce ordering heuristics to improve the objective values while preserving the optimality bound. Simulation experiments over randomly generated graph environments verify the effectiveness of our methods to achieve quick coverage and enhanced pose graph reliability, and benchmark the performance of the approximation algorithms and the greedy-based algorithm in the loop edge selection problem. Our implementations will be open-source at https://github.com/bairuofei/CGE.

Recent advances in multi-agent reinforcement learning (MARL) have significantly enhanced cooperation capabilities within multi-robot teams. However, the application to heterogeneous teams poses the critical challenge of combinatorial generalization—adapting learned policies to teams with new compositions of varying sizes and robots capabilities. This challenge is paramount for dynamic real-world scenarios where teams must swiftly adapt to changing environmental and task conditions. To address this, we introduce a novel transformer-based MARL method for heterogeneous multirobot cooperation. Our approach leverages graph neural networks and self-attention mechanisms to effectively capture the intricate dynamics among heterogeneous robots, facilitating policy adaptation to team size variations. Moreover, by treating robot team decisions as sequential inputs, a capability-oriented decoder is introduced to generate actions in an auto-regressive manner, enabling decentralized decision-making that tailored each robot’s varying capabilities and heterogeneity type. Furthermore, we evaluate our method across two heterogeneous cooperation scenarios in both simulated and real-world environments, featuring variations in team number and robot capabilities. Comparative results reveal our method’s superior generalization performance compared to existing MARL methodologies, marking its potential for real-world multi-robot applications.

Unmanned aerial vehicles (UAVs) for contact-based tasks at height can greatly improve the safety of the human workers involved. However, performing contact-based tasks with typical under-actuated UAVs is non-trivial. Due to their coupled translational and rotational dynamics and their limited station-keeping performance under physical disturbances, it is difficult to maintain precise and consistent contact. We address these problems in the context of physical interaction with vertical, cylindrical target objects, such as trees. We present a novel UAV design with a pair of tilt-rotors and a landing gear that can reconfigure into a front-mounted, two-fingered gripper. While the tilt-rotors provide horizontal force toward the target object without pitching the UAV forward, the reconfigurable landing gear enables the UAV to obtain support from the target object. Such support results in an approximately 80% improvement in position- and heading-keeping performance. Moreover, the landing gear is designed as a cable-driven under-actuated system, which requires only one actuator to control both the reconfiguration and the grasping (i.e., five degrees of freedom in total). Such a minimalist design helps keep the UAV power consumption for interactions low. This marks progress towards safe, high-precision physical interaction against vertical, cylindrical target objects. Our UAV in action: https://youtu.be/D-65vldox_A.

Simulation has become a powerful tool in robotics, enabling safe and rapid testing of robots prior to deployment. It also holds great potential to aid in real-time decision-making by emulating the surroundings on-the-fly using onboard perception modules. In this work, we introduce a simulation-based decision-making framework for the safe autonomous landing of a multirotor unmanned aerial vehicle (UAV) over difficult terrains. Our framework consists of first obtaining a terrain model using onboard sensors. It then assesses a range of possible landing configurations using a digital twin of the UAV in a high-fidelity physics-based simulation with PX4 firmware running in-the-loop. Finally, it identifies the safest simulated landing configuration and uses this to perform the landing in the real world. We validate our framework over complex, previously unseen terrains with a real multirotor. Our framework matches human-level judgment on how best to land on multiple instances of complex terrain and perform safe landings in real-world tests. This can be seen at https://youtu.be/G3nzaBFDHBY. These results show the potential of our approach for aerial robotics applications involving physical interaction.

This article investigates the multirobot efficient search (MuRES) for a nonadversarial moving target problem from the multiagent reinforcement learning (MARL) perspective. MARL is deemed as a promising research field for cooperative multiagent applications. However, one of the main bottlenecks of applying MARL to the MuRES problem is the nonstationarity introduced by multiple learning agents. With learning agents simultaneously updating their policies, the environment cannot be modeled as astationaryMarkov decision process, which results in the inapplicability of fundamental reinforcement learning techniques such as deep$Q$-network and policy gradient (PG). In view of that, we adopt the centralized training and decentralized execution scheme and thereby propose a cross-entropy regularized policy gradient (CE-PG) method to train the learning agents/robots. We let the robotscommitto a predetermined policy during execution, collect the trajectories, and then perform centralized training for the corresponding policy improvement. In this way, the nonstationarity problem is overcome, in that the robots do not update their policies during execution. During the centralized training stage, we improve the canonical PG method to consider the interactions among robots by adding a cross-entropy regularization term, which essentially functions to “disperse” the robots in the environment. Extensive simulation results and comparisons with state of the art show CE-PG's superior performance, and we also validate the algorithm with a real multirobot system in an indoor moving target search scenario.

Lower limb exoskeletons offer support for patients suffering from mobility disorders due to injury, stroke, etc. But these devices are not used in day-to-day life and environments due to their limited human-computer interface to perceive and handle different terrains and tasks. In this paper, we introduce a simple vision-based environment perception pipeline for lower- limb exoskeletons for obstacle crossing tasks. The proposed pipeline consists of three stages, namely, ground plane and obstacle detection, estimating obstacle location and dimensions, and obstacle tracking. To reduce noisy artifacts and reliably detect obstacles, we propose a similarity metric based on color, gradient orientation, and 2D surface normal. Depth map of the detected obstacle region is utilized for estimating the obstacle location and dimensions. Also, we consider two obstacle tracking modes for obstacle crossing, visual tracking using a RGB-D camera and positional tracking using a SLAM camera. The proposed vision-based perception pipeline is integrated with an exoskeleton, where we propose a control scheme that can vary step length adaptively to successfully cross detected obstacles. We conduct offline and online experiments to validate the proposed perception pipeline and provide insights on the same. Our experiments show that the proposed pipeline allows exoskeletons to understand their environment and successfully cross obstacles.

Forecasting future trajectories of multiple pedestrians in a crowded environment is a challenging problem due to the complex interactions among the pedestrians. The interactions can be asymmetric and their influences may vary over time. Moreover, each pedestrian can exhibit different behavior at any given time and context and thus they may have multiple future possible trajectories. In this work, we present a Gated Graph Convolutional Network (GatedGCN) based trajectory prediction model that explicitly deal with the asymmetric influences among the adjacent pedestrians through edge-wise gating mechanism. Through GatedGCN only, an overall average improvement of 16% and 18% was achieved on the two performance metrics over the state-of-the-art trajectory forecasting methods. Next, we tackle the problem of learning multi-modal distributions of each pedestrian trajectory using variational auto-encoders (VAEs). However, trajectories sampled from the learned distribution usually ignore the factors affecting the pedestrian motion such as collision avoidance and the target destination. While many of the existing approaches focus on learning such factors during the trajectory encoding process, we proposed a novel self-critical learning approach based on Actor-Critic framework to learn such factors during the trajectory generation process. We empirically show that our method creates fewer number of collisions than the existing methods on popular trajectory forecasting benchmarks.

The recent years have seen the increasing importance of cognitive models for improved robot navigation. In this paper, a novel cognitive navigation package, which consists of topometric map representation and a three-level path planner, is proposed. The topometric maps are built from architectural floor plans with additional features within a limited number of selected regions. The inherent discrepancies between floor plans and the robot’s actual sensory readings are handled by the three-level path planner. The unique feature of this approach is that accurate localization is only required at the selected regions. At the other regions the robot will rely on the guiding directions towards the goal rather than on its accurate position on the map for navigation. Experiments show that our approach can endow a robot with capability for semantic interpretation and localization in unseen and dynamic environments.

Vision-based vehicle detection under bad weather conditions is still a challenging problem. Adhesive raindrops on windshield have been known to diffract light and distort parts of the scene behind them. In this paper, we propose a Vehicle-Aware Generative Adversarial Networks (VAGAN) to improve vehicle detection from rain images. We train a Generative Adversarial Network (GAN) on image pairs, each comprising of an original rain image and one that is manually labelled with colored bounding boxes of the vehicles therein. The latter represents a fake version of the original image emphasizing the regions of interest. To further enhance vehicle awareness, we exploit the fact that the vehicle rear lights are usually turned on during rainy conditions to compute a saliency map of image, and use it formulate a background preserving constrain on the learning vehicle loss function. We show that this novel adversarial framework is able to generate new images with colored regions overlay over vehicles, hence effectively learning to differentiate image background from vehicles. The final vehicle detection in the generated images is not affected by image translation noise because we can simply use color segmentation to localize the vehicles. Experimental results on a large dataset show that our approach is an effective way to solve vehicle detection in rain images, achieving state-of-the-art performance.

Multi-human tracking in the crowded environment is a challenging problem due to occlusions, pose change, viewpoint variation and cluttered background. In this work, we propose a robust feature learning for tracking-by-detection methods based on second-order attention network that can capture higher-order relationships between salient features at the early stages of Convolutional Neural Network (CNN). Guided Second-Order Attention Network (GSAN) that, unlike the existing attention learning methods which are weakly-supervised, uses a supervisory signal based on the quality of the self-learned attention maps. More specifically, GSAN looks into the attended maps of a person having the highest confidence and supervise itself to look into the correct regions in the images of the person. Attention maps learned this way are spatially aligned and thus robust to camera-view changes and body pose variations. We verify the effectiveness of our approach by comparing with the state-of-the-art methods on challenging person re-identification and multi object tracking (MOT) datasets.

A new approach for autonomous exploration in an unknown scenario based on the concept of frontiers is proposed in this paper. Exploration frontiers introduced by [4], are the regions on the boundary between open space and unexplored space. A mobile robot is able to construct its map by adding new space and moving to unvisited frontiers until the entire environment has been explored. However, the original frontier strategy, suffering from local minima, only considers distance and size of unknown spaces, resulting in low exploration efficiency in complex environments. By making use of the robot heading information using wheel odometry and coarse graph representation of the environment, the modified exploration method is able to balance the mapping coverage and time expenditure to a greater extent. The proposed method is experimentally verified on a mobile platform, exploring a real-world office environment cluttered with a variety of obstacles.

While much effort has been devoted to deep learning object detection, relatively limited attention has been paid to object detection in bad weather, e.g. rain, snow or haze. In heavy rain, the raindrop on the front windshield can make it difficult to detect object from an in-car camera. The conventional way to cope with this has been to use radar as the main detection sensor. However, radar is highly susceptible to false positives. Furthermore, many entry level radar sensors only return the centroid of each detected object, rather than its size and extent. In addition, due to lack of texture input, radar cannot discriminate a vehicle from a non-vehicle object, e.g. roadside pole. This motivates us to detect vehicle by fusing radar and vision. In this paper, we first calibrate the radar and camera with respect to the ground plane. The radar detections are then projected to the camera image for target width estimation. Empirical evaluation on a large database shows that there is a natural synergy in both sensors, as the image based estimation is found to be greatly facilitated by the accuracy of the radar detection.

Spatial verification is a key step to remove outliers for accurate feature matching in visual place recognition. In this paper, we propose a novel method for outlier detection using a hierarchical spatial verification scheme. Given a set of putative correspondences between a pair of images, we convert the matching problem into a 4D transformation space and identify promising similarity transformations using Hough voting. In the hierarchical scheme, we first use a hypothesize-and-verify technique to identify groups of correspondences according to each similarity transformation. Second, the group with the largest number of correspondences serves as a standard to subsequently remove outliers in other groups by explicit geometric consistency checking. We have compared the proposed method with the state-of-the-art solutions on five popular public datasets to show that our method has better performance in place recognition and loop closure detection.

A robot might get lost due to abrupt wheel slippage, unsteady movements on uneven floors, collision with obstacles, blocked perception sensors or kidnapping. When this happens, the robot has to recover by itself. This is necessary but has not been satisfactorily resolved. In this paper, a generic lost robot self-recovery framework is proposed. It has self-exploration capability assisted by an efficient place recognition module using hybrid topological-metric maps. The metric map is used for path planning and navigation while the topological map is used for re-localization when lost. As soon as the robot detects that it is lost, self exploration is activated and starts to explore while performing place recognition using the topological map. Once the place is re-identified, the proximate global location is obtained and the robot performs fine localization using the metric map, recovery itself and continue to its destination. The proposed system has been implemented on a mobile robot operating in a typical office environment. Experiments conducted show a robot can be efficiently and reliably recover itself when it gets lost within or outside of the map.

Existing subspace clustering methods typically employ shallow models to estimate underlying subspaces of unlabeled data points and cluster them into corresponding groups. However, due to the limited representative capacity of the employed shallow models, those methods may fail in handling realistic data without the linear subspace structure. To address this issue, we propose a novel subspace clustering approach by introducing a new deep model-Structured AutoEncoder (StructAE). The StructAE learns a set of explicit transformations to progressively map input data points into nonlinear latent spaces while preserving the local and global subspace structure. In particular, to preserve local structure, the StructAE learns representations for each data point by minimizing reconstruction error w.r.t. itself. To preserve global structure, the StructAE incorporates a prior structured information by encouraging the learned representation to preserve specified reconstruction patterns over the entire data set. To the best of our knowledge, StructAE is one of first deep subspace clustering approaches. Extensive experiments show that the proposed StructAE significantly outperforms 15 state-of-the-art subspace clustering approaches in terms of five evaluation metrics.

In this paper, we recast the subspace clustering as a verification problem. Our idea comes from an assumption that the distribution between a given sample x and cluster centers Omega is invariant to different distance metrics on the manifold, where each distribution is defined as a probability map (i.e. soft-assignment) between x and Omega. To verify this so-called invariance of distribution, we propose a deep learning based subspace clustering method which simultaneously learns a compact representation using a neural network and a clustering assignment by minimizing the discrepancy between pair-wise sample-centers distributions. To the best of our knowledge, this is the first work to reformulate clustering as a verification problem. Moreover, the proposed method is also one of the first several cascade clustering models which jointly learn representation and clustering in end-to-end manner. Extensive experimental results show the effectiveness of our algorithm comparing with 11 state-of-the-art clustering approaches on four data sets regarding to four evaluation metrics.

It is believed that animals sense the environment by encoding the spatial information into an internal representation. Grid cells in the rat entorhinal cortex are found with periodic firing fields and have been hypothesized as the basis for the ‘cognitive map’. This paper presents a grid-cell based computational model for cognitive map building together with an analysis of the learning from grid cells to place cells. Experiment results demonstrate that the learning from grid cells to place cells plays an important role affecting the performance of map building and that the proposed computational model provides an alternative approach for robot mapping.

For reliable human-robot interaction, the robot must know the person's action in order to plan the appropriate way to interact or assist the person. As part of the pre-processing stage of action recognition, the robot also needs to recognize the various body parts and posture of the person. But estimation of posture and body parts is challenging due to the articulated nature of the human body and the huge intra-class variations. To address this challenge, we propose two schemes using Hierarchical-ELM (H-ELM) for posture detection into either upright or non-upright posture. In the first scheme, we follow a whole body detector approach, where a H-ELM classifier is trained on several whole body postures. In the second scheme, we follow a body part detection approach, where separate H-ELM classifiers are detected for each body part. Using the detected body parts a final decision is made on the posture of the person. We have conducted several experiments to compare the performance of both approaches under different scenarios like view angle changes, occlusion etc. Our experimental results show that body part H-ELM based posture detection works better than other proposed framework even in the presence of occlusion.

Body part segmentation and detection in videos is a useful analysis for many computer vision tasks such as action recognition and video search. Conventional methods mainly focus on body part detection assuming upright posture of the human body. Recently, a body part detection framework was proposed to include non-upright postures. This method consists of 2 parts, initial segmentation and computation of body part likelihood score for each segment. In this paper, we propose improvements to this approach. Firstly, we propose a novel motion based body part segmentation using kinematic features to identify segments which undergo similar motion in the video based on a consistency or error measure. Secondly, we replace the Extreme Learning Machine classifier in the original work with deep learning to investigate it's performance. For accurate detection, deep learning requires a lot of training data and it has so far been used only in high resolution images. Here we apply deep learning for body part detection in low resolution cases. We conduct experiments to study and analyse the effect of the improvements proposed.

Recognition of natural emotions from human faces is an interesting topic with a wide range of potential applications, such as human-computer interaction, automated tutoring systems, image and video retrieval, smart environments, and driver warning systems. Traditionally, facial emotion recognition systems have been evaluated on laboratory controlled data, which is not representative of the environment faced in real-world applications. To robustly recognize the facial emotions in real-world natural situations, this paper proposes an approach called extreme sparse learning, which has the ability to jointly learn a dictionary (set of basis) and a nonlinear classification model. The proposed approach combines the discriminative power of extreme learning machine with the reconstruction property of sparse representation to enable accurate classification when presented with noisy signals and imperfect data recorded in natural settings. In addition, this paper presents a new local spatio-temporal descriptor that is distinctive and pose-invariant. The proposed framework is able to achieve the state-of-the-art recognition accuracy on both acted and spontaneous facial emotion databases.

Given a video sequence, the task of action recognition is to identify the most similar action among the action sequences learned by the system. Such human action recognition is based on evidence gathered from videos. It has wide application including surveillance, video indexing, biometrics, telehealth, and human-computer interaction. Vision-based human action recognition is affected by several challenges due to view changes, occlusion, variation in execution rate, anthropometry, camera motion, and background clutter. In this survey, we provide an overview of the existing methods based on their ability to handle these challenges as well as how these methods can be generalized and their ability to detect abnormal actions. Such systematic classification will help researchers to identify the suitable methods available to address each of the challenges faced and their limitations. In addition, we also identify the publicly available datasets and the challenges posed by them. From this survey, we draw conclusions regarding how well a challenge has been solved, and we identify potential research areas that require further work.

This paper describes the gesture recognition system developed by the Institute for Infocomm Research (I2R) for the 2013 ICMI CHALEARN Multi-modal Gesture Recognition Challenge. The proposed system adopts a multi-modal approach for detecting as well as recognizing the gestures. Automated gesture detection is performed using both audio signals and information about hand joints obtained from the Kinect sensor to segment a sample into individual gestures. Once the gestures are detected and segmented, features extracted from three different modalities, namely, audio, 2-dimensional video (RGB), and skeletal joints (Kinect) are used to classify a given sequence of frames into one of the 20 known gestures or an unrecognized gesture. Mel frequency cepstral coefficients (MFCC) are extracted from the audio signals and a Hidden Markov Model (HMM) is used for classification. While Space-Time Interest Points (STIP) are used to represent the RGB modality, a covariance descriptor is extracted from the skeletal joint data. In the case of both RGB and Kinect modalities, Support Vector Machines (SVM) are used for gesture classification. Finally, a fusion scheme is applied to accumulate evidence from all the three modalities and predict the sequence of gestures in each test sample. The proposed gesture recognition system is able to achieve an average edit distance of 0.2074 over the 275 test samples containing 2,742 unlabeled gestures. While the proposed system is able to recognize the known gestures with high accuracy, most of the errors are caused due to insertion, which occurs when an unrecognized gesture is misclassified as one of the 20 known gestures.

This paper summarizes our contribution to the Technicolor Rich Multimedia Retrieval from Input Videos Grand Challenge. We hold the view that semantic analysis of a given news video is best performed in the text domain. Starting with a noisy text obtained from applying Automatic Speech Recognition (ASR), a graph-based approach is then used to enrich the text by propagating labels from visually similar videos culled from parallel (YouTube) News sources. From the enriched text, we next extract salient keywords to form a query to a news video search engine, retrieving a larger corpus of related news video. Compared to a baseline method that only uses the ASR text, significant improvement in precision has been obtained, indicating that retrieval has benefited from the ingestion of the external labels. Capitalizing on the enriched metadata, we find that videos are more amenable to the Wikipedia-based Explicit Semantic Analysis (ESA), resulting in better support for subtopic news video retrieval. We apply our methods to an in-house live news search portal, and report on several best practices.

Most face recognition systems require faces to be detected and localized a priori. In this paper, an approach to simultaneously detect and localize multiple faces having arbitrary views and different scales is proposed. The main contribution of this paper is the introduction of a face constellation, which enables multiview face detection and localization. In contrast to other multiview approaches that require many manually labeled images for training, the proposed face constellation requires only a single reference image of a face containing two manually indicated reference points for initialization. Subsequent training face images from arbitrary views are automatically added to the constellation (registered to the reference image) based on finding the correspondences between distinctive local features. Thus, the key advantage of the proposed scheme is the minimal manual intervention required to train the face constellation. We also propose an approach to identify distinctive correspondence points between pairs of face images in the presence of a large amount of false matches. To detect and localize multiple faces with arbitrary views, we then propose a probabilistic classifier-based formulation to evaluate whether a local feature cluster corresponds to a face. Experimental results conducted on the FERET, CMU, and FDDB datasets show that our proposed approach has better performance compared to the state-of-the-art approaches for detecting faces with arbitrary pose.

The conventional mouth gender recognition is based on a static image and ignore the dynamic information. In this paper, we propose a lip movement gender recognition method to improve the accuracy by exploring the dynamic information while a user is speaking. In order to overcome the difficulty caused by the nonlinear distribution of the lip images, Gausian Mixture Models (GMM) is adopted to represent the lip images. A similarity measure is defined to measure the difference between the successive frames. Gender recognition, as a soft biometric trait, can provide useful information for improving the performance of the speaker recognition systems. The accuracy of voice-based speaker gender recognition is high if the condition of the environment is good. But it will be drastically decreased if the test is conducted in a noisy environment. In this paper, we showed that lip movement, considered as a sequence of mouth images, can provide additional information than mouth alone for recognizing gender Experimental result obtained showed the effectiveness of the proposed method which is comparable to using just the voice information.

Although facial expression plays an important role in human interaction, automated facial expression analysis is still a challenging task. This paper presents a novel facial descriptor based on Phase Congruency (PC) and Local Binary Pattern (LBP) for facial expression recognition. The proposed descriptor, named Binary Pattern of Phase Congruency (BPPC), is an oriented and multi-scale local descriptor that is able to encode various patterns of face images. It is constructed by applying LBP on the oriented PC images. We evaluated the proposed method using the Cohn-Kanade (CK+) database. In our experiment, we achieved an overall detection rate of 93.83% for the six basic emotions. This shows the effectiveness of the proposed method.

The Local Binary pattern (LBP) is a well-known feature and has been widely used for human identification. However, the amount of information extracted is limited which reduces the LBP discriminative power. Recently, some enhancements have been proposed by adding preprocessing stages or considering more neighbor pixels to enrich the extracted feature. In this paper, we propose Uniformly-sampled Thresholds for LBP (UTLBP) operator that increases the richness of information derived from the LBP feature. It outperforms other features in various probe sets of the large CAS-PEAL database for face recognition. Moreover, we collected a database of 25 families to verify the superiority of the proposed feature in the family verification. Results show that using the UTLBP, the total error in face recognition and family verification is reduced up to 8% and 3% respectively comparing to the state of the art LBP. It improves the missing family member verification performance up to 3% where, contrary to expectation, increasing the LBP operator radius worsens the performance by 2%.

Abstract Fingerprint orientation plays important roles in fingerprint enhancement, fingerprint classification, and fingerprint recognition. This paper critically reviews the primary advances on fingerprint orientation estimation. Advantages and limitations of existing methods have been addressed. Issues on future development have been discussed. Copyright © 2010 John Wiley & Sons, Ltd.

Abstract A critical issue in the development of an automated fingerprint identification system (AFIS) is to detect the reference point accurately and reliably. Most existing techniques are based on the singular points of the fingerprint which is sensitive to fingerprint artifacts and is not well defined for arch type fingerprints. In this study, we present a topological interpretation of fingerprint reference point, where the reference points are posed as topological features of fingerprint structure and can be detected seamlessly from either arch type or non‐arch type fingerprints. Extensive experiments demonstrate the advantages of the proposed method over the existing ones. Copyright © 2010 John Wiley & Sons, Ltd.

Facial age classification is an approach to classify face images into one of several predefined age groups. One of the difficulties in applying learning techniques to the age classification problem is the large amount of labeled training data required. Acquiring such training data is very costly in terms of age progress, privacy, human time, and effort. Although unlabeled face images can be obtained easily, it would be expensive to manually label them on a large scale and getting the ground truth. The frugal selection of the unlabeled data for labeling to quickly reach high classification performance with minimal labeling efforts is a challenging problem. In this paper, we present an active learning approach based on an online incremental bilateral two-dimension linear discriminant analysis (IB2DLDA) which initially learns from a small pool of labeled data and then iteratively selects the most informative samples from the unlabeled set to increasingly improve the classifier. Specifically, we propose a novel data selection criterion called the furthest nearest-neighbor (FNN) that generalizes the margin-based uncertainty to the multiclass case and which is easy to compute, so that the proposed active learning algorithm can handle a large number of classes and large data sizes efficiently. Empirical experiments on FG-NET and Morph databases together with a large unlabeled data set for age categorization problems show that the proposed approach can achieve results comparable or even outperform a conventionally trained active classifier that requires much more labeling effort. Our IB2DLDA-FNN algorithm can achieve similar results much faster than random selection and with fewer samples for age categorization. It also can achieve comparable results with active SVM but is much faster than active SVM in terms of training because kernel methods are not needed. The results on the face recognition database and palmprint/palm vein database showed that our approach can handle problems with large number of classes. Our contributions in this paper are twofold. First, we proposed the IB2DLDA-FNN, the FNN being our novel idea, as a generic on-line or active learning paradigm. Second, we showed that it can be another viable tool for active learning of facial age range classification.

Gabor Wavelets are widely used to extract facial features since they are robust against illumination and pose changes. Due to the limitation in computational power, the common practice is to down-sample the face image to reduce number of Gabor features generated. As not all of the generated Gabor features are necessary, the main objective of this paper is to develop an efficient removal scheme of redundant filters in order to employ images with large dimension in face data processing. In particular, Genetic Algorithm is used to provide a computational and fast selection of feature ensemble. The base classifiers are trained by the AdaBoost algorithm with the Gabor feature set extracted from each single Gabor filter. By employing the joint diversity, Genetic Algorithm is then applied to select the most discriminate ensemble of classifiers followed by the optimum decision making rule on the classifiers outputs. The proposed approach is implemented in the family classification problem which has large intra-group variation. This method also allows us to select more discriminate filters from higher scales and finer orientations for those families with very young children to improve the performance with the same complexity and calculation load of the conventional Gabor wavelet set.

In this paper, a novel face representation in terms of dense local image descriptors is proposed. Scale Invariant Feature Transform (SIFT) and Gabor, two of the most popular local image descriptors, at dense grid pixels of a face image are used to represent the face. The efficiency of the representation has been investigated in gender recognition. There are four problems when applying the SIFT to facial gender recognition. (1) There may be only a few keypoints that can be found in a face image due to the missing texture and ill-illumined faces; (2) The SIFT descriptors at the keypoints (we called it sparse SIFT) are distinctive whereas alternative descriptors at non-keypoints (e.g. grid) could cause negative impact on the accuracy; (3) Most of the existing methods employ SIFT descriptors matching in which relative larger image size is required in order that enough keypoints can be found to support the matching and (4) The matching is assumed that the faces are well registered. We provide solutions to the above difficulties in this paper and the problem of recognizing gender using the combination of SIFT descriptors and Gabor of face images is studied. The Gabor representations of the face images are fused with the dense SIFT at the feature-level to improve the accuracy. AdaBoost is adopted to select features and form a strong classifier. The experimental results on a large set of faces have shown that the proposed method can achieve high accuracies even for faces that are not aligned.

Age estimation is an important enabling capability for the near future, especially in applications related to Human Computer Interaction. Perhaps due to technical difficulties, age estimation has only recently begun to receive more attention. One of the more important pre-processing steps before age estimation is facial alignment, which spatially transforms a face image to align certain facial features, in order to maximize classification accuracy. However the capability for facial alignment in automated age estimation literature is commonly assumed, and the effects of facial alignment is not addressed, even though it is an important issue especially for live deployment. In this paper, we present, to our knowledge, the results of the first systematic investigation on the effects of facial alignment on age estimation accuracy, and conclude with some directions on further topics of investigation in this area.

A fundamental problem in computer vision is to enhance images under various lighting conditions. This paper proposes an approach to lighting correction through reconstructing the image based on relative gradients. Compared with the absolute gradients as exemplified in the originally acquired image, the relative gradients are more stable under the variation of lighting. Extensive experiments have been carried out over images under a wide range of lighting variations. Comparisons with popular lighting correction methods are presented.

The increasing penetration rates of high speed broadband networks have infused strong impetus for IPTV adoption and usage. One of the more intriguing possibilities of personalizing the IPTV experience is to install automated age classification capabilities onto a display set. This allows the detection, by vision-based methods, of the demographic profile of the IPTV user and automatically suggest or stream services, advertisements and adjust settings to provide a personalized IPTV experience to the user. We describe a vision-based age estimation system, capable of estimating the age of a person using the near frontal face with a mean average error of 5.49 years. We discuss the likely scenarios where such a capability can be applied, and the likely impact it will have on the user experience. Finally, the paper explores related user issues and the possible solutions within the IPTV context.

Fingerprint orientation plays important roles in fingerprint recognition. This paper proposes a framework for modeling the fingerprint orientation field based on the variational principle. The proposed method does not require any prior information about the structure of acquired fingerprints. Comparison has been made with respect to state-of-the-arts in fingerprint orientation modeling.

Image visibility is a fundamental issue in the field of computer vision. This paper investigates the connection between histogram and image visibility, where the concept of entropy is employed to depict the information content of the histogram. It turns out that image visibility is more dependent on the observed intensity levels with higher frequencies and the distribution of their locations in the range of intensity levels. With this in mind, the concept of visible entropy is proposed. The usefulness of the proposed visibility measure has been evaluated using a number of realistic images.

This paper presents a novel method for fingerprint orientation modeling, which executes in two phases. Firstly, the orientation field is reconstructed through fitting to a lower order Legendre polynomial basis to capture the global orientation pattern. Then the preliminary model around the singular region is dynamically refined by fitting to a higher order Legendre polynomial basis. The singular region is automatically detected through the analysis on the orientation residual field between the original orientation field and the orientation model. The method has been evaluated using the FVC 2004 data sets and compared with state-of-the-arts. Experiments turn out that the propose method attains higher accuracy in fingerprint matching and singularity preservation.

Face image based age categorization is an approach to classify face images into one of several pre-defined age-groups. It is challenging because the aging variation is specific to a given individual and is determined by not only the person's gene, but also by many external factors, such as exposure, weather conditions (e.g. ambient humidity), health, gender, living style and living location. Age categorization is a multiclass problem. One of the AdaBoost or SVM extensions for solving the multiclass problem is the combination of the method of error-correcting output codes (ECOC) with boosting using a decision tree based classifier or binary SVM classifier. In this paper, we apply this extension to solve the age categorization problem. Gabor and LBP aging features are extracted and combined at the feature level to represent the face images. Experimental results on FG-NET and Morph database are reported to demonstrate its effectiveness and robustness. The ECOC can achieve nearly similar results when it was combined with AdaBoost or SVM. However, ECOC plus AdaBoost is much faster than ECOC plus SVM. The results obtained using the fused LBP and Gabor features are better than the one when using either LBP or Gabor alone.

Singular point detection is an important issue in fingerprint image analysis. General methods like Poincare index method can detect singular points in non-arch type fingerprints but fail on arch-type fingerprints. Some more sophisticated methods like complex filter method also face the same problem. In this paper, we propose a systematic method for detecting singular points in fingerprint images which utilizes the most fundamental topological feature of acquired fingerprints as the basis for singular point identification. The method differentiates the input fingerprint between arch type and non-arch type. For non-arch type fingerprints singular points are detected as intersection points in a c(>2) level segmentation map. As for arch type fingerprints, singular points are identified from the symmetric line of the fingerprint structure. The method is evaluated using the NIST DB4 database and compared with the complex filter method. The proposed method attains near 90% success rate in detecting singular points and the displacement from ground truth is comparable to that of the complex filter method. Moreover our method is more than once faster in CPU time.

A technology for protecting privacy in video surveillance is presented in this paper. Human identity that can contain privacy intrusive information is protected. By integrating mean-shift and active contour, faces can be tracked and blurred in each frame of a video sequence. In the initial frame, faces are located by a face detector. We extend the Adaboost multiview face detector to detect the low-resolution faces. In order to improve the efficiency of the detection and tracking, the background subtraction is used to constrain the face search region. The face is modeled as an ellipse and the centre of the ellipse is predicted using mean shift. The position and scale of the mean shift are updated using the active contour. The combined mean shift and active contour improves the robustness of the tracking. Experimental results show that the algorithm is robust to occlusion and scale variation.

Fingerprint recognition is the most popular biometrics but the current fingerprint authentication system can still be easily circumvented by fake fingers. In this paper, we propose a novel method to detect fake finger using tactile sensation. It relies on an additional ability of a live finger, the sense of touch. Such tactile perception capability is absent in the dead or a totally fake finger. Based on this concept, an electrotactile display system capable of presenting tactile pattern is developed. This is then correlated with the pattern shown visually to the user. The system does not produce any visible, auditory or other clues and as such, its strength is only dependent on the number of visible patterns available for the user to select. Preliminary result obtained from the prototype system shows that the proposed approach is indeed able to detect gelatin fake finger worn over live fingers, even when the gelatin is only 1mm thick.

The issue of how to represent the palm features for effective classification is still an open problem. In this paper, we propose a novel palm representation, the “Junction Points” (JP) set, which is formed by the two set of line segments extracted from the registered palmprint and palm vein images respectively. Unlike the existing approaches, the JP set, containing position and orientation information, is a more compact feature that significantly reduces the storage requirement. We compare the proposed JP approach with the line-based methods on a large dataset. Experimental results show that the proposed JP approach provides a better representation and achieves lower error rate in palm verification.

For a finite set of points lying on a lower dimensional manifold embedded in a high-dimensional data space, algorithms have been developed to study the manifold structure. However, many algorithms will fail if data are noisy. We propose a method based on Gaussian process latent variable models for manifold denoising with the following advantages: (1), it is probabilistic, which naturally handles noise and missing data; (2), it works well for very high dimensional data with small sample size; (3), it can recover the low-dimensional submanifolds corrupted by high-dimensional noise; and (4), it deals well with multimodal manifolds.

This paper proposes to characterize the fingerprint orientation using a novel topological representation, which transforms the orientation field into a map composed of a set of geometric objects and analyzes the properties of each geometric object based on two assumptions: regional coherence assumption and convexity assumption. Different from prior works on fingerprint orientation analysis, this approach has a capability to identify where the errors happen, not only the ldquorandom errorsrdquo (which is assumed by local-filtering and global modeling methods), but also the ldquostructure irregularitiesrdquo, which often occur in real prints and impose difficulty on existing techniques.

In this paper, we proposed, a new approach to track the object of interest using bi-model algorithm. The bi-model comprises a long term model and a short term model to allow continuous tracking of the object of interest. To measure the similarity of the bi-model and the candidate targets, an invariant spatiogram descriptor was proposed. Compared to the original spatiogram the new definition is invariant to rotation without sacrificing the performance. The experiments on our dataset and the publicly available datasets verify the effectiveness of our proposed algorithm.

A real-time vision system operating at an outdoor swimming pool is presented in this paper. The system is designed to automatically recognize different swimming activities and to detect occurrence of early drowning incidents. We have named this system the Drowning Early Warning System (DEWS). One key challenge we faced in the problem is the relatively high level of noise in the steps of foreground detection and behavior recognition. Therefore, a set of methods in the fields of background subtraction, denoising, data fusion and blob splitting are proposed, which have been motivated by characteristics of aquatic background and crowded scenario at the pool. In the step to detect an early drowning incident, visual indicators of distress and drowning are incorporated through a set of foreground descriptors. A module comprising data fusion and hidden Markov modeling is developed to learn unique traits of different swimming behaviors, in particular, those early drowning events. The experiment of this work reports realistic on-site evaluations performed. Examples of interesting behaviors, i.e., distress, drowning, treading and numerous swimming styles, are simulated and collected. Experimental results show that we have established a prototype system which is robust and beyond the stage of proof-of-concept.

Unimodal analysis of palmprint and palm vein has been investigated for person recognition. However, they are not robust to noise and spoof attacks. In this paper, we present a multimodal personal identification system using palmprint and palm vein images with fusion applied at the image level. The palmprint and palm vein images are fused by a novel integrated line-preserving and contrast-enhancing fusion method. Based on our proposed fusion rule, the modified multiscale edges of palmprint and palm vein images are combined as well as the image contrast and the interaction points (IPs) of the palmprints and vein lines are enhanced. The IPs are novel features obtained in our fused images. A novel palm representation, called "Laplacianpalm" feature, is extracted from the fused images by Locality Preserving Projections (LPP). We compare the recognition performance using the unimodal and the proposed fused images. We also compared the proposed "Laplacianpalm " approach with the Fisherpalm and Eigenpalm on a large dataset. Experimental results show that the proposed multimodal approach provides a better representation and achieves lower error rates in palm recognition.

In a large number of fingerprint database, the naïve way to identify a person is to match the query image with all entries of the database. To speed up this processing, exclusive classification assigns each fingerprint into one of several subgroups. Thus the query image only needs to compare with the images in one subgroup. While exclusive classification is hard because of small inter-class variability and the large intra-class variability. And it is not so efficient as well due to the small number of classes and non-uniform distribution.

Singular point is crucial for fingerprint identification. However, detection of the singular point is not always perfect. Up to now, there is no effective criterion to validate the singular point detected. This paper introduces a new interactive mechanism to detect and validate a singular point. The method comprises two stages: 1. the candidate singular point is located using the complex symmetrical filters. The magnitude of the candidate singular point represents the singularity information; 2. For each detected singular point, a model-based method is used to reconstruct the orientation near the candidate point. The reconstructed orientation is used to represent its orientation information. By combining both singularities and computing the difference between the reconstructed orientation and the original orientation, the candidate singular point can be validated. Experimental results show that this method can detect the singular point even with partially corrupted orientation pattern and remove falsely detected singular points

Principal component analysis (PCA) is one of the most traditional linear dimensionality reduction algorithms. Kernel principal component analysis (kernel PCA), generalization of PCA, is a nonlinear feature extraction method. However, both PCA and kernel PCA are lack of class information in their feature subspace. In this paper, weighted kernel principal component analysis (WKPCA) is proposed for feature extraction with the application of face recognition. Weights that represent inter-class relationships are incorporated into kernel matrix. Images in training and testing set are projected onto the subspace obtained from weighted kernel matrix. The feature extraction procedure is in a framework of genetic algorithms (GAs) with the fitness as classification accuracy on cross-validation data from training set. The experimental results of WKPCA are compared with PCA and kernel PCA on a combo database (ORL, Yale, UMIST databases), and show that proposed WKPCA algorithm performs best in face recognition

A new method to model 3D face and detect the facial features in infrared image is proposed. We developed a method for creating photo-realistic 3D facial models from stereo of a human subject. In a vision system include a visible camera and an infrared camera, 3D geometric relationship between the infrared camera and visible camera are calibrated. Eye and mouth corners are detected in the visible image and the head pose with respect to the visible camera is estimated based on the corners. The corresponding feature points in the IR image can be found by the head pose and the known 3D geometric relationship between the visible and infrared cameras. By doing so, the skin temperature range within the infrared image can be superimposed over the visible face image and vice versa. Experimental results on real images have verified the efficiency of the algorithm.

This paper presents a real-time foreground detection method for monitoring swimming activities at an outdoor swimming pool. Robust performance and high accuracy of detecting objects-of-interest are two central issues of concern. Therefore, in this paper, a considerable amount of attention has been placed on the following aspects: 1) to establish a better method of modeling aquatic background, which exhibitis dynamic characteristics with random spatial movements, and 2) to establish a method of enhancing the visibility of the foreground by removing specular reflection at nighttime. First, the development of a new background modeling method is reported. In the proposed approach, the background is modeled as a composition of homogeneous blob movements. With an implementation of a spatial searching process, the proposed method shows capability in associating and distinguishing movements caused by the background. Hence, this contributes to better performance in foreground detection. On the issue of enhancing the visibility of the foreground, a decision-based filtering scheme is proposed as a preprocessing step. A defined concept term, fluctuation measure, is defined for classifying each pixel to be one of the predefined types. This has allowed suitable spatial or spatiotemporal filters to be applied accordingly for color the compensation step. All of these developments are evaluated by testing live on a busy Olympic-size outdoor public swimming pool. Both qualitative and quantitative evaluations are reported. This provides a comprehensive study of the system.

The performance of SVM-based image retrieval is often constrained by the scarcity of training samples. The total number of image samples labeled by users in a retrieval session is very limited, and this small number of labeled samples cannot effectively represent the true distributions of positive and negative image classes, especially for the negative image class. This paper proposes a novel approach to deal with this problem. Instead of treating it as a problem, the mere existence of the small number of labeled images and their desired distribution in the kernel space is considered as prior knowledge from image retrieval to aid the design of the kernel used by SVMs. This is achieved by maximizing a criterion, such as one based on scatter matrices, through gradient-based search methods, incurring very little computational overhead to real-time retrieval process. Experimental results on two benchmark image databases demonstrate the improved retrieval performance by the dynamically designed kernel and hence the effectiveness of the proposed approach for SVM based image retrieval

By exploiting the specialist capabilities of each classifier, a combined classifier may yield results which would not be possible with a single classifier. In this paper, we propose to combine the fingerprint and speaker verification decisions using a functional link network. This is to circumvent the nontrivial trial-and-error and iterative training effort as seen in backpropagation neural networks which cannot guarantee global optimal solutions. In many data fusion applications, as individual classifiers to be combined would have attained a certain level of classification accuracy, the proposed functional link network can be used to combine these classifiers by taking their outputs as the inputs to the network. The proposed network is first applied to a pattern recognition problem to illustrate its approximation capability. The network is then used to combine the fingerprint and speaker verification decisions with much improved receiver operating characteristics performance as compared to several decision fusion methods from the literature.

The idea of user-specific multimodal biometrics is pioneered by Jain, A. K., et al. (2002) and further exploited by Toh, K-A, et al. (2004) recently. In this paper, we look into several issues pertaining to user-specific multimodal biometric verification. These issues include the small sample size problem, learning of local decision hyperplanes and setting of local thresholds. For small sample size problem, the noise-injection technique and a feature scaling-space technique are considered. For local learning, we adopt a recently proposed reduced polynomial since it has fast single-step computation and accurate estimation. For setting of local decision thresholds, nine baselines are identified. Extensive experiments are performed on a moderate data set and relatively conclusive results are observed.

Fingerprint orientation is crucial for automatic fingerprint identification. However, recovery of orientation is still difficult especially in noisy region. A way to aid recovery of the orientation is to provide an orientation model. In this paper, an orientation model for the entire fingerprint orientation using high order phase portrait is suggested. Proper analysis of the orientation pattern at the singular point regions is provided. Then a low-order phase portrait near each of the singular point is added as constraint to the high-order phase portrait to provide accurate orientation modeling for the entire fingerprint image. The main advantage of the proposed approach is that the nonlinear model itself is able to model all type of fingerprint orientations completely. Experiments and visual analysis show the effectiveness of the proposed model.

Fingerprint image quality analysis is crucial in eliminating poor fingerprint images, which will affect the performance of the automatic fingerprint identification system. In this article, two types of new quality measures will be introduced: ridge and valley clarity and global orientation flow to calculate the overall image quality score that can be used to quantitatively determine the quality of the fingerprint image. In order to evaluate the performance of the proposed algorithm, the quality measure is used to rank the performance of a fingerprint recognition system and the ranking is compared with the quality measure rated manually. The result shows that the proposed scheme will return a score that ensures its reliability to indicate the quality of a given fingerprint image.

This paper discusses methods in evaluating fingerprint image quality on a local level. Feature vectors covering directional strength, sinusoidal local ridge/valley pattern, ridge/valley uniformity and core occurrences are first extracted from fingerprint image subblocks. Each subblock is then assigned a quality level through pattern classification. Three different classifiers are employed to compare each of its different effectiveness. Positive results have been obtained based on our database.

An adaptive spatio-temporal filtering scheme based on a novel concept of motion frequency is proposed as the preprocessing step to enhance human detection under a noisy aquatic environment. In this framework, each pixel is first classified into one of three categories quantified by its motion frequency, each of which is filtered using an appropriate filtering scheme. For regions affected by glistening reflections and glares, a color compensation filter is specially developed to improve partly hidden human detection and minimize errors due to moving background elements. Additionally, a blob-based verification procedure is introduced to remove the target's shadow. Experimental results demonstrate the effectiveness of the algorithm and its role in enhancing the robustness of an aquatic surveillance system for outdoor swimming pools at nighttime.

The multivariate polynomial model provides an effective way to describe complex nonlinear input-output relationships since it is tractable for optimization, sensitivity analysis, and prediction of confidence intervals. However, for high-dimensional and high-order problems, multivariate polynomial regression becomes impractical due to its huge number of product terms. This is especially true for the case of a full interaction model. In this paper, we propose a reduced multivariate polynomial model to circumvent the dimensionality problem with some compromise in its approximation capability. In multimodal biometrics and many classifiers fusion applications, as individual classifiers to be combined would have attained a certain level of classification accuracy, this reduced multivariate polynomial model can be used to combine these classifiers in the next level of classification taking their outputs as the inputs to the reduced multivariate polynomial model. The model is first applied to a well-known pattern classification problem to illustrate its classification capability. The reduced multivariate polynomial model is then applied to combine two biometric verification systems with improved receiver operating characteristics performance as compared to an optimal weighing method and a few commonly used classifiers.

In this paper, we treat the problem of combining fingerprint and speech biometric decisions as a classifier fusion problem. By exploiting the specialist capabilities of each classifier, a combined classifier may yield results which would not be possible in a single classifier. The Feedforward Neural Network provides a natural choice for such data fusion as it has been shown to be a universal approximator. However, the training process remains much to be a trial-and-error effort since no learning algorithm can guarantee convergence to optimal solution within finite iterations. In this work, we propose a network model to generate different combinations of the hyperbolic functions to achieve some approximation and classification properties. This is to circumvent the iterative training problem as seen in neural networks learning. In many decision data fusion applications, since individual classifiers or estimators to be combined would have attained a certain level of classification or approximation accuracy, this hyperbolic functions network can be used to combine these classifiers taking their decision outputs as the inputs to the network. The proposed hyperbolic functions network model is first applied to a function approximation problem to illustrate its approximation capability. This is followed by some case studies on pattern classification problems. The model is finally applied to combine the fingerprint and speaker verification decisions which show either better or comparable results with respect to several commonly used methods.

Automatically understanding events happening at a site is the ultimate goal of visual surveillance system. We investigate the challenges faced by automated surveillance systems operating in hostile conditions and demonstrate the developed algorithms via a system that detects water crises within highly dynamic aquatic environments. An efficient segmentation algorithm based on robust block-based background modelling and thresholding-with-hysteresis methodology enables swimmers to be reliably detected amid reflections, ripples, splashes and rapid lighting changes. Partial occlusions are resolved using a Markov Random Field framework that enhances the tracking capability of the system. Visual indicators of water crises are identified based on professional knowledge of water crises detection, based on which a set of swimmer descriptors has been defined. Through seamlessly fusing the extracted swimmer descriptors based on a novel functional link network, the system achieves promising results for water crises detection. The developed algorithms have been incorporated into a live system with robust performance for different hostile environments faced by an outdoor swimming pool.

Minutiae extraction is a crucial step in an automatic fingerprint identification system. However, the presence of noise in poor-quality images causes a large number of extraction errors, including the dropping of true minutiae and production of false minutiae. A study on these errors reveals that postprocessing is effective in removing false minutiae while keeping true ones. Furthermore, the overall processing efficiency could be improved because of the reduction in total minutia number. In this paper, we present a novel fingerprint image postprocessing algorithm. It is developed based on several rules, which are generalized through a study on the errors that commonly occur in minutiae extraction and their effects on the overall verification performance. Thorough experimental tests demonstrate the proposed postprocessing algorithm to be both effective and efficient.

The multivariate polynomials model provides an effective way to describe complex nonlinear input-output relationships as it is tractable for optimization, sensitivity analysis, and prediction of confidence intervals. However, for high dimensional and high order problems, multivariate polynomial regression becomes impractical due to its prohibitive number of product terms. This is especially true for the case of a full interaction model. In this paper, we propose a reduced multivariate polynomials model to circumvent the dimensionality problem with some compromise in the approximation capability. When applied to multi-modal biometrics fusion, this mode! is demonstrated to improve the combined classification performance in terms of classification accuracy.

Discusses methods to estimate the quality as well as validity of a fingerprint image. Orientation certainty is used to certify the localized texture pattern of the fingerprint images while ridge to valley structure is analyzed to detect invalid images. Global uniformity and continuity ensures that the image is valid as a whole. 150 images with various qualities are evaluated using the proposed algorithm and quality benchmark we defined. A monotonic relationship is found indicating that the proposed algorithm is feasible in detecting low quality as well as invalid fingerprint images.

In this paper, we address the false rejection problem due to the small solid state sensor area available for fingerprint image capture. We propose a minutiae data synthesis approach to circumvent this problem. The main advantages of this approach over the existing image mosaicing approach include low memory storage requirements and low computational complexity. Moreover, the possible matching search overhead due to data redundancy can be reduced. Extensive experiments are conducted to determine the best transformation suitable for minutiae alignment. Among the three transformations presented, affine transformation is found to be most suited for minutiae alignment. We demonstrate the idea of synthesis with an example using physical fingerprint images. The proposed synthesis system is also shown to reduce the number of false rejects caused by the use of different fingerprint regions for matching.

Proposes a fingerprint minutia matching technique, which matches the fingerprint minutiae by using both the local and global structures of minutiae. The local structure of a minutia describes a rotation and translation invariant feature of the minutia in its neighborhood. It is used to find the correspondence of two minutiae sets and increase the reliability of the global matching. The global structure of minutiae reliably determines the uniqueness of fingerprint. Therefore, the local and global structures of minutiae together provide a solid basis for reliable and robust minutiae matching. The proposed minutiae matching scheme is suitable for an online processing due to its high processing speed. Experimental results show the performance of the proposed technique.

This paper presents an improved approach of minutiae detection that adaptively traces the gray level ridge of the originals fingerprint image with piecewise linear lines of different length. While tracing the ridges, the fingerprint image is smoothed with an oriented smoothing filter only at selected pixels where smoothing is necessary. After tracing all the ridges, a piece-wise linear skeleton image is obtained. Each ridge in the skeleton is labeled with a number so that each minutiae is associated with one or two ridge numbers. The post-processing is based not only on the location relationship of the minutiae, but also the associated ridge relationship and the certainty level of the minutiae. The performance of this approach is objectively assessed by using two large fingerprint databases.