LiDAR-based object detection plays a crucial role in autonomous driving, yet remains vulnerable to ghost object attacks, where adversarially crafted point clouds trigger false detections. Traditional attack methods often require a large number of points and substantial computational resources, making them inapplicable in real-world scenarios. Although prior research has explored reducing the number of required attack points, real-world feasibility remains a challenge due to the high computational cost and attack generation time. While LiDAR object detection can be compromised given enough adversarially crafted points, we believe there are some critical points, which we call reduced attack budget, which are sufficient to attack the detector. Motivated by this consideration, we propose a novel geometric- and heuristic-based approach capable of generating effective ghost object attacks by leveraging the target’s contour characteristics. By strategically selecting a minimal set of adversarial points, as few as 20 points, our approach maintains high detection confidence, reduces attack execution time and attack budget by up to 4 and 73 times, respectively, in this paper. Our method has been verified on both simulated and real-world datasets (KITTI). The experimental results demonstrate that our optimized attack significantly reduces computational overhead and consequently improves the attack’s real-time feasibility.

Domain adaptive object detection aims to adapt detection models to domains where annotated data is unavailable. Existing methods have been proposed to address the domain gap using the semi-supervised student-teacher framework. However, a fundamental issue arises from the class imbalance in the labelled training set, which can result in inaccurate pseudo-labels. The relationship between classes, especially where one class is a majority and the other minority, has a large impact on class bias. We propose Class-Aware Teacher (CAT) to address the class bias issue in the domain adaptation setting. In our work, we ap-proximate the class relationships with our Inter-Class Relation module (ICRm) and exploit it to reduce the bias within the model. In this way, we are able to apply augmentations to highly related classes, both inter- and intra-domain, to boost the performance of minority classes while having minimal impact on majority classes. We further reduce the bias by implementing a class-relation weight to our classification loss. Experiments conducted on various datasets and ablation studies show that our method is able to address the class bias in the domain adaptation setting. On the Cityscapes$\rightarrow$Foggy Cityscapes dataset, we attained a 52.5 mAp, a substantial improvement over the 51.2 mAP achieved by the state-of-the-art method.11www.github.com/mecarill/cat

LIDAR-based 3D object detection aims to enhance the situational awareness of autonomous vehicles. Despite recent advancements in this technology, there has been evidence that the susceptibility of 3D object detectors to signal spoofing is high, leading to the erroneous detection of "ghost objects" or the failure to detect genuine ones. While prior work has investigated the design of these new attacks and new defenses, the effect of weather conditions, which is a hot topic in autonomous vehicle research, on both attacks and defenses has never been studied. Inspired by this observation, in this paper, we present a novel genetic algorithm-based attack, entitled Rain-Reaper, that leverages on the effect of rain and identifies critical detection points used by 3D detectors. We show that adverse weather conditions not only diminish detection distance and accuracy but also expose the limitations of existing defenses. We have found that the unique characteristics of wet roads lead to underperforming defenses, thus, leading to a false sense of confidence in them. The effectiveness and efficiency of the attack and the robustness of the defenses have been evaluated with both simulated and real data. Our Rain-Reaper demonstrates a high attack success rate while successfully evading existing defenses with an adversarial point budget of up to 8.8 times smaller than previously demonstrated state-of-the-art attacks.

Object detection at night is a challenging problem due to the absence of night image annotations. Despite several domain adaptation methods, achieving high-precision results remains an issue. False-positive error propagation is still observed in methods using the well-established student-teacher framework, particularly for small-scale and low-light objects. This paper proposes a two-phase consistency unsupervised domain adaptation network, 2PCNet, to address these issues. The network employs high-confidence bounding-box predictions from the teacher in the first phase and appends them to the student's region proposals for the teacher to re-evaluate in the second phase, resulting in a combination of high and low confidence pseudolabels. The night images and pseudo-labels are scaled-down before being used as input to the student, providing stronger small-scale pseudo-labels. To address errors that arise from low-light regions and other night-related attributes in images, we propose a night-specific augmentation pipeline called NightAug. This pipeline involves applying random augmentations, such as glare, blur, and noise, to daytime images. Experiments on publicly available datasets demonstrate that our method achieves superior results to state-of-the-art methods by 20%, and to supervised models trained directly on the target data.11www.github.com/mecarill/2pcnet

The lane detection systems in ADAS and self-driving applications need to be resilient to adverse weather like rain, to minimize the chance of serious accidents. However, it is still challenging to maintain high precision in these conditions. State-of-the-art segmentation-based detectors use a naive post-processing method to quickly extract lane points from the segmented output based on the local maxima, but it is prone to failure when the segmentation contains errors. In this paper, we present an Adaptive Lane point EXtractor (ALEX) which overcomes this limitation by harnessing statistical properties from local regions of the segmentation to implicitly identify and compensate for errors. ALEX merges both local maximum and local mean statistics with CNN attributes to build a holistic hybrid feature set. Lane points are predicted from this representation by estimating their location on a reduced-size confidence map. A lateral offset is predicted to compute the precise location on the full-sized scene, while a class label prediction denotes which lane class the point belongs to. Besides the segmentation output and lane point ground truth, no additional information is required by ALEX, making it effectively agnostic to weather conditions and segmentation methods. Evaluation on two publicly available clear-weather databases and one rain-translated database verified that our approach consistently improved accuracy while reducing false positive and false negative rates, regardless of weather or segmentation method.

Vision-based vehicle detection under bad weather conditions is still a challenging problem. Adhesive raindrops on windshield have been known to diffract light and distort parts of the scene behind them. In this paper, we propose a Vehicle-Aware Generative Adversarial Networks (VAGAN) to improve vehicle detection from rain images. We train a Generative Adversarial Network (GAN) on image pairs, each comprising of an original rain image and one that is manually labelled with colored bounding boxes of the vehicles therein. The latter represents a fake version of the original image emphasizing the regions of interest. To further enhance vehicle awareness, we exploit the fact that the vehicle rear lights are usually turned on during rainy conditions to compute a saliency map of image, and use it formulate a background preserving constrain on the learning vehicle loss function. We show that this novel adversarial framework is able to generate new images with colored regions overlay over vehicles, hence effectively learning to differentiate image background from vehicles. The final vehicle detection in the generated images is not affected by image translation noise because we can simply use color segmentation to localize the vehicles. Experimental results on a large dataset show that our approach is an effective way to solve vehicle detection in rain images, achieving state-of-the-art performance.

Traffic light recognition (TLR) detects the traffic light from an image and then estimates the state of the light signal. TLR is important for autonomous vehicles because running against a red light could cause a deadly car accident. For a practical TLR system, computation time, varying illumination conditions, and false positives are three key challenges. In this paper, a novel real-time method is proposed to recognize a traffic light with high dynamic imaging and deep learning. In our approach, traffic light candidates are robustly detected from low exposure/dark frames and accurately classified using a deep neural network in consecutive high exposure/bright frames. This dual-channel mechanism can make full use of undistorted color and shape information in dark frames as well as the rich context in bright frames. In the dark channel, a non-parametric multi-color saliency model is proposed to simultaneously extract lights with different colors. A multiclass classifier with convolutional neural network (CNN) model is then adopted to reduce the number of false positives in the bright channel. The performance is further boosted by incorporating temporal trajectory tracking. In order to speed up the algorithm, a prior detection mask is generated to limit the potential search regions. Intensive experiments on a large dual-channel dataset show that the proposed approach outperforms the state-of-the-art real-time deep learning object detector, which could cause more false positives because it uses bright images only. The algorithm has been integrated into our autonomous vehicle and can work robustly on real roads.

While much effort has been devoted to deep learning object detection, relatively limited attention has been paid to object detection in bad weather, e.g. rain, snow or haze. In heavy rain, the raindrop on the front windshield can make it difficult to detect object from an in-car camera. The conventional way to cope with this has been to use radar as the main detection sensor. However, radar is highly susceptible to false positives. Furthermore, many entry level radar sensors only return the centroid of each detected object, rather than its size and extent. In addition, due to lack of texture input, radar cannot discriminate a vehicle from a non-vehicle object, e.g. roadside pole. This motivates us to detect vehicle by fusing radar and vision. In this paper, we first calibrate the radar and camera with respect to the ground plane. The radar detections are then projected to the camera image for target width estimation. Empirical evaluation on a large database shows that there is a natural synergy in both sensors, as the image based estimation is found to be greatly facilitated by the accuracy of the radar detection.

Vehicle following is one of the fundamental functions of an autonomous driving system. Detection and recognition of tail light signal is important to prevent an autonomous vehicle from rear-end collisions or accidents. Although sensors like acoustic sonar or commercialized Advanced Driving Assistance System (ADAS) products such as mobileye could be used for rear-end collision warning, a cost-effective approach is expected. In this paper, we have developed a novel two-stage approach to detect vehicles and recognize brake lights from a single image in real-time. Unlike previous approaches where pair taillight has to be extracted explicitly, we use vehicle rear appearance image instead. On a large database, “Brake Lights Patterns” (BLP) are learned by a multi-layer perception neural network. Given an image, the vehicles can be classified as “brake” or “normal” using the deep classifier. The vehicle can be detected quickly and robustly by combining multi-layer lidar (IBEO Lux fusion system) and a camera. Road segmentation and a novel vanishing point region of interest (ROI) determination method are explored to further speed up the detection and improve the robustness. The experimental results conducted on some real on-road videos have shown the robustness and efficiency of the proposed approach.

This paper describes the gesture recognition system developed by the Institute for Infocomm Research (I2R) for the 2013 ICMI CHALEARN Multi-modal Gesture Recognition Challenge. The proposed system adopts a multi-modal approach for detecting as well as recognizing the gestures. Automated gesture detection is performed using both audio signals and information about hand joints obtained from the Kinect sensor to segment a sample into individual gestures. Once the gestures are detected and segmented, features extracted from three different modalities, namely, audio, 2-dimensional video (RGB), and skeletal joints (Kinect) are used to classify a given sequence of frames into one of the 20 known gestures or an unrecognized gesture. Mel frequency cepstral coefficients (MFCC) are extracted from the audio signals and a Hidden Markov Model (HMM) is used for classification. While Space-Time Interest Points (STIP) are used to represent the RGB modality, a covariance descriptor is extracted from the skeletal joint data. In the case of both RGB and Kinect modalities, Support Vector Machines (SVM) are used for gesture classification. Finally, a fusion scheme is applied to accumulate evidence from all the three modalities and predict the sequence of gestures in each test sample. The proposed gesture recognition system is able to achieve an average edit distance of 0.2074 over the 275 test samples containing 2,742 unlabeled gestures. While the proposed system is able to recognize the known gestures with high accuracy, most of the errors are caused due to insertion, which occurs when an unrecognized gesture is misclassified as one of the 20 known gestures.

The conventional mouth gender recognition is based on a static image and ignore the dynamic information. In this paper, we propose a lip movement gender recognition method to improve the accuracy by exploring the dynamic information while a user is speaking. In order to overcome the difficulty caused by the nonlinear distribution of the lip images, Gausian Mixture Models (GMM) is adopted to represent the lip images. A similarity measure is defined to measure the difference between the successive frames. Gender recognition, as a soft biometric trait, can provide useful information for improving the performance of the speaker recognition systems. The accuracy of voice-based speaker gender recognition is high if the condition of the environment is good. But it will be drastically decreased if the test is conducted in a noisy environment. In this paper, we showed that lip movement, considered as a sequence of mouth images, can provide additional information than mouth alone for recognizing gender Experimental result obtained showed the effectiveness of the proposed method which is comparable to using just the voice information.

Facial age classification is an approach to classify face images into one of several predefined age groups. One of the difficulties in applying learning techniques to the age classification problem is the large amount of labeled training data required. Acquiring such training data is very costly in terms of age progress, privacy, human time, and effort. Although unlabeled face images can be obtained easily, it would be expensive to manually label them on a large scale and getting the ground truth. The frugal selection of the unlabeled data for labeling to quickly reach high classification performance with minimal labeling efforts is a challenging problem. In this paper, we present an active learning approach based on an online incremental bilateral two-dimension linear discriminant analysis (IB2DLDA) which initially learns from a small pool of labeled data and then iteratively selects the most informative samples from the unlabeled set to increasingly improve the classifier. Specifically, we propose a novel data selection criterion called the furthest nearest-neighbor (FNN) that generalizes the margin-based uncertainty to the multiclass case and which is easy to compute, so that the proposed active learning algorithm can handle a large number of classes and large data sizes efficiently. Empirical experiments on FG-NET and Morph databases together with a large unlabeled data set for age categorization problems show that the proposed approach can achieve results comparable or even outperform a conventionally trained active classifier that requires much more labeling effort. Our IB2DLDA-FNN algorithm can achieve similar results much faster than random selection and with fewer samples for age categorization. It also can achieve comparable results with active SVM but is much faster than active SVM in terms of training because kernel methods are not needed. The results on the face recognition database and palmprint/palm vein database showed that our approach can handle problems with large number of classes. Our contributions in this paper are twofold. First, we proposed the IB2DLDA-FNN, the FNN being our novel idea, as a generic on-line or active learning paradigm. Second, we showed that it can be another viable tool for active learning of facial age range classification.

In this paper, a novel face representation in terms of dense local image descriptors is proposed. Scale Invariant Feature Transform (SIFT) and Gabor, two of the most popular local image descriptors, at dense grid pixels of a face image are used to represent the face. The efficiency of the representation has been investigated in gender recognition. There are four problems when applying the SIFT to facial gender recognition. (1) There may be only a few keypoints that can be found in a face image due to the missing texture and ill-illumined faces; (2) The SIFT descriptors at the keypoints (we called it sparse SIFT) are distinctive whereas alternative descriptors at non-keypoints (e.g. grid) could cause negative impact on the accuracy; (3) Most of the existing methods employ SIFT descriptors matching in which relative larger image size is required in order that enough keypoints can be found to support the matching and (4) The matching is assumed that the faces are well registered. We provide solutions to the above difficulties in this paper and the problem of recognizing gender using the combination of SIFT descriptors and Gabor of face images is studied. The Gabor representations of the face images are fused with the dense SIFT at the feature-level to improve the accuracy. AdaBoost is adopted to select features and form a strong classifier. The experimental results on a large set of faces have shown that the proposed method can achieve high accuracies even for faces that are not aligned.

Age estimation is an important enabling capability for the near future, especially in applications related to Human Computer Interaction. Perhaps due to technical difficulties, age estimation has only recently begun to receive more attention. One of the more important pre-processing steps before age estimation is facial alignment, which spatially transforms a face image to align certain facial features, in order to maximize classification accuracy. However the capability for facial alignment in automated age estimation literature is commonly assumed, and the effects of facial alignment is not addressed, even though it is an important issue especially for live deployment. In this paper, we present, to our knowledge, the results of the first systematic investigation on the effects of facial alignment on age estimation accuracy, and conclude with some directions on further topics of investigation in this area.

The increasing penetration rates of high speed broadband networks have infused strong impetus for IPTV adoption and usage. One of the more intriguing possibilities of personalizing the IPTV experience is to install automated age classification capabilities onto a display set. This allows the detection, by vision-based methods, of the demographic profile of the IPTV user and automatically suggest or stream services, advertisements and adjust settings to provide a personalized IPTV experience to the user. We describe a vision-based age estimation system, capable of estimating the age of a person using the near frontal face with a mean average error of 5.49 years. We discuss the likely scenarios where such a capability can be applied, and the likely impact it will have on the user experience. Finally, the paper explores related user issues and the possible solutions within the IPTV context.

Face image based age categorization is an approach to classify face images into one of several pre-defined age-groups. It is challenging because the aging variation is specific to a given individual and is determined by not only the person's gene, but also by many external factors, such as exposure, weather conditions (e.g. ambient humidity), health, gender, living style and living location. Age categorization is a multiclass problem. One of the AdaBoost or SVM extensions for solving the multiclass problem is the combination of the method of error-correcting output codes (ECOC) with boosting using a decision tree based classifier or binary SVM classifier. In this paper, we apply this extension to solve the age categorization problem. Gabor and LBP aging features are extracted and combined at the feature level to represent the face images. Experimental results on FG-NET and Morph database are reported to demonstrate its effectiveness and robustness. The ECOC can achieve nearly similar results when it was combined with AdaBoost or SVM. However, ECOC plus AdaBoost is much faster than ECOC plus SVM. The results obtained using the fused LBP and Gabor features are better than the one when using either LBP or Gabor alone.

A technology for protecting privacy in video surveillance is presented in this paper. Human identity that can contain privacy intrusive information is protected. By integrating mean-shift and active contour, faces can be tracked and blurred in each frame of a video sequence. In the initial frame, faces are located by a face detector. We extend the Adaboost multiview face detector to detect the low-resolution faces. In order to improve the efficiency of the detection and tracking, the background subtraction is used to constrain the face search region. The face is modeled as an ellipse and the centre of the ellipse is predicted using mean shift. The position and scale of the mean shift are updated using the active contour. The combined mean shift and active contour improves the robustness of the tracking. Experimental results show that the algorithm is robust to occlusion and scale variation.

The issue of how to represent the palm features for effective classification is still an open problem. In this paper, we propose a novel palm representation, the “Junction Points” (JP) set, which is formed by the two set of line segments extracted from the registered palmprint and palm vein images respectively. Unlike the existing approaches, the JP set, containing position and orientation information, is a more compact feature that significantly reduces the storage requirement. We compare the proposed JP approach with the line-based methods on a large dataset. Experimental results show that the proposed JP approach provides a better representation and achieves lower error rate in palm verification.

In this paper, we proposed, a new approach to track the object of interest using bi-model algorithm. The bi-model comprises a long term model and a short term model to allow continuous tracking of the object of interest. To measure the similarity of the bi-model and the candidate targets, an invariant spatiogram descriptor was proposed. Compared to the original spatiogram the new definition is invariant to rotation without sacrificing the performance. The experiments on our dataset and the publicly available datasets verify the effectiveness of our proposed algorithm.

A new deformable model called active b-snake model (ABM) is presented for object boundary extraction. First, an affine-invariant landmark point assignment strategy is proposed to avoid manually assign landmarks. Second, an adaptive control point insertion algorithm is used to enhance the flexibility of B-Snake to describe complex shape. Third, for modeling the shape distribution and appearance characteristics of landmark points in the training samples, a statistical framework is embedded into ABM. Finally, the minimum mean square error (MMSE) approach with control point insertion provides fine deformation to the desired object boundaries. Experimental results show that ABM can be used to achieve more accurate object extraction.

Unimodal analysis of palmprint and palm vein has been investigated for person recognition. However, they are not robust to noise and spoof attacks. In this paper, we present a multimodal personal identification system using palmprint and palm vein images with fusion applied at the image level. The palmprint and palm vein images are fused by a novel integrated line-preserving and contrast-enhancing fusion method. Based on our proposed fusion rule, the modified multiscale edges of palmprint and palm vein images are combined as well as the image contrast and the interaction points (IPs) of the palmprints and vein lines are enhanced. The IPs are novel features obtained in our fused images. A novel palm representation, called "Laplacianpalm" feature, is extracted from the fused images by Locality Preserving Projections (LPP). We compare the recognition performance using the unimodal and the proposed fused images. We also compared the proposed "Laplacianpalm " approach with the Fisherpalm and Eigenpalm on a large dataset. Experimental results show that the proposed multimodal approach provides a better representation and achieves lower error rates in palm recognition.

A new method to model 3D face and detect the facial features in infrared image is proposed. We developed a method for creating photo-realistic 3D facial models from stereo of a human subject. In a vision system include a visible camera and an infrared camera, 3D geometric relationship between the infrared camera and visible camera are calibrated. Eye and mouth corners are detected in the visible image and the head pose with respect to the visible camera is estimated based on the corners. The corresponding feature points in the IR image can be found by the head pose and the known 3D geometric relationship between the visible and infrared cameras. By doing so, the skin temperature range within the infrared image can be superimposed over the visible face image and vice versa. Experimental results on real images have verified the efficiency of the algorithm.

In this paper, a framework of Discriminant Low-dimensional Subspace Analysis (DLSA) method is proposed to deal with the Small Sample Size (SSS) problem in face recognition area. Firstly, it is rigorously proven that the null space of the total covariance matrix, S t, is useless for recognition. Therefore, a framework of Fisher discriminant analysis in a low-dimensional space is developed by projecting all the samples onto the range space of S t. Two algorithms are proposed in this framework, i.e., Unified Linear Discriminant Analysis (ULDA) and Modified Linear Discriminant Analysis (MLDA). The ULDA extracts discriminant information from three subspaces of this lowdimensional space. The MLDA adopts a modified Fisher criterion which can avoid the singularity problem in conventional LDA. Experimental results on a large combined database have demonstrated that the proposed ULDA and MLDA can both achieve better performance than the other state-of-the-art LDA-based algorithms in recognition accuracy. 1

To solve the Small Sample Size (SSS) problem, the recent linear discriminant analysis using the 2D matrix-based data representation model has demonstrated its superiority over that using the conventional vector-based data representation model in face recognition [7]. But the explicit reason why the matrix-based model is better than vectorized model has not been given until now. In this paper, a framework of Generalized 2D Fisher Discriminant Analysis (G2DFDA) is proposed. Three contributions are included in this framework: 1) the essence of these ’2D ’ methods is analyzed and their relationships with conventional ’1D ’ methods are given, 2) a Bilateral and 3) a Kernel-based 2D Fisher Discriminant Analysis methods are proposed. Extensive experiment results show its excellent performance. 1

A novel framework called 2D Fisher discriminant analysis (2D-FDA) is proposed to deal with the small sample size (SSS) problem in conventional one-dimensional linear discriminant analysis (1D-LDA). Different from the 1D-LDA based approaches, 2D-FDA is based on 2D image matrices rather than column vectors so the image matrix does not need to be transformed into a long vector before feature extraction. The advantage arising in this way is that the SSS problem does not exist any more because the between-class and within-class scatter matrices constructed in 2D-FDA are both of full-rank. This framework contains unilateral and bilateral 2D-FDA. It is applied to face recognition where only few training images exist for each subject. Both the unilateral and bilateral 2D-FDA achieve excellent performance on two public databases: ORL database and Yale face database B.

This paper addresses the small sample size (SSS) problem in linear discriminant analysis (LDA) utilizing a so called 2D Fisher discriminant analysis (2D-FDA) algorithm. As opposed to traditional LDA-based approaches, 2D-FDA is based on 2D image matrices rather than 1D vectors so the image matrix does not need to be transformed into a vector before feature extraction. The between-class scatter and the within-class scatter is constructed using the original image matrices. The advantage arising in this way is that the SSS problem existing in traditional linear discriminant analysis does not occur any more. To test the performance of 2D-FDA with small number of training samples, a series of experiments are conducted on two public databases: ORL and Yale face database B. In both two trials, the 2D-FDA outperforms the other linear subspace methods when there are only very limited training images for each subject.

A new approach for estimating 3D head pose form a monocular image is proposed. Our approach employs general prior knowledge of face structure and the corresponding geometrical constraints provided by the location of vanishing point to determine pose of human faces. Eye-lines and mouth-line are assumed parallel in 3D space, and the vanishing point formed by the intersection of the eyeline and mouth-line in the image can be used to infer 3D orientation and location of human face. Perspective invariance of cross ratio and harmonic range is used to locate the vanishing point stably. The robustness analysis of the algorithm with synthesis data and real face images are enclosed.

We present a novel approach, called the "one-circle " algorithm, for measuring the eye gaze using a monocular image that zooms in on only one eye of a person. Observing that the iris contour is a circle, we estimate the normal direction of this iris circle, considered as the eye gaze, from its elliptical image. From basic projective geometry, an ellipse can be back-projected into space onto two circles of different orientations. However, by using an anthropometric property of the eyeball, the correct solution can be disambiguated. This allows us to obtain a higher resolution image of the iris with a zoom-in camera and thereby achieving higher accuracies in the estimation. The robustness of our gaze determination approach was verified statistically by the extensive experiments on synthetic and real image data. The two key contributions are that we show the possibility of finding the unique eye gaze direction from a single image of one eye and that one can obtain better accuracy as a consequence of this.

There are two components to the human visual line-of-sight: pose of human head and the orientation of the eye within their sockets. We have investigated these two aspects but will concentrate on eye gaze estimation. We present a novel approach called the "one-circle" algorithm for measuring the eye gaze using a monocular image that zooms in on only one eye of a person. Observing that the iris contour is a circle, we estimate the normal direction of this iris circle, considered as the eye gaze, from its elliptical image. From basic projective geometry, an ellipse can be back-projected into space onto two circles of different orientations. However, by using a geometric constraint, namely, that the distance between the eyeball's center and the two eye corners should be equal to each other, the correct solution can be disambiguated. This allows us to obtain a higher resolution image of the iris with a zoom-in camera, thereby achieving higher accuracies in the estimation. A general approach that combines head pose determination with eye gaze estimation is also proposed. The searching of the eye gaze is guided by the head pose information. The robustness of our gaze determination approach was verified statistically by the extensive experiments on synthetic and real image data. The two key contributions are that we show the possibility of finding the unique eye gaze direction from a single image of one eye and that one can obtain better accuracy as a consequence of this.

Abstract In this paper, we present a new approach for measuring eye-gaze of human via images of the irises. Two iris contours are approximately modeled as two circles having known radii and we estimate the ellipses of their projections onto image plane, respectively. The eye-gaze, defined in this paper as the unit surface normal to the supporting plane of the iris, can be estimated from the projection of the iris contour (limbus). The unique solution of the eye-gaze was obtained based on a geometric constraint, namely that the difference between the two normal directions to the supporting planes of the left and right iris should be minimal, irrespective of eyeball rotations and head movements. Furthermore, the 3D location of the center of the irises can be calculated if the radius of the iris contour is given. The robustness of this approach was verified by experiments on synthetic and real image data. In order to obtain a higher resolution of the iris, a zoom-in camera is used. This creates an additional problem of having to relate the camera for estimating the head-pose to the iris camera. A general approach that combines head-pose determination with gaze estimation is proposed.

A new approach for detecting front-view facial contour is described. A novel contribution is to eliminate the ears and neck from the segmented face region by using morphological operations and knowledge about the face structure. Some consistent rules are derived from studying a large number of cases to locate some contour points from the skeleton of the skin-color segmented frontal face region. We extend our previous work (J.G. Wang and E. Sung, 1999), to solving face contour detection of tilted faces as well. The face contour can then be fitted as an ellipse using these points. The least inertia axis of the facial region is used as symmetry axis of the facial region that guarantees the facial contour can also be detected from a tilted face image. Experiments have been done with a number of images of a frontal view of the head including some with a slight pan obtained from an Internet face database. Also, a number of face images captured by a color CCD camera are tried using our method. The robustness is confirmed with a correct detection rate of over 94%.

This paper describes the work of an on-going project investigating the incorporation of physics-based modeling for simulating dynamic interactions in robotic aided manipulations. The research aims at achieving better simulation of robot behaviors in real situations, making it a useful tool for off-line programming. By allowing manipulation strategy simulation and performance visualization of a designed operation to be carried out within a virtual environment, the system will be a valuable tool for handling strategy design, testing, modification, and optimization without fear of damaging the physical systems. As an environment for demonstrating tasks to robots, the systems will also be a useful platform for robot teaching and training. The system has particular promise in flexible assembly where it is more economical to perform the reconfiguration of a work cell and to generate or modify handling strategies for new tasks without having to shut down the real system. This paper reports the initial work carried out with the preliminary results given.