Domain shift is a problem commonly encountered when developing automated histopathology pipelines. The performance of machine learning models such as convolutional neural networks within automated histopathology pipelines is often diminished when applying them to novel data domains due to factors arising from differing staining and scanning protocols. The Dual-Channel Auto-Encoder (DCAE) model was previously shown to produce feature representations that are less sensitive to appearance variation introduced by different digital slide scanners. In this work, the Multi-Channel Auto-Encoder (MCAE) model is presented as an extension to DCAE which learns from more than two domains of data. Experimental results show that the MCAE model produces feature representations that are less sensitive to inter-domain variations than the comparative StaNoSA method when tested on a novel synthetic dataset. This was apparent when applying the MCAE, DCAE, and StaNoSA models to three different classification tasks from unseen domains. The results of this experiment show the MCAE model out performs the other models. These results show that the MCAE model is able to generalise better to novel data, including data from unseen domains, than existing approaches by actively learning normalised feature representations.

The study of mouse social behaviors has been increasingly undertaken in neuroscience research. However, automated quantification of mouse behaviors from the videos of interacting mice is still a challenging problem, where object tracking plays a key role in locating mice in their living spaces. Artificial markers are often applied for multiple mice tracking, which are intrusive and consequently interfere with the movements of mice in a dynamic environment. In this article, we propose a novel method to continuously track several mice and individual parts without requiring any specific tagging. First, we propose an efficient and robust deep-learning-based mouse part detection scheme to generate part candidates. Subsequently, we propose a novel Bayesian-inference integer linear programming (BILP) model that jointly assigns the part candidates to individual targets with necessary geometric constraints while establishing pair-wise association between the detected parts. There is no publicly available dataset in the research community that provides a quantitative test bed for part detection and tracking of multiple mice, and we here introduce a new challenging Multi-Mice PartsTrack dataset that is made of complex behaviors. Finally, we evaluate our proposed approach against several baselines on our new datasets, where the results show that our method outperforms the other state-of-the-art approaches in terms of accuracy. We also demonstrate the generalization ability of the proposed approach on tracking zebra and locust.

Facial phenotyping for medical prediagnosis has recently been successfully exploited as a novel way for the preclinical assessment of a range of rare genetic diseases, where facial biometrics is revealed to have rich links to underlying genetic or medical causes. In this paper, we aim to extend this facial prediagnosis technology for a more general disease, Parkinson's Diseases (PD), and proposed an Artificial-Intelligence-of-Things (AIoT) edge-oriented privacy-preserving facial prediagnosis framework to analyze the treatment of Deep Brain Stimulation (DBS) on PD patients. In the proposed framework, a novel edge-based privacy-preserving framework is proposed to implement private deep facial diagnosis as a service over an AIoT-oriented information theoretically secure multi-party communication scheme, while data privacy has been a primary concern toward a wider exploitation of Electronic Health and Medical Records (EHR/EMR) over cloud-based medical services. In our experiments with a collected facial dataset from PD patients, for the first time, we proved that facial patterns could be used to evaluate the facial difference of PD patients undergoing DBS treatment. We further implemented a privacy-preserving information theoretical secure deep facial prediagnosis framework that can achieve the same accuracy as the non-encrypted one, showing the potential of our facial prediagnosis as a trustworthy edge service for grading the severity of PD in patients.

Simultaneous feature selection and clustering is a major challenge in unsupervised learning. In particular, there has been significant research into saliency measures for features that result in good clustering. However, as datasets become larger and more complex, there is a need to adopt a finer-grained approach to saliency by measuring it in relation to a part of a model. Another issue is learning the feature saliency and advanced model parameters. We address the first by presenting a novel Gaussian mixture model, which explicitly models the dependency of individual mixture components on each feature giving a new component-based feature saliency measure. For the second, we use Markov Chain Monte Carlo sampling to estimate the model and hidden variables. Using a synthetic dataset, we demonstrate the superiority of our approach, in terms of clustering accuracy and model parameter estimation, over an approach using a model-based feature saliency with expectation maximisation. We performed an evaluation of our approach with six synthetic trajectory datasets obtaining an average clustering accuracy of 97 percent. To demonstrate the generality of our approach, we applied it to a network traffic flow dataset obtaining an accuracy of 93 percent for intrusion detection. Finally, we performed a comparison with state-of-the-art clustering techniques using three real-world trajectory datasets of vehicle traffic. Our approach achieved an average clustering accuracy of 96 percent compared to 77-95 percent for the other techniques. In conclusion, for the datasets considered, component based feature saliency measures gave improved clustering over those based on whole models.

Multipopulation is an effective optimization component often embedded into evolutionary algorithms to solve optimization problems. In this paper, a new multipopulation-based multiobjective genetic algorithm (MOGA) is proposed, which uses a unique cross-subpopulation migration process inspired by biological processes to share information between subpopulations. Then, a Markov model of the proposed multipopulation MOGA is derived, the first of its kind, which provides an exact mathematical model for each possible population occurring simultaneously with multiple objectives. Simulation results of two multiobjective test problems with multiple subpopulations justify the derived Markov model, and show that the proposed multipopulation method can improve the optimization ability of the MOGA. Also, the proposed multipopulation method is applied to other multiobjective evolutionary algorithms (MOEAs) for evaluating its performance against the IEEE Congress on Evolutionary Computation multiobjective benchmarks. The experimental results show that a single-population MOEA can be extended to a multipopulation version, while obtaining better optimization performance.

Many approaches to unconstrained face identification exploit small patches which are unaffected by distortions outside their locality. A larger area usually contains more discriminative information, but may be unidentifiable due to local appearance changes across its area, given limited training data. We propose a novel block-based approach, as a complement to existing patch-based approaches, to exploit the greater discriminative information in larger areas, while maintaining robustness to limited training data. A testing block contains several neighboring patches, each of a small size. We identify the matching training block by jointly estimating all of the matching patches, as a means of reducing the uncertainty of each small matching patch with the addition of the neighboring patch information, without assuming additional training data. We further propose a multiscale extension in which we carry out block-based matching at several block sizes, to combine complementary information across scales for further robustness. We have conducted face identification experiments using three datasets, the constrained Georgia Tech dataset to validate the new approach, and two unconstrained datasets, LFW and UFI, to evaluate its potential for improving robustness. The results show that the new approach is able to significantly improve over existing patch-based face identification approaches, in the presence of uncontrolled pose, expression, and lighting variations, using small training datasets. It is also shown that the new block-based scheme can be combined with existing approaches to further improve performance.

Frontal view gait recognition for people identification has been carried out using single RGB, stereo RGB, Kinect 1.0, and Doppler radar. However, existing methods based on these camera technologies suffer from several problems. Therefore, we propose a four-part method for frontal view gait recognition based on the fusion of multiple features acquired from a Time-of-Flight (ToF) camera. We have developed a gait data set captured by a ToF camera. The data set includes two sessions recorded seven months apart, with 46 and 33 subjects, respectively, each with six walks with five covariates. The four-part method includes: a new human silhouette extraction algorithm that reduces the multiple reflection problem experienced by ToF cameras; a frame selection method based on a new gait cycle detection algorithm; four new gait image representations; and a novel fusion classifier. Rigorous experiments are carried out to compare the proposed method with state-of-the-art methods. The results show distinct improvements over recognition rates for all covariates. The proposed method outperforms all major existing approaches for all covariates and results in 66.1% and 81.0% Rank 1 and Rank 5 recognition rates, respectively, in overall covariates, compared with a best state-of-the-art method performance of 35.7% and 57.7%.

Automated recognition of mouse behaviors is crucial in studying psychiatric and neurologic diseases. To achieve this objective, it is very important to analyze the temporal dynamics of mouse behaviors. In particular, the change between mouse neighboring actions is swift in a short period. In this paper, we develop and implement a novel hidden Markov model (HMM) algorithm to describe the temporal characteristics of mouse behaviors. In particular, we here propose a hybrid deep learning architecture, where the first unsupervised layer relies on an advanced spatial-temporal segment Fisher vector encoding both visual and contextual features. Subsequent supervised layers based on our segment aggregate network are trained to estimate the state-dependent observation probabilities of the HMM. The proposed architecture shows the ability to discriminate between visually similar behaviors and results in high recognition rates with the strength of processing imbalanced mouse behavior datasets. Finally, we evaluate our approach using JHuang's and our own datasets, and the results show that our method outperforms other state-of-the-art approaches.

Exemplar-based (or Corpus-based) speech enhancement algorithms have great potential but are typically slow due to needing to search through the entire corpus. The properties of speech can be exploited to improve these algorithms. Firstly, a corpus can be clustered by a phonetic ordering into a search tree which can be used to find a best matching segment. This dramatically reduces the search space, reducing the time complexity of searching a corpus of n segments from O(n) to O(log(n)). Secondly, clustering can be used to give a lossy compression of a speech corpus by replacing original segments with codewords. These techniques are shown in comparison with sequential search and non-compressed corpora using a simple speech enhancement algorithm. A combination of these techniques for a corpus of a quarter of WSJO results in a speedup of approximately 3000×.

Manual measurement of mouse behavior is highly labor intensive and prone to error. This investigation aims to efficiently and accurately recognize individual mouse behaviors in action videos and continuous videos. In our system each mouse action video is expressed as the collection of a set of interest points. We extract both appearance and contextual features from the interest points collected from the training datasets, and then obtain two Gaussian Mixture Model (GMM) dictionaries for the visual and contextual features. The two GMM dictionaries are leveraged by our spatial-temporal stacked Fisher Vector (FV) to represent each mouse action video. A neural network is used to classify mouse action and finally applied to annotate continuous video. The novelty of our proposed approach is: (i) our method exploits contextual features from spatiotemporal interest points, leading to enhanced performance, (ii) we encode contextual features and then fuse them with appearance features, and (iii) location information of a mouse is extracted from spatio-temporal interest points to support mouse behavior recognition. We evaluate our method against the database of Jhuang et al. (Jhuang et al., 2010) and the results show that our method outperforms several state-of-the-art approaches.

Conventional speech enhancement methods, based on frame, multiframe, or segment estimation, require knowledge about the noise. This paper presents a new method that aims to reduce or effectively remove this requirement. It is shown that by using the zero-mean normalized correlation coefficient (ZNCC) as the comparison measure, and by extending the effective length of speech segment matching to sentence-long speech utterances, it is possible to obtain an accurate speech estimate from noise without requiring specific knowledge about the noise. The new method, thus, could be used to deal with unpredictable noise or noise without proper training data. This paper is focused on realizing and evaluating this potential. We propose a novel realization that integrates full-sentence speech correlation with clean speech recognition, formulated as a constrained maximization problem, to overcome the data sparsity problem. Then we propose an efficient implementation algorithm to solve this constrained maximization problem to produce speech sentence estimates. For evaluation, we build the new system on one training dataset and test it on two different test datasets across two databases, for a range of different noises including highly nonstationary ones. It is shown that the new approach, without any estimation of the noise, is able to significantly outperform conventional methods that use optimized noise tracking, in terms of various objective measures including automatic speech recognition.

In this paper, we introduce a novel approach to face recognition which simultaneously tackles three combined challenges: (1) uneven illumination; (2) partial occlusion; and (3) limited training data. The new approach performs lighting normalization, occlusion de-emphasis and finally face recognition, based on finding the largest matching area (LMA) at each point on the face, as opposed to traditional fixed-size local areabased approaches. Robustness is achieved with novel approaches for feature extraction, LMA-based face image comparison and unseen data modeling. On the extended YaleB and AR face databases for face identification, our method using only a single training image per person, outperforms other methods using a single training image, and matches or exceeds methods which require multiple training images. On the labeled faces in the wild face verification database, our method outperforms comparable unsupervised methods. We also show that the new method performs competitively even when the training images are corrupted.

Given the success of patch-based approaches to image denoising, this paper addresses the ill-posed problem of patch size selection. Large patch sizes improve noise robustness in the presence of good matches, but can also lead to artefacts in textured regions due to the rare patch effect; smaller patch sizes reconstruct details more accurately but risk over-fitting to the noise in uniform regions. We propose to jointly optimize each matching patch's identity and size for grayscale image denoising, and present several implementations. The new approach effectively selects the largest matching areas, subject to the constraints of the available data and noise level, to improve noise robustness. Experiments on standard test images demonstrate our approach's ability to improve on fixed-size reconstruction, particularly at high noise levels, on smoother image regions.

It is shown that under certain conditions it is possible to obtain a good speech estimate from noise without requiring noise estimation. We study an implementation of the theory, namely wide matching, for speech enhancement. The new approach performs sentence-wide joint speech segment estimation subject to maximum recognizability to gain noise robustness. Experiments have been conducted to evaluate the new approach with variable noises and SNRs from -5 dB to noise free. It is shown that the new approach, without any estimation of the noise, significantly outperformed conventional methods in the low SNR conditions while retaining comparable performance in the high SNR conditions. It is further suggested that the wide matching and deep learning approaches can be combined towards a highly robust and accurate speech estimator.

Although visual surveillance has emerged as an effective technology for public security, privacy has become an issue of great concern in the transmission and distribution of surveillance videos. For example, personal facial images should not be browsed without permission. To cope with this issue, face image scrambling has emerged as a simple solution for privacy-related applications. Consequently, online facial biometric verification needs to be carried out in the scrambled domain, thus bringing a new challenge to face classification. In this paper, we investigate face verification issues in the scrambled domain and propose a novel scheme to handle this challenge. In our proposed method, to make feature extraction from scrambled face images robust, a biased random subspace sampling scheme is applied to construct fuzzy decision trees from randomly selected features, and fuzzy forest decision using fuzzy memberships is then obtained from combining all fuzzy tree decisions. In our experiment, we first estimated the optimal parameters for the construction of the random forest and, then, applied the optimized model to the benchmark tests using three publically available face datasets. The experimental results validated that our proposed scheme can robustly cope with the challenging tests in the scrambled domain and achieved an improved accuracy over all tests, making our method a promising candidate for the emerging privacy-related facial biometric applications.

With the rapid development of Internet-of-Things (IoT), face scrambling has been proposed for privacy protection during IoT-targeted image/video distribution. Consequently, in these IoT applications, biometric verification needs to be carried out in the scrambled domain, presenting significant challenges in face recognition. Since face models become chaotic signals after scrambling/encryption, a typical solution is to utilize the traditional data-driven face recognition algorithms. While chaotic pattern recognition is still a challenging task, in this paper, we propose a new ensemble approach-many-kernel random discriminant analysis (MK-RDA)-to discover discriminative patterns from the chaotic signals. We also incorporate a salience-aware strategy into the proposed ensemble method to handle the chaotic facial patterns in the scrambled domain, where the random selections of features are made on semantic components via salience modeling. In our experiments, the proposed MK-RDA was tested rigorously on three human face data sets: the ORL face data set, the PIE face data set, and the PUBFIG wild face data set. The experimental results successfully demonstrate that the proposed scheme can effectively handle the chaotic signals and significantly improve the recognition accuracy, making our method a promising candidate for secure biometric verification in the emerging IoT applications.

Visual salience is an intriguing phenomenon observed in biological neural systems. Numerous attempts have been made to model visual salience mathematically using various feature contrasts, either locally or globally. However, these algorithmic models tend to ignore the problem’s biological solutions, in which visual salience appears to arise during the propagation of visual stimuli along the visual cortex. In this paper, inspired by the conjecture that salience arises from deep propagation along the visual cortex, we present a Deep Salience model where a multi-layer model based on successive Markov random fields (sMRF) is proposed to analyze the input image successively through its deep belief propagation. As a result, the foreground object can be automatically separated from the background in a fully unsupervised way. Experimental evaluation on the benchmark dataset validated that our Deep Salience model can consistently outperform many state-of-the-art salience models, yielding the higher rates in the precision-recall tests and attaining the better scores in F-measure and mean-square error tests.

This paper studies single-channel speech separation, assuming unknown, arbitrary temporal dynamics for the speech signals to be separated. A data-driven approach is described, which matches each mixed speech segment against a composite training segment to separate the underlying clean speech segments. To advance the separation accuracy, the new approach seeks and separates the longest mixed speech segments with matching composite training segments. Lengthening the mixed speech segments to match reduces the uncertainty of the constituent training segments, and hence the error of separation. For convenience, we call the new approach Composition of Longest Segments, or CLOSE. The CLOSE method includes a data-driven approach to model long-range temporal dynamics of speech signals, and a statistical approach to identify the longest mixed speech segments with matching composite training segments. Experiments are conducted on the Wall Street Journal database, for separating mixtures of two simultaneous large-vocabulary speech utterances spoken by two different speakers. The results are evaluated using various objective and subjective measures, including the challenge of large-vocabulary continuous speech recognition. It is shown that the new separation approach leads to significant improvement in all these measures.

This paper presents a novel method of audio-visual feature-level fusion for person identification where both the speech and facial modalities may be corrupted, and there is a lack of prior knowledge about the corruption. Furthermore, we assume there are limited amount of training data for each modality (e.g., a short training speech segment and a single training facial image for each person). A new multimodal feature representation and a modified cosine similarity are introduced to combine and compare bimodal features with limited training data, as well as vastly differing data rates and feature sizes. Optimal feature selection and multicondition training are used to reduce the mismatch between training and testing, thereby making the system robust to unknown bimodal corruption. Experiments have been carried out on a bimodal dataset created from the SPIDRE speaker recognition database and AR face recognition database with variable noise corruption of speech and occlusion in the face images. The system's speaker identification performance on the SPIDRE database, and facial identification performance on the AR database, is comparable with the literature. Combining both modalities using the new method of multimodal fusion leads to significantly improved accuracy over the unimodal systems, even when both modalities have been corrupted. The new method also shows improved identification accuracy compared with the bimodal systems based on multicondition model training or missing-feature decoding alone.

In intelligent video surveillance systems, scalability (of the number of simultaneous video streams) is important. Two key factors which hinder scalability are the time spent in decompressing the input video streams, and the limited computational power of the processor. This paper demonstrates how a combination of algorithmic and hardware techniques can overcome these limitations, and significantly increase the number of simultaneous streams. The techniques used are processing in the compressed domain, and exploitation of the multicore and vector processing capability of modern processors. The paper presents a system which performs background modeling, using a Mixture of Gaussians approach. This is an important first step in the segmentation of moving targets. The paper explores the effects of reducing the number of coefficients in the compressed domain, in terms of throughput speed and quality of the background modeling. The speedups achieved by exploiting compressed domain processing, multicore and vector processing are explored individually. Experiments show that a combination of all these techniques can give a speedup of 170 times on a single CPU compared to a purely serial, spatial domain implementation, with a slight gain in quality.

Saliency detection has been a desirable way for robotic vision to find the most noticeable objects in a scene. In this paper, a robust manifold-based saliency estimation method has been developed to help capture the most salient objects in front of robotic eyes, namely cameras. In the proposed approach, an image is considered as a manifold of visual signals (stimuli) spreading over a connected grid, and local visual stimuli are compared against the global image variation to model the visual saliency. With this model, manifold learning is then applied to minimize the local variation while keeping the global contrast, and turns the RGB image into a multi-channel image. After the projection through manifold learning, histogram-based contrast is then computed for saliency modeling of all channels of the projected images, and mutual information is introduced to evaluate each single-channel saliency map against prior knowledge to provide cues for the fusion of multiple channels. In the last step, the fusion procedure combines all single-channel saliency maps according to their mutual information score, and generates the final saliency map. In our experiment, the proposed method is evaluated using one of the largest publicly available image datasets. The experimental results demonstrate that our algorithm consistently outperforms the state-of-the-art unsupervised saliency detection methods, yielding higher precision and better recall rates. Furthermore, the proposed method is tested on a video-type test dataset where a moving camera is trying to catch up with the walking person---a salient object in the video sequence. Our experimental results show that the proposed approach can successful accomplish this task, revealing its potential use for similar robotic applications.

In this paper we demonstrate a simple and novel illumination model that can be used for illumination invariant facial recognition. This model requires no prior knowledge of the illumination conditions and can be used when there is only a single training image per-person. The proposed illumination model separates the effects of illumination over a small area of the face into two components; an additive component modelling the mean illumination and a multiplicative component, modelling the variance within the facial area. Illumination invariant facial recognition is performed in a piecewise manner, by splitting the face image into blocks, then normalizing the illumination within each block based on the new lighting model. The assumptions underlying this novel lighting model have been verified on the YaleB face database. We show that magnitude 2D Fourier features can be used as robust facial descriptors within the new lighting model. Using only a single training image per-person, our new method achieves high (in most cases 100%) identification accuracy on the YaleB, extended YaleB and CMU-PIE face databases.

Given single-channel recordings of simultaneous speakers, we may need to identify the individual speakers for separating their voices. In this paper, we consider the problem of identifying two simultaneous speakers based on single-channel data, i.e., speakerpair identification. We model the problem as identifying speakers using noisy speech with partial temporal corruption, which corresponds to the heavily mixed speech frames. Inclusion of these noisy frames will damage the accuracy of both speakers identification. In this paper, we propose a new approach to automatically and optimally select the single-speaker dominated speech frames for identification. The new algorithm has been evaluated using two databases: 1) the GRID speech separation database and 2) the Wall Street Journal (WSJ0) database. The new approach has shown better performance than other approaches. On the Grid database, for example, the new approach outperformed the state of the art IBM approach in 5 out of 6 test conditions.

This paper presents a novel method of audio-visual fusion for person identification where both the speech and facial modalities may be corrupted, and there is a lack of prior knowledge about the corruption. Furthermore, we assume there is a limited amount of training data for each modality (e.g., a short training speech segment and a single training facial image for each person). A new representation and a modified cosine similarity are introduced for combining and comparing bimodal features with limited training data as well as vastly differing data rates and feature sizes. Optimal feature selection and multicondition training are used to reduce the mismatch between training and testing, thereby making the system robust to unknown bimodal corruption. Experiments have been carried out on a bimodal data set created from the SPIDRE and AR databases with variable noise corruption of speech and occlusion in the face images. The new method has demonstrated improved recognition accuracy.

Temporal dynamics and speaker characteristics are two important features of speech that distinguish speech from noise. In this paper, we propose a method to maximally extract these two features of speech for speech enhancement. We demonstrate that this can reduce the requirement for prior information about the noise, which can be difficult to estimate for fast-varying noise. Given noisy speech, the new approach estimates clean speech by recognizing long segments of the clean speech as whole units. In the recognition, clean speech sentences, taken from a speech corpus, are used as examples. Matching segments are identified between the noisy sentence and the corpus sentences. The estimate is formed by using the longest matching segments found in the corpus sentences. Longer speech segments as whole units contain more distinct dynamics and richer speaker characteristics, and can be identified more accurately from noise than shorter speech segments. Therefore, estimation based on the longest recognized segments increases the noise immunity and hence the estimation accuracy. The new approach consists of a statistical model to represent up to sentence-long temporal dynamics in the corpus speech, and an algorithm to identify the longest matching segments between the noisy sentence and the corpus sentences. The algorithm is made more robust to noise uncertainty by introducing missing-feature based noise compensation into the corpus sentences. Experiments have been conducted on the TIMIT database for speech enhancement from various types of nonstationary noise including song, music, and crosstalk speech. The new approach has shown improved performance over conventional enhancement algorithms in both objective and subjective evaluations.

Temporal dynamics and speaker characteristics are two important features of speech that distinguish speech from noise. In this paper, we propose a method to maximally extract these two features of speech for speech enhancement. We demonstrate that this can reduce the requirement for prior information about the noise, which can be difficult to estimate for fast-varying noise. Given noisy speech, the new approach estimates clean speech by recognizing long segments of the clean speech as whole units. In the recognition, clean speech sentences, taken from a speech corpus, are used as examples. Matching segments are identified between the noisy sentence and the corpus sentences. The estimate is formed by using the longest matching segments found in the corpus sentences. Longer speech segments as whole units contain more distinct dynamics and richer speaker characteristics, and can be identified more accurately from noise than shorter speech segments. Therefore, estimation based on the longest recognized segments increases the noise immunity and hence the estimation accuracy. The new approach consists of a statistical model to represent up to sentence-long temporal dynamics in the corpus speech, and an algorithm to identify the longest matching segments between the noisy sentence and the corpus sentences. The algorithm is made more robust to noise uncertainty by introducing missing-feature based noise compensation into the corpus sentences. Experiments have been conducted on the TIMIT database for speech enhancement from various types of nonstationary noise including song, music, and crosstalk speech. The new approach has shown improved performance over conventional enhancement algorithms in both objective and subjective evaluations.

In this paper, an unsupervised scheme for stereo video matting is presented, where stereo motion analysis is combined to provide an automatic multi-layer clustering scheme of alpha components. With this multi-layer matting scheme, objects in both foreground and background can be extracted for background substitution. The experiment shows that the proposed scheme can have a better performance in term of automatic grouping of alpha components in comparison with min-cut matte grouping.

In this paper, a novel pattern recognition scheme, global harmonic subspace analysis (GHSA), is developed for face recognition. In the proposed scheme, global harmonic features are extracted at the semantic scale to capture the 2-D semantic spatial structures of a face image. Laplacian Eigenmap is applied to discriminate faces in their global harmonic subspace. Experimental results on the Yale and PIE face databases show that the proposed GHSA scheme achieves an improvement in face recognition accuracy when compared with conventional subspace approaches, and a further investigation shows that the proposed GHSA scheme has impressive robustness to noise.

In this paper, a novel video-based multimodal biometric verification scheme using the subspace-based low-level feature fusion of face and speech is developed for specific speaker recognition for perceptual human-computer interaction (HCI). In the proposed scheme, human face is tracked and face pose is estimated to weight the detected facelike regions in successive frames, where ill-posed faces and false-positive detections are assigned with lower credit to enhance the accuracy. In the audio modality, mel-frequency cepstral coefficients are extracted for voice-based biometric verification. In the fusion step, features from both modalities are projected into nonlinear Laplacian Eigenmap subspace for multimodal speaker recognition and combined at low level. The proposed approach is tested on the video database of ten human subjects, and the results show that the proposed scheme can attain better accuracy in comparison with the conventional multimodal fusion using latent semantic analysis as well as the single-modality verifications. The experiment on MATLAB shows the potential of the proposed scheme to attain the real-time performance for perceptual HCI applications.

This paper investigates the problem of face recognition with partially occluded images without assuming prior information about the distortion, and with only a single training image or a small number of training images for each class to be identified. A new approach is presented, which is an extension of our previous posterior union model. The new approach is formulated by using a similarity measure in place of the probability measure, thereby allowing the use of a single training image to represent a class. The new approach achieves improved robustness to partial occlusion by focusing the recognition mainly on the matched local regions, which are selected automatically subject to an optimality criterion to maximize the similarity of the correct class. Two databases, XM2VTS and AR, have been used to evaluate the new approach. The results indicate that the new system is able to perform as well as an oracle model for dealing with various simulated and realistic partial distortions/occlusions without requiring prior information.

In this paper, we suggest a solution for the problem of scene-of-crime shoeprints retrieval based on the use of multi-scale Harris points, which are a set of very distinctive points of interest in an image, combined with SIFT descriptor. We show that such combination can overcome the issue of retrieval of partial prints in the presence of scale and rotation distortions with Gaussian noise perturbation. Excellent results were obtained for synthetic scene images, clearly outperforming published results in the literature.

This paper presents a new approach to face recognition where the images are subject to unknown, partial distortion/occlusion. The new approach is a probabilistic decision-based neural network (PDBNN), built on a statistical method called the posterior union model (PUM). PUM is an approach for ignoring severely mismatched local features and focusing the recognition mainly on the matched local features. It thereby improves the robustness while assuming no prior information about the corruption. We call the new approach the posterior union decision-based neural network (PUDBNN). The new PUDBNN has been evaluated on two face image databases, XM2VTS and ORL, using testing images subjected to various types of partial distortion and occlusion. The new system has demonstrated improved performance over other systems.

One of the most difficult problems in automatic shoeprint classification is the matching of partial shoeprint images. This task becomes more challenging in the presence of geometric distortions (e.g. translated and/or rotated partial prints). In this paper, we evaluate the performance of advanced correlation filters (ACFs) for the automatic classification of partial shoeprints. The optimal trade-off synthetic discriminant function (OTSDF) filter and the unconstrained OTSDF (UOTSDF) filter, in particular, were used to match partial shoeprint images with different qualities. Experimental assessment using a shoeprint image database has demonstrated the efficient classification performance of ACFs compared to other state-of-the-art methods.

This paper deals with the retrieval of scene-of-crime (or scene) shoeprint images from a reference database of shoeprint images by using a new local feature detector and an improved local feature descriptor. Our approach is based on novel modifications and improvements of a few recent techniques in this area: (1) the scale adapted Harris detector, which is an extension to multi-scale domains of the Harris corner detector; (2) automatic scale selection by the characteristic scale of a local structure. (3) SIFT (Scale-Invariant Feature Transform), one of the most widely investigated descriptors in recent years. Like most of other local feature representations, the proposed approach can also be divided into two stages: (i) a set of distinctive local features are selected by first detecting scale adaptive Harris corners where each of them is associated with a scale factor, and then selecting as the final result only those corners whose scale matches the scale of blob-like structures around them. Here, the scale of a blob-like structure is detected by the Laplace-based scale selection, (ii). for each feature detected, an enhanced SIFT descriptor is computed to represent this feature. Our improvements lead two novel methods which we call the Modified Harris-Laplace (MHL) detector, and the enhanced SIFT descriptor. In this paper, we demonstrate the application of the proposed scheme to the shoeprint image retrieval problem using six sets of synthetic scene images, 50 images for each, and a database of 500 reference shoeprint images. The retrieval performance of the proposed approach is significantly better, in terms of cumulative matching score, than the existing methods investigated in this application area, such as edge directional histogram, power spectral distribution, and pattern & topological spectra.

This paper deals with the retrieval of scene-of-crime (or scene) shoeprint images from a reference database of shoeprint images by using a new local feature detector and an improved local feature descriptor. Our approach is based on novel modifications and improvements of a few recent techniques in this area: (1) the scale adapted Harris detector, which is an extension to multi-scale domains of the Harris corner detector; (2) automatic scale selection by the characteristic scale of a local structure. (3) SIFT (Scale-Invariant Feature Transform), one of the most widely investigated descriptors in recent years. Like most of other local feature representations, the proposed approach can also be divided into two stages: (i) a set of distinctive local features are selected by first detecting scale adaptive Harris corners where each of them is associated with a scale factor, and then selecting as the final result only those corners whose scale matches the scale of blob-like structures around them. Here, the scale of a blob-like structure is detected by the Laplace-based scale selection. (ii). for each feature detected, an enhanced SIFT descriptor is computed to represent this feature. Our improvements lead two novel methods which we call the Modified Harris-Laplace (MHL) detector, and the enhanced SIFT descriptor. In this paper, we demonstrate the application of the proposed scheme to the shoeprint image retrieval problem using six sets of synthetic scene images, 50 images for each, and a database of 500 reference shoeprint images. The retrieval performance of the proposed approach is significantly better, in terms of cumulative matching score, than the existing methods used in this application area, such as edge directional histogram, power spectral distribution, and pattern & topological spectra.

In this paper, a method for automatically recognizing partial shoeprint images for use in forensic science is presented. The technique uses the phase-only correlation (POC) for shoeprints matching. The main advantage of this method is its capability to match low quality shoeprint images accurately and efficiently. In order to achieve superior performance, the use of a spectral weighting function is also proposed. Experiments were conducted on a database of images of 100 different shoes available on the market. For experimental evaluation, test images including different perturbations such as noise addition, blurring and textured background addition were generated. Results have shown that the proposed method is very practical and provides high performance when processing low quality partial-prints. The use of a weighting function provides an improvement in the recognition rate in particularly difficult cases.

This paper presents a high-level hardware description environment developed at Queen's University, Belfast, U.K., which aims to bridge the gap between application design and hardware description. The environment, called application-to-hardware (A2H), allows for efficient compilation of high-level application descriptions to field programmable gate array (FPGA) hardware in the form of EDIF netlist in seconds. A key concept in bridging the gap while retaining the hardware efficiency, is that of hardware skeletons. A hardware skeleton is a parameterized description of a task-specific architecture, to which the user can supply not only value parameters but also functions or even other skeletons. A skeleton contains built-in rules, which capture optimizations specific to the target hardware at the implementation phase. The rule-based logic programming language Prolog has been chosen as the base notation for the A2H environment. This paper includes descriptions of hardware skeletons abstractions in the particular context of image processing applications. The current implementation of our system targets Xilinx XC4000 and Virtex series FPGAs.

This paper presents a logic-based approach to hardware abstraction and composition based on the logic programming language Prolog. This is an attempt to satisfy the dual requirement of abstract hardware design and hardware efficiency. Central to this approach is a hardware description environment called HIDE, which provides more abstract and elegant hardware descriptions and compositions than are possible in traditional hardware description languages such as VHDL or Verilog. The environment enables highly scaleable and parameterized composition of blocks using a small set of constructors e.g. 'horizontal' and 'vertical' for 2D circuit abstractions and the novel 'above' constructor for 3D circuit compositions. It also generates preplaced configurations in EDIF (and VHDL) format for Xilinx FPGAs (field programmable gate arrays).

This paper gives a design framework for the implementation of the 2D (two-dimensional) orthogonal discrete wavelet transform (DWT) on FPGA (field programmable gate array). The architecture is based on the pyramid algorithm analysis. It maps spatially the multistage filter banks of the DWT on Xilinx Virtex-e FPGA family using on chip buffering. The architecture takes advantage from the low rate of the high transform stages to reuse the logic. In this paper, we propose an FIR structure to handle the computation along the borders using symmetry extension, a new BlockRam configuration for multi ports shift register, and a mathematical approach to predict and reduce the error dynamic range due to wordlength rounding. For an MxM image size input, our architecture has a period of M/sup 2/ clock cycles, and requires the minimum storage size. The architecture is highly scalable for different filter lengths and number of octaves. The implementation results for a specific 2D Doubechies-4 wavelet transform are included.

FIR (Finite Impulse Response) filters are often used in digital signal processing. This paper presents architecture for FIR filters on Xilinx Virtex FPGAs (field programmable gate arrays). The architecture is particularly useful for handling the problem of signal boundaries filtering, which occurs in finite length signal processing (e.g. image processing). Based on a bit parallel arithmetic, our architecture is fully scalable and parameterized. It cleverly exploits the Shift Register Logic (SRL16) component of the Virtex family. The implementation leads to considerable area savings compared to the conventional implementation (based on a hard router) with no speed penalty. A case study based on the implementation of the standard low filter of the Daubechies-8 wavelet on Xilinx Virtex-E FPGAs is presented.

This paper presents a novel approach to hardware abstraction based on the logic programming language Prolog. This is an attempt to satisfy the dual requirement of abstract hardware design and hardware efficiency. Central to this approach is a hardware description environment called HIDE, which provides more abstract hardware descriptions and compositions than are possible in traditional hardware description languages such as VHDL or Verilog. HIDE enables highly scaleable and parameterised composition of blocks using a small set of abstract constructors such as the horizontal and vertical constructors for 2D circuit abstractions, and the novel above constructor for 3D circuit abstractions. It also generates pre-placed configurations in EDIF (and VHDL) format for Xilinx FPGAs. The paper presents the syntax and semantics of HIDE and illustrates our logic-based approach in the construction of a high performance Matrix Multiplier core for Xilinx Virtex FPGAs.

This paper gives a design framework for the implementation of the 2-D Orthogonal Discrete Wavelet Transform (DWT) on FPGA. The architecture is based on the Pyramid Algorithm Analysis. Our architecture spatially maps the multistage filter banks of the DWT onto the Xilinx Virtex-E FPGA family. In this paper we propose a novel FIR structure to handle the computation along the borders using symmetric extension. The paper includes a new detailed mathematical approach to determine the architecture's dynamic range as well as predicting and reducing the error dynamic range due to wordlength rounding. For an NxN image size input, our architecture has a period of N2 clock cycles, and requires only the minimum storage size. The architecture is highly scalable for different filter lengths and number of octaves. The implementation results for a specific 2-D Daubechies-4 Wavelet Transform are included.

This paper describes an architecture based on a serial iterative algorithm for Image Connected Component Labelling with a hardware complexity O(N) for an NxN image. The algorithm iteratively scans the input image, performing a recursive non-zero maximum neighbourhood operation. A complete forward pass is followed by an inverse pass in which the image is scanned in reverse order. The process is repeated until no change in the image occurs. The algorithm has been coded in Handel C language and targeted to a Celoxica RC1000-PP PCI board, which is based on a Virtex XCV2000E-6 FPGA. The whole design was fully implemented and tested on real hardware in less than 24 man-hours of work. The implementation speed (pixel throughput) is virtually independent of the image size and is equal to ~34MHz. For 1024x1024 input images, the whole circuit consumes 566 Slices and 5 BlockRAMs and can run at 34 MHz, leading to a 32 pass/sec performance.

This paper analyses the effects of fixed-point arithmetic in FIR filter based architectures, based on the roundoff statistical noise model. A novel approach, which allows diminishing the error dynamic range and predicting its value according to the wordlength precision, is suggested. This permits the user to preset the fraction precision according to the sought architecture's precision. The efficiency of this approach is demonstrated through the 2D DWT biorthogonal 9&7 transform.

This paper presents a high-level hardware description environment based on the logic programming language Prolog, called HIDE. The latter has been designed in an attempt to address the problem of abstract hardware design and hardware efficiency. HIDE provides more abstract hardware descriptions and compositions than are possible in traditional hardware description languages such as VHDL or Verilog. It enables highly scaleable and parameterised composition of blocks using a small set of constructors (e.g. horizontal, vertical composition), and generates pre-placed configurations in EDIF format for Xilinx Virtex FPGAs. The paper presents the syntax and semantics of HIDE and illustrates its use in the construction of a bit parallel multiplier core for Xilinx Virtex FPGAs.

Symmetric FIR filters, which provide linear phases, are frequently used in digital signal processing. This paper presents the design and implementation of a novel architecture for symmetric FIR filters on Xilinx Virtex FPGAs. The architecture is particularly useful for handling the problem of processing signal boundaries, which occurs in finite length signal processing (e.g. image processing). Based on bit parallel arithmetic, our architecture is fully scalable and parameterised. It takes into account the details of the symmetry and exploits the features of Xilinx Virtex FPGAs. The implementation leads to considerable area savings compared to conventional implementations (based on a hard router), at the expense of using a clock doubler, which reduces the overall processing speed. The latter is however still high enough to achieve real time performance. Moreover, our architecture can match the speed of a conventional implementation if the filter output is going to be decimated, as it is the case in multirate applications (e.g. wavelets).

Although FPGA technology offers the potential of designing high performance systems at low cost for a wide range of applications, its programming model is prohibitively low level requiring either a dedicated FPGA-experienced programmer or basic digital design knowledge. To allow a signal/image processing end-user to benefit from this kind of device, the level of design abstraction needs to be raised, even beyond a hardware description language level (e.g. VHDL). This approach will help the application developer to focus on signal/image processing algorithms rather than on low-level designs and implementations. This paper aims to present a framework for an FPGA-based coprocessor dedicated to discrete wavelet transforms (DWT). The proposed approach will help the end-user to generate FPGA configurations for DWT at the highest level without any knowledge of the low-level design styles and architectures.

The inherent reprogrammability of field programmable gate arrays (FPGAs) gives them some of the flexibility of software while keeping the performance advantages of an application specific hardware solution. However, the main disadvantage of FPGAs is the low level of their programming model. Although software tools have been drastically improved since the early days of this new technology, they still require the user to think at the hardware level rather than at the algorithmic level. To bridge the gap between the application and implementation levels, we present a high level software environment for FPGA based real time video processing, which aims to hide hardware details completely from the user. Our approach is to provide a flexible FPGA-based image processing coprocessor with a very high level programming interface based on the core operators of image algebra. Our system has been successfully implemented on VISICOM's VigraVision/sup TM/ PCI board giving real time processing of video data.

The most common clues left at a crime scene when a crime is committed are shoeprint impressions. These impressions are useful in the detection of criminals and the linking of crime scenes. A novel technique for use in the detection and classification of shoeprint impressions has been developed. The technique is based on fractal based feature extraction and pattern matching methods. The computerized system developed has been extensively tested on a large database of real shoeprint impressions and is robust to small variations of image orientations and/or translations.

A new bi-directional bit serial-parallel multiplication architecture is presented. The proposed structure is regular and modular, and requires nearest neighbour communication links only, which makes it more efficient for VLSI implementation. Furthermore, a judicious deployment of latches in the circuit ensures that the multiplier operates on two coefficients of the multiplicand at the same time thus speeding up the process. Comparison of the new multiplier structure with previous ones has shown the superiority of the new architecture.

Digit-serial architectures obtained using traditional unfolding and folding techniques cannot be pipelined beyond a certain level because of the presence of feedback loops. In this paper, a novel approach for the design of pipelined digit serial-parallel multipliers is presented.

This paper describes a domain specific programming model for execution on parallel and distributed architectures. The model has initially been targeted at the application area of image processing, though the techniques developed may be more generally applicable to other domains where an algebraic or library-based approach is common. Efficiency is achieved by the concept of a self-optimising class library of primitive image processing operations, which allows programs to be written in a high level, algebraic notation and which is automatically parallelised (using an application-specific data parallel approach). The class library is extended automatically with optimised operations, generated by a transformation system, giving improved execution performance. The parallel implementation of the model described here is based on MPI and has been tested on a C40 processor network, a quad-processor Unix workstation, and a network of PCs running Linux. Timings are included to indicate the impact of the automatic optimisation facility (rather than the effect of parallelisation). Copyright © 1999 John Wiley & Sons, Ltd.

This paper describes the use of Prolog as an implementation language for a compiler for a parallel image processing language. The target machine for which code is generated is an abstract model for parallel image processing. The structure of the final compiler closely reflects the BNF syntax definition of the image processing language thus allowing ease of implementation and maintainability to be achieved with relatively little effort.

This paper presents an algorithm for determining the hierarchical structure of a set of procedures, such that procedures are nested as tightly as possible. The algorithm is developed by first defining the problem more precisely. Arguments in favour of its optimality are presented. The algorithm has been used in the automatic generation of readable top-down parsers from extended BNF grammars.

A compiler for a Pascal-based language Actus is described. The language is suitable for the expression of the type of parallelism offered by both array and vector processors. The implementation described is for the Cray-1 computer. An objective of the implementation has been to construct an optimizing compiler which can be readily adapted for a range of array and vector processors. As a result the machine-dependent sections of the compiler have been clearly identified.

Abstract This paper describes a recent experiment in the design, implementation and application of a programming language designed specifically for distributed systems. The fundamental nature of such systems is used in the derivation of an experimental language, which is described and illustrated. Having outlined an implementation, the project is evaluated and some of the more significant findings are reported. In particular, the paper discusses (and argues for) an explicit language facility for expressing the distribution of a program over an actual system.

Abstract A system is described which inputs EBNF syntax equations as text, checks them and builds a corresponding syntax graph representation. An EBNF parser, with full error recovery, is included. The system is designed using the principles of modular decomposition and data abstraction, and is presented as a case study in the application of these principles to program design. The system is programmed in Pascal‐plus, and has been used as a basis for the automatic generation of parsers.

Abstract This paper describes, in an informal manner, the programming language ACTUS which was designed to facilitate programming array processing and vector processing ‘supercomputers’. ACTUS extends the program structuring and data structuring facilities of Pascal to the synchronous parallel environment as represented by array and vector processor architectures. A knowledge of Pascal is assumed and only the parallel features of ACTUS are described.