Although there have been some promising results in computer lipreading, there has been a paucity of data on which to train automatic systems. However the recent emergence of the TCD-TIMIT corpus, with around 6000 words, 59 speakers and seven hours of recorded audio-visual speech, allows the deployment of more recent techniques in audio-speech such as Deep Neural Networks (DNNs) and sequence discriminative training. In this paper we combine the DNN with a Hidden Markov Model (HMM) to the, so called, hybrid DNN-HMM configuration which we train using a variety of sequence discriminative training methods. This is then followed with a weighted finite state transducer. The conclusion is that the DNN offers very substantial improvement over a conventional classifier which uses a Gaussian Mixture Model (GMM) to model the densities even when optimised with Speaker Adaptive Training. Sequence adaptive training offers further improvements depending on the precise variety employed but those improvements are of the order of ~10\% improvement in word accuracy. Putting these two results together implies that lipreading is moving from something of rather esoteric interest to becoming a practical reality in the foreseeable future.

Recent improvements in tracking and feature extraction mean that speaker-dependent lip-reading of continuous speech using a medium size vocabulary (around 1000 words) is realistic. However, the recognition of previously unseen speakers has been found to be a very challenging task, because of the large variation in lip-shapes across speakers and the lack of large, tracked databases of visual features, which are very expensive to produce. By adapting a technique that is established in speech recognition but has not previously been used in lip-reading, we show that error-rates for speaker-independent lip-reading can be very significantly reduced. Furthermore, we show that error-rates can be even further reduced by the additional use of Deep Neural Networks (DNN). We also find that there is no need to map phonemes to visemes for context-dependent visual speech transcription.

To undertake machine lip-reading, we try to recognise speech from a visual signal. Current work often uses viseme classification supported by language models with varying degrees of success. A few recent works suggest phoneme classification, in the right circumstances, can outperform viseme classification. In this work we present a novel two-pass method of training phoneme classifiers which uses previously trained visemes in the first pass. With our new training algorithm, we show classification performance which significantly improves on previous lip-reading results.

Visual-only speech recognition is dependent upon a number of factors that can be difficult to control, such as: lighting; identity; motion; emotion and expression. But some factors, such as video resolution are controllable, so it is surprising that there is not yet a systematic study of the effect of resolution on lip-reading. Here we use a new data set, the Rosetta Raven data, to train and test recognizers so we can measure the affect of video resolution on recognition accuracy. We conclude that, contrary to common practice, resolution need not be that great for automatic lip-reading. However it is highly unlikely that automatic lip-reading can work reliably when the distance between the bottom of the lower lip and the top of the upper lip is less than four pixels at rest.

Computer lip-reading is one of the great signal processing challenges. Not only is the signal noisy, it is variable. However it is almost unknown to compare the performance with human lip-readers. Partly this is because of the paucity of human lip-readers and partly because most automatic systems only handle data that are trivial and therefore not representative of human speech. Here we generate a multiview dataset using connected words that can be analysed by an automatic system, based on linear predictive trackers and active appearance models, and human lip-readers. The automatic system we devise has a viseme accuracy of ≈ 46% which is comparable to poor professional human lip-readers. However, unlike human lip-readers our system is good at guessing its fallibility.

Computer lip-reading systems are usually designed to work using a full-frontal view of the face. However, many human experts tend to prefer to lip-read using an angled view. In this paper we consider issues related to the best viewing angle for an automated lip-reading system. In particular, we seek answers to the following questions: (1) Do computers lip-read better using a frontal or a non-frontal view of the face? (2) What is the best viewing angle for a computer lip-reading system? (3) How can a computer lip-reading system be made to work independently of viewing angle? We investigate these issues using a purpose built audio-visual dataset that contains simultaneous recordings of a speaker reciting continuous speech at five angles. We find that the system performs best on a non-frontal view, perhaps because lip gestures, such as lip-protrusion and lip-rounding, are more pronounced when viewing from an angle. We also describe a simple linear mapping that allows us to map any view of the face to the view that we find to be optimal. Hence we present a view-independent lip-reading system.

This paper proposes a learnt data-driven approach for accurate, real-time tracking of facial features using only intensity information. Constraints such as a-priori shape models or temporal models for dynamics are not required or used. Tracking facial features simply becomes the independent tracking of a set of points on the face. This allows us to cope with facial configurations not present in the training data. Tracking is achieved via linear predictors which provide a fast and effective method for mapping pixel-level information to tracked feature position displacements. To improve on this, a novel and robust biased linear predictor is proposed in this paper. Multiple linear predictors are grouped into a rigid flock to increase robustness. To further improve tracking accuracy, a novel probabilistic selection method is used to identify relevant visual areas for tracking a feature point. These selected flocks are then combined into a hierarchical multi-resolution LP model. Experimental results also show that this method performs more robustly and accurately than AAMs, without any a priori shape information and with minimal training examples.

A new method for adapting digital images so that they are suitable for color blind viewers is presented. In contrast to earlier automatic methods which formulate the problem of adapting images for color blind observers as one of optimization, we demonstrate how it is possible to allow a user to compute a very wide range of adaptations in reasonable time under the control of a single variable. We demonstrate how the algorithm can be delivered as an adaptive technology via a simple interface, and evaluate the efficacy of our method using psychovisual experiments with simulated color blind users and a standard color vision test.

Important visual information often disappears when color documents are viewed by color blind people. The algorithm introduced here maps colors using the World Wide Web Consortium evaluation criteria so that detail is preserved for color blind viewers, especially dichromats. The algorithm has four parts: 1) select a representative set of colors from the source document; 2) compute target color distances using color and brightness differences; 3) solve an optimization step that preserves the target distances for a particular class of color blind viewer; and 4) interpolate the mapped colors across the remaining colors in the document. We demonstrate the efficacy of our method using simulations and critique our method in the context of earlier work.

This work proposes a noise reduction system for use in high noise environments such as motor racing. First, an analysis of the audio data received from the driver reveals that the main noise sources are from the engine, airflow and tyres. These are found to relate to engine speed, road speed and throttle position information that is received in a data stream from the car's on-board computer. A two-stage noise compensation strategy is proposed which first suppresses engine harmonics using adaptive filtering with engine speed reference information taken from the data stream. Second, a maximum a posteriori (MAP) prediction of the tyre and airflow noise is made from data stream values and this is combined with spectral subtraction for noise suppression. Human listening tests reveal that both noise reduction stages lead to good improvements in the intelligibility of the speech with a comparative mean opinion score (CMOS) of +1.57 being obtained.

Two new texture features, based on morphological scale-space processors are introduced. The new methods are shown to have good performance over a variety of tests. We demonstrate that if texture classifies are to be used in real world scenes, then the choice of test is critical and that Brodatz-like tests are unlikely to represent reality. 1

This paper constructs a texture feature extractor based on a morphological scale-space. It produces features that are invariant to rotation of the texture. The features are used with a very simple k-nearest neighbour classifier and tested using the Outex methodology. The classifier has comparable performance to a number of benchmark classifiers but uses fewer features. The algorithm is quick to compute and provides insight into the underlying structure of texture.

Artists pictures rarely have photo-realistic detail. Tools to create pictures from digital photographs might, therefore, include methods for removing detail. These tools such as Gaussian and anisotropic diffusion filters and connected-set morphological filters (sieves) remove detail whilst maintaining scale-space causality, in other words new detail is not created using these operators. Non-photorealistic rendering is, therefore, a potential application of these vision techniques. It is shown that certain scale-space filters preserve the appropriate edges of retained segments of interest. The resulting images have fewer extrema and are perceptually simpler than the original. A second artistic goal is to accentuate the centre of attention by reducing detail away from the centre. The process also removes the detail providing perceptual cues about photographic texture. This allows the ‘eye ’ to readily accept alternative, artistic, textures introduced to further create an artistic impression. Moreover, the edges bounding segments accurately represent shapes in the original image and so provide a starting point for sketches. 1

The problem of extracting spots from DNA microarrays is a problem of considerable scientific and economic utility. In this paper we introduce a new approach based on a scale-space analysis of the image. We augment this with a machine learning system that guides an operator by classifying spots into those that require further attention and those that are already segmented correctly. We compare conventional k-nearest neighbor techniques with generalized linear models and multilayer perceptrons using confidence intervals and McNemar's test.

A criterion function for selecting the optimal payload size in data transmission over a binary symmetrical communication channel is presented. The work assumes that the packets comprise a fixed size header and variable length payload. The conventional criterion function is based on the ratio between payload size and mean number of transmitted bits per packet, which includes re-transmissions of corrupted packets. Applying a logarithm to the inverse of this criterion function makes the resulting function very suitable for mathematical analysis. This also facilitates parameter sensitivity studies without recourse to numerical methods. As such, this helps to visualise and understand the problem of optimal payload length selection when teaching courses in signal processing for communications and multimedia.

The multimodal nature of speech is often ignored in human-computer interaction, but lip deformations and other body motion, such as those of the head, convey additional information. We integrate speech cues from many sources and this improves intelligibility, especially when the acoustic signal is degraded. The paper shows how this additional, often complementary, visual speech information can be used for speech recognition. Three methods for parameterizing lip image sequences for recognition using hidden Markov models are compared. Two of these are top-down approaches that fit a model of the inner and outer lip contours and derive lipreading features from a principal component analysis of shape or shape and appearance, respectively. The third, bottom-up, method uses a nonlinear scale-space analysis to form features directly from the pixel intensity. All methods are compared on a multitalker visual speech recognition task of isolated letters.

This paper explores a new method for analysing and comparing image histograms. The technique amounts to a novel way of backprojecting an image into one with fewer, statistically significant colours. When the method is tested with the COIL-100 and MPEG-7 data sets it is shown to have a performance that is as good as the best methods using fewer enteries than the original histogram. Therefore it offers the potential for extending the use of histograms into high dimensional feature spaces. 1

Color histograms are a central feature in many image retrieval systems. Indeed they are part of the MPEG-7 standard. But histograms suffer from the "curse of dimensionality " in which the number of bins increases exponentially with the number of dimensions. There is therefore an imperative for methods for simplifying histograms. This paper presents a new method for simplifying histograms based on a cascade of increasing-scale graph morphology filters. The system we choose preserves scale space causality and so preserves the modes of the histogram. The method is quick to compute so is therefore a practically useful feature. We present results using the MPEG-7 Common Color Dataset that show that these new compressed features have a retrieval performance that is equivalent to full histograms.

A new way of representing histograms is presented. The representation is based on a hierarchical scale-space decomposition that forms a scale tree. The tree is a mapping of the original histogram and provides a useful starting point for the visualization and analysis of high-dimensional histograms that are commonplace in image retrieval systems.

Modern neural and adaptive systems often have complicated error performance surfaces with many local extrema. Visualising and understanding these surfaces is critical to effective tuning of these systems but almost all visualisation methods are confined to two dimensions. Here we show how to use a morphological scale-space transform to convert these multi-dimensional complex error surfaces into two-dimensional trees where the leaf nodes are local minima and other nodes represent decision points such as saddle points and points of inflection.

Design of digital infinite impulse response (IIR) filters is a compulsory topic in most signal processing courses. Most often, it is taught by using the bilinear transform to map an analogue counterpart into the corresponding digital filter. The usual approach is to define a mapping between the complex variables s and z, and hence, by substitution, derive a mapping between /spl omega/, analogue frequency, and /spl theta/, sampled frequency. This is rather elliptical, since the real aim is to establish the correspondence between the frequency response of a prototype analogue system H(j/spl omega/), and H(e/sup j/spl theta//), the response of the sampled system. Here we provide a rigorous analysis for the mutual invertibility between the analogue frequency /spl omega/, and the digital frequency /spl theta/ for this case. Based upon the definition of the tan and arctan functions, conditions of existence, uniqueness and continuity of such a mutually inverse mapping are derived. Based upon these results, simple proofs for the mutually inverse mappings /spl omega//spl rarr//spl theta/ and /spl theta//spl rarr//spl omega/ are given. This is supported by appropriate diagrams. This problem arose as a student question while teaching DSP.

Images are re-mapped as scale-space trees. The minimal data structure is then augmented by "complement nodes" to increase the practical value of the representation. It is then shown how the resulting ctree can be used to remove noise from images, provide a hierarchical way to estimate a dense disparity map from a stereo pair and to provide a basic segmentation of images for image retrieval purposes.

The method known as SSD, sum of squared difference, is a popular and moderately robust method for dense stereo matching, but it has several known limitations. Various refinements have been proposed, including actively changing the window shape during processing and multi-resolution analysis. We use the multi-scale tree representation of the image to generate window shapes that are well matched to image objects. We use the structure of the tree to refine our disparity search, creating a disparity tree. This tree is a close representation of the objects in the scene, and is a step towards creating a symbolic representation of objects within a scene. We find that the new method performs particularly well in regions of low texture: operating conditions that cause conventional methods to fail.

A useful representation of an image would be an object tree in which nodes represent objects, or parts of objects, and which includes at least one node that, together with its children, represents each object: a grandmother node. It is shown that scale-trees, obtained from greyscale images, approximate such a tree. It is then shown how they may be modified using other attributes to more closely become object trees. The result is a data structure that pro-vides “handles ” for every element of the image that can be used for manipu-lating the image. This segmentation has potential for object recognition. 1

In images with low texture the performance of conventional dense stereo can be poor. The usual solution to this is to use a large window but this itself can be problematic as the large window can blur important features and hence lead to errors in the disparity estimate. Here it is shown that, not only do connected set morphology operators overcome this problem, they perform best in regions of low texture. A further observation is that, since the operators give a heirarchical decompostion, there is a possibility of not only using these operators to choose a new window, but also to motivate a new matching method. 1

There are several systems that are claimed to generate scale-spaces but how do they compare? This paper examines four significant contenders using both synthetic and real models of occlusion in images. We find that anisotropic diffusion, while effective at removing noise, is not suitable for the estimation of position or scale of simple objects. Linear diffusion may be used for scale selection but has poor performance in occlusion and noise. Of the morphological scale-spaces that we discuss, we find that filtering with flat, connected set structuring elements is not only quicker to compute than diffusion-based scale-spaces, but the results are also perturbed less in the presence of common image distortions such as noise or occlusion.

Systems that attempt to recover the spoken word from image sequences usually require complicated models of the mouth and its motions. Here we describe a new approach based on a fast mathematical morphology transform called the sieve. We form statistics of scale measurements in one and two dimensions and these are used as a feature vector for standard Hidden Markov Models (HMMs).

Decomposition by extrema is put into the context of linear vision systems and scale-space. It is proved that discrete one-dimensional, M- and N-sieves neither introduce new edges as the scale increases nor create new extrema. They share this property with diffusion based filters. They are robust and preserve edges of large scale features.

Decomposition by extrema is put into the context of linear vision systems and scale-space. One dimensional discrete M- and N-sieves neither introduce new edges as the scale increases nor create new extrema. They share this property with diffusion based filters. Furthermore M- and N-sieve algorithms are extremely fast with order complexity n. Used to decompose an image, the resulting granularity is appropriate for pattern recognition.>