Vertigo is a type of dizziness characterised by the subjective feeling of movement despite being stationary. One in four individuals in the community experience symptoms of dizziness at any given time, and it can be challenging for clinicians to diagnose the underlying cause. When dizziness is the result of a malfunction in the inner-ear, the eyes flicker and this is called nystagmus. In this article we describe the first use of Deep Neural Network architectures applied to detecting nystagmus. The data used in these experiments was gathered during a clinical investigation of a novel medical device for recording head and eye movements. We describe methods for training networks using very limited amounts of training data, with an average of 11 mins of nystagmus across four subjects, and less than 24 hours of data in total, per subject. Our methods work by replicating and modifying existing samples to generate new data. In a cross-fold validation experiment, we achieve an average F1 score of 0.59 (SD = 0.24) across all four folds, showing that the methods employed are capable of identifying periods of nystagmus with a modest degree of accuracy. Notably, we were also able to identify periods of pathological nystagmus produced by a patient during an acute attack of Ménière's Disease, despite training the network on nystagmus that was induced by different means.

Recent improvements in tracking and feature extraction mean that speaker-dependent lip-reading of continuous speech using a medium size vocabulary (around 1000 words) is realistic. However, the recognition of previously unseen speakers has been found to be a very challenging task, because of the large variation in lip-shapes across speakers and the lack of large, tracked databases of visual features, which are very expensive to produce. By adapting a technique that is established in speech recognition but has not previously been used in lip-reading, we show that error-rates for speaker-independent lip-reading can be very significantly reduced. Furthermore, we show that error-rates can be even further reduced by the additional use of Deep Neural Networks (DNN). We also find that there is no need to map phonemes to visemes for context-dependent visual speech transcription.

Ball tracking is a key technology in processing and analyzing a ball game. Because of the complexity of visual scenes, a large number of objects are often selected as candidates for the ball, leading to incorrect identification, and conversely, the true position of the ball may sometimes be missed because of occlusion and blur, which can both be frequent and severe. Several tennis ball tracking algorithms have been reported in literature. In this paper, we propose a two-layered data association method to improve the robustness of tennis ball tracking. At the local layer, a shift token transfer method is proposed, based on shift window processing, to generate a set of short trajectories or “ trajectorylets .” At the global layer, a unique ball trajectory is obtained by applying a dynamic programming based splice method to a directed acyclic graph consisting of trajectorylets. We evaluated our approach on tennis matches from the Australian Open and the U.S. Open, and the results obtained show that our approach outperforms current state-of-the-art approach.

This paper is concerned with automatic speech recognition (ASR) for accented speech. Given a small amount of speech from a new speaker, is it better to apply speaker adaptation to the baseline, or to use accent identification (AID) to identify the speaker’s accent and select an accent-dependent acoustic model? Three accent-based model selection methods are investigated: using the ‘true’ accent model, and unsupervised model selection using i-Vector and phonotactic-based AID. All three methods outperform the unadapted baseline. Most significantly, AID-based model selection using 43s of speech performs better than unsupervised speaker adaptation, even if the latter uses five times more adaptation data. Combining unsupervised AIDbased model selection and speaker adaptation gives an average relative reduction in ASR error rate of up to 47%.

Ball-tracking is a key technology in processing and analyzing a ball game. Because of the complexity of visual scenes, a large number of objects are usually selected as candidates for the ball, leading to incorrect identification, and conversely, the true position of the ball may sometimes be missed. In this paper, we propose a two layered data association method to improve the robustness of ball-tracking. At a local layer, we use a sliding window based Token Transfer method to generate a set of sub-trajectory candidates. At a global layer, a single ball trajectory is obtained by applying a dynamic programming based splice method to a graph consisting of the sub-trajectory candidates. We evaluated our approach on tennis matches from the Australian Open and the U.S. Open, and the results obtained show that our approach outperforms the state-of-art approach by around 30%.

This paper presents new approaches to improve the detection of two key audio events in a sport game (tennis) using contextual information. When analysing a tennis match using only audio information, the sound of the ball being struck and the occurrence of a line judge's shout can be obscured by players' grunts or shouts. Furthermore, if models of these two important events are trained from labelled training-data, there is often considerable audio mis-match with the test-data, which means that detection performance can be very poor. To handle this problem, we regard the players' grunts as useful contextual information that indicates the position of the events of interest. We show how to use an unsupervised learning method to build an improved model of the ball-hit event using grunt information. We can then use high-level information to distinguish grunts from line-judge shouts. This approach gives simultaneous improvements in detection of both ball-hits and line judge shouts, and is portable between different matches, unlike approaches based on the use of manually labelled training-data.

Automatic visual language identification (VLID) is the technology of using information derived from the visual appearance and movement of the speech articulators to identify the language being spoken, without the use of any audio information. This technique for language identification (LID) is useful in situations in which conventional audio processing is ineffective (very noisy environments), or impossible (no audio signal is available). Research in this field is also beneficial in the related field of automatic lip-reading. This paper introduces several methods for visual language identification (VLID). They are based upon audio LID techniques, which exploit language phonology and phonotactics to discriminate languages. We show that VLID is possible in a speaker-dependent mode by discriminating different languages spoken by an individual, and we then extend the technique to speaker-independent operation, taking pains to ensure that discrimination is not due to artifacts, either visual (e.g., skin-tone) or audio (e.g., rate of speaking). Although the low accuracy of visual speech recognition currently limits the performance of VLID, we can obtain an error-rate of <; 10% in discriminating between Arabic and English on 19 speakers and using about 30 s of visual speech.

We describe a robust, unsupervised method of automatic gender identification from speech. We first design a baseline gender classifier based on MFCC features, and add a second classifier that uses context-dependent but text-independent pitch features. The results of these classifiers are then examined for disagree- ments in gender classification. Any disagreements are resolved by the use of a novel pitch-shifting mechanism applied to the ut- terances. We show how the acoustic context classifier provides very good gender identification results, and how these are fur- ther enhanced by the pitch-shifting process. Furthermore this enhancement is preserved across a set of different corpora.

We describe a novel approach to inferring the scoring rules of a tennis game by analysing the chair umpire’s speech. In a tennis match, the chair umpire, amongst other tasks, announces the scores. Hence his or her speech is the key resource for inferring the scoring rules of tennis, a task that can be accomplished by correlating the events on the court with these score announcements. In this work, the learning procedure consists of two steps: speech recognition followed by rule inference. For speech recognition, we use a two coupled language models one for words and one for scores. The first makes make use of the internal structure of a score, the second, the dependency of a score on the previous score. For rule inference, we utilize a multigram model to segment the recognised score streams into variable-length score sequences, each of them corresponding to a game in a tennis match. The approach is applied to four complete tennis matches, and shows both enhanced recognition performance, and a promising approach to inferring the scoring rules of the game. Speech recognition, rule inference, multigram model

We describe a novel framework for inferring the low-level structure of a sports game (tennis) using only the information available on the audio track of a video recording of the game. Our goal is to segment the games into a sequence of points, the natural unit for describing a tennis match. The framework is hierarchical, consisting of, at the lowest level, identification of audio events, followed by “match” (i.e., semantic) events and at the highest level, game points. Different techniques that are appropriate to the characteristics of each of these events are used to detect them and these techniques are coupled in a probabilistic framework. The techniques consist of Gaussian mixture models and a hierarchical language model to detect sequences of audio events, a maximum entropy Markov model to infer “match” events from these audio events and multigrams to infer the segmentation of a sequence of match events into sequences of points in a a tennis game. Our results are promising, giving an F-score for the final detection of points of >; 0.7.

We investigate the automatic labelling of “events” from an audio recording of a sports game. We describe a technique that utilises a hierarchy of language models, which are a low-level model of acoustic observations and a high-level model of audio events that occur during a game: these models are integrated using a maximum entropy approach. Our models of the audio events also utilise duration and voicing information as well as spectral content, and we show that further discrimination between events is possible using these features. Results on different tennis games show that the use of these techniques is better than using an approach that does not use modelling of dependencies between frames and events or extra information in the form of duration and voicing.

We describe experiments in visual-only language identification (VLID), in which only lip shape, appearance and motion are used to determine the language of a spoken utterance. In previous work, we had shown that this is possible in speaker-dependent mode, i.e. identifying the language spoken by a multi-lingual speaker. Here, by appropriately modifying techniques that have been successful in audio language identification, we extend the work to discriminating two languages in speaker-independent mode. Our results indicate that even with viseme accuracy as low as about 34%, reasonable discrimination can be obtained. A simulation of degraded accuracy viseme recognition performance indicates that high VLID accuracy should be achievable with viseme recognition errors of the order of 50%.

This paper describes how the detection of key audio events in a sports game (tennis) can be enhanced by the use of high-level information. High-level features are able to provide useful constraints on the detection procedure, and thus to improve detection performance. We define two types of event based information: event dependency and inter-event timing. These respectively characterize the identity of the next event and the time at which the next event will occur. Probabilistic models of high-level constraints are developed, and then integrated into our event detection framework. We test this approach on audio tracks extracted from two different tennis games. The results show that significant improvements in both accuracy and computational efficiency are obtained when applying high-level information.

We describe experiments in visual-only language identification, in which only lip-shape and lip-motion are used to determine the language of a spoken utterance. We focus on the task of discriminating between two or three languages spoken by the same speaker, and we have recorded a suitable database for these experiments. We use a standard audio language identification approach in which the feature vectors are tokenized and then a language model for each language is estimated over a stream of tokens. Although rate of speaking appeared to affect our results, it was found that different languages spoken at rather similar speeds were as well discriminated as a single language spoken at three extreme speeds, indicating that there is a language effect present in our results.

In previous work, we described how learning the pattern of recognition errors made by an individual using a certain ASR system leads to increased recognition accuracy compared with a standard MLLR adaptation approach. This was the case for low-intelligibility speakers with dysarthric speech, but no improvement was observed for normal speakers. In this paper, we describe an alternative method for obtaining the training data for confusion-matrix estimation for normal speakers which is more effective than our previous technique. We also address the issue of data sparsity in estimation of confusion-matrices by using non-negative matrix factorization (NMF) to discover structure within them. The confusion-matrix estimates made using these techniques are integrated into the ASR process using a technique termed as metamodels, and the results presented here show statistically significant gains in word recognition accuracy when applied to normal speech.

In this paper, we describe the design of an ASR system that is based on identifying and extracting formulaic phrases from a corpus and then, rather than building statistical models of them, performing example-based recognition of these phrases. We describe a method for combining formulaic phrases into a bigram language model that results in a 13% decrease in WER on a monophone HMM recogniser over the baseline. We show that using this model with phrase templates in the example-based recogniser gives a significant improvement in WER compared to word templates, but performance still falls short of the HMM recogniser. We also describe an LDA decision tree classifier that reduces the search space of the DTW decoder by 40% while at the same time decreasing WER.

Standard speaker adaptation algorithms perform poorly on dysarthric speech because of the limited phonemic repertoire of dysarthric speakers. In a previous paper, we proposed the use of metamodels to correct dysarthric speech. Here, we report on an improved technique that makes use of a cascade of Weighted Finite-State Transducers (WFSTs) at the confusion-matrix, word and language levels. This approach outperforms both standard MLLR and metamodels.

Dysarthria is a motor speech disorder characterized by weakness, paralysis, or poor coordination of the muscles responsible for speech. Although automatic speech recognition (ASR) systems have been developed for disordered speech, factors such as low intelligibility and limited vocabulary decrease speech recognition accuracy. In this paper, we introduce a technique that can increase recognition accuracy in speakers with low intelligibility by incorporating information from an estimate of the speaker's phoneme confusion matrix. The technique performs much better than standard speaker adaptation when the number of sentences available from a speaker for confusion matrix estimation or adaptation is low, and has similar performance for larger numbers of sentences.

There are some speech understanding applications in which training transcriptions are unavailable, and hence the vocabulary is unknown, but the task is to recognise key words and phrases within an utterance rather than to attempt a complete, accurate transcription. An example of such a task is call-routing, when transcriptions of training utterances (which are very expensive to produce) are unavailable. In such cases, phoneme rather than word recognition is appropriate. However, phoneme recognition of spontaneous speech spoken by a large multi-accent population over telephone connections is very inaccurate. To improve accuracy, we describe a technique in which we segment the waveform into subword-like units and use clustering and an iteratively refined language model to correct the errors in the recognised phonemes. The method was shown to work well on telephone quality spontaneous speech, raising the phoneme accuracy from 28.1% after the first iteration to 47.3% after three iterations.

Our goal is to extract information from a telephone call in order to route the call to one of a number of destinations. We assume that we do not know "a priori" the vocabulary used in the application and so we use phonetic recognition followed by identification of salient phone sequences. In previous work, we showed that using a separate language model during recognition for each route gave improved performance over using a single model. However, this technique decodes each utterance in terms of the salient sequences of each call route, which leads to insertion and substitution errors that degrade performance. In this paper, we introduce the use of mixture language models for speech recognition in the context of call route classification. The benefit of technique can has the efficiency of multiple language models to get accurate recognition on salient phoneme sequences; on the other hand, it can give help in classification, even if the size of some call routes have just 50~60 utterances. It avoids building HMMs for some salient phoneme sequences to decide whether it is correct of occurring in the utterance.

Call-routing is a technology that attempts to route automatically a telephone query from a customer to one of a number of destinations. In vector-based call-routing, a query is represented in a high-dimensional vector space whose axes correspond to words, or sequences of words, that appear in the vocabulary used by callers. In this paper, we examine three different discriminative techniques applied to call-routing. Although some of these techniques give very substantial reductions in error-rate on the training-set, performance on the test-set is disappointing, the most likely reason being a lack of generalisation. Using examples of mis-classified calls, we speculate on why this might occur and propose an improved approach.

In previous work, we experimented with different techniques of vector-based call routing, using the transcriptions of the queries to compare algorithms. In this paper, we base the routing decisions on the recogniser output rather than transcriptions and examine the use of confidence measures (CMs) to combat the problems caused by the "noise" in the recogniser output. CMs are derived for both the words output from the recogniser and for the routings themselves and are used to investigate improving both routing accuracy and routing confidence. Results are given for a 35 route retail store enquiry-point task. They suggest that although routing error is controlled by the recogniser error-rate, confidence in routing decisions can be improved using these techniques.

Call-routing is now an established technology to automate customers' telephone queries. However, transcribing calls for training purposes for a particular application requires considerable human effort, and it would be preferable for the system to learn routes without transcriptions being provided. This paper introduces a technique for fully automatic routing. It is based on firstly identifying salient acoustic morphemes in a phonetic decoding of the input speech, followed by Linear Discriminant Analysis (LDA) to improve classification. Experimental results on an 18 route retail store enquiry point task using this technique are compared with results obtained using word-level transcriptions.

In concatenative text-to-speech (TTS) synthesis systems unit selection aims to reduce the number of concatenation points in the synthesized speech and make concatenation joins as smooth as possible. This research considers synthesis of completely new utterances from non-uniform units, whereby the most appropriate units, according to acoustic and phonetic criteria, are selected from a myriad of similar speech database candidates. A Viterbi-style algorithm dynamically selects the most suitable database units from a large speech database by considering concatenation and target costs. Concatenation costs are derived from mel filter bank amplitudes, whereas target costs are considered in terms of the phonemic and phonetic properties of required units. Within subjects and between subjects ANOVA [9] evaluation of listeners' scores showed that the TTS system with this method of unit selection was preferred in 52% of test sentences.

This paper proposes an integrated speech front-end for both speech recognition and speech reconstruction applications. Speech is first decomposed into a set of frequency bands by an auditory model. The output of this is then used to extract both robust pitch estimates and MFCC vectors. Initial tests used a 128 channel auditory model, but results show that this can be reduced significantly to between 23 and 32 channels. A detailed analysis of the pitch classification accuracy and the RMS pitch error shows the system to be more robust than both comb function and LPC-based pitch extraction. Speech recognition results show that the auditory-based cepstral coefficients give very similar performance to conventional MFCCs. Spectrograms and informal listening tests also reveal that speech reconstructed from the auditory-based cepstral coefficients and pitch has similar quality to that reconstructed from conventional MFCCs and pitch. 1.

The multimodal nature of speech is often ignored in human-computer interaction, but lip deformations and other body motion, such as those of the head, convey additional information. We integrate speech cues from many sources and this improves intelligibility, especially when the acoustic signal is degraded. The paper shows how this additional, often complementary, visual speech information can be used for speech recognition. Three methods for parameterizing lip image sequences for recognition using hidden Markov models are compared. Two of these are top-down approaches that fit a model of the inner and outer lip contours and derive lipreading features from a principal component analysis of shape or shape and appearance, respectively. The third, bottom-up, method uses a nonlinear scale-space analysis to form features directly from the pixel intensity. All methods are compared on a multitalker visual speech recognition task of isolated letters.

Systems that attempt to recover the spoken word from image sequences usually require complicated models of the mouth and its motions. Here we describe a new approach based on a fast mathematical morphology transform called the sieve. We form statistics of scale measurements in one and two dimensions and these are used as a feature vector for standard Hidden Markov Models (HMMs).

There is increasing interest in systems which attempt to automate a task or a transaction using speech input and output. To function effectively with imperfect speech recognition, such systems require an estimate of which words in the output from the recogniser are likely to be correct and which can probably be disregarded as incorrect, i.e. a confidence-measure for each decoded word. We define a measure for evaluating the effectiveness of a post-classifier which estimates confidence-measures, and describe the development of a post-classifier for words decoded from the SWITCHBOARD database, which uses statistics derived from a Viterbi decoder. Without any grouping of the decoded word-classes, the post-classifier increased the probability of deciding whether a decoded word was correct or incorrect by 32%. When grouping was used, longer words showed an improvement of 65%.

A general approach to speaker adaptation in speech recognition is described, in which speaker differences are treated as arising from a parameterized transformation. Given some unlabeled data from a particular speaker, a process is described which maximizes the likelihood of this data by estimating the transformation parameters at the same time as refining estimates of the labels. The technique is illustrated using isolated vowel spectra and phonetically motivated linear spectrum transformations and is shown to give significantly better performance than nonadaptive classification.>

The authors present two simple tests for deciding whether the difference in error rates between two algorithms tested on the same data set is statistically significant. The first (McNemar's test) requires the errors made by an algorithm to be independent events and is found to be most appropriate for isolated-word algorithms. The second (a matched-pairs test) can be used even when errors are not independent events and is more appropriate for connected speech.>