One potential problem in real classification applications is that the amount of labeled training data is insufficient since it is usually time-consuming to label data manually. When multiple modalities are available, it is possible to train an initial classifier for each modality using a small amount of labeled data, and then re-train each classifier using unlabeled data associated with the labels generated from the other modalities. This can be achieved by the well-known CO-training algorithm. Assuming that two modalities are available, it only takes the information from the other modality but not that from the self modality into account when choosing data, which usually results in slow convergence of classification accuracy. This may make the CO-training procedure time-consuming. To overcome this, we present a novel modification to the original CO-training algorithm, which is concerned with how new samples are chosen at each iteration to re-train the classifiers in order to improve the convergence of classification accuracy. In our method, the new data is chosen based on the weighted scores which are generated from both modalities instead of only the scores from the other modality as in the original CO-training. We apply both the modified and original CO-training methods on multi-modal person identification task using speech and vision. Experiments on a publicly available database show that our method outperforms the original CO-training by a large margin, in terms of convergence of classification accuracy on a separate testing data set.

The problem of text-dependent speaker verification under noisy conditions is becoming ever more relevant, due to increased usage for authentication in real-world applications. Classical methods for noise reduction such as spectral subtraction and Wiener filtering introduce distortion and do not perform well in this setting. In this work we compare the performance of different noise reduction methods under different noise conditions in terms of speaker verification when the text is known and the system is trained on clean data (mis-matched conditions). We furthermore propose a new approach based on dictionary-based noise reduction and compare it to the baseline methods.

The use of social robots for elderly care is becoming ever more relevant, thus introducing new challenges which need to be solved to achieve acceptable performance. One fundamental task for a social robot is to move to the person of interest in order to start interacting or perform a service. In this paper we address the task of a robot having to navigate to a possibly occluded person, which needs assistance, based only on audio and range information. Our approach is based on forming a heuristic cost function which is based on combining two audio features, and then moving to the optimum position indicated by this cost function after every interaction with the person. The method is compared to a greedy approach in 20 different tasks using a loud speaker playing at approximately 60dB sound pressure level (SPL) to mimic a human speaker, and the proposed method shows superior performance. A second experiment with an increase of 13dB SPL of the loud speaker is conducted and the proposed method is able to handle this.

While miscue detection in general is a well explored research field little attention has so far been paid to miscue detection in dyslexic read speech. This domain differs substantially from the domains that are commonly researched, as for example dyslexic read speech includes frequent regressions and long pauses between words. A system detecting miscues in dyslexic read speech is presented. It includes an ASR component employing a forced-alignment like grammar adjusted for dyslexic input and uses the GOP score and phone duration to accept or reject the read words. Experimental results show that the system detects miscues at a false alarm rate of 5.3% and a miscue detection rate of 40.1%. These results are worse than current state of the art reading tutors perhaps indicating that dyslexic read speech is a challenge to handle.

This paper presents a voice activity detection (VAD) method using the measurement of a posteriori signal-to-noise ratio (SNR) weighted energy.The motivations are manifold: 1) the difference in frame-to-frame energy provides a great discrimination for speech signals, 2) speech segments, besides their characteristics, are accounted also on their reliability e.g.measured by SNR, 3) the a posteriori SNR for noise-only segments will theoretically equal to 0 dB, being ideal for VAD, and 4) both energy and a posteriori SNR are easy to estimate, resulting in a low complexity.The method is experimentally shown to be superior to a number of referenced methods and standards.

This paper presents a variable frame rate (VFR) analysis method that uses an a posteriori signal-to-noise ratio (SNR) weighted energy distance for frame selection. The novelty of the method consists in the use of energy distance (instead of cepstral distance) to make it computationally efficient and the use of SNR weighting to emphasize the reliable regions in speech signals. The VFR method is applied to speech recognition in two scenarios. First, it is used for improving speech recognition performance in noisy environments. Secondly, the method is used for source coding in distributed speech recognition where the target bit rate is met by adjusting the frame rate, yielding a scalable coding scheme. Prior to recognition in the server, frames are repeated so that the original frame rate is restored. Very encouraging results are obtained for both noise robustness and source coding. Index Terms: speech recognition, speech analysis, variable frame rate, noise robustness, source coding

The nonlocal means (NL-means) algorithm recently proposed for image denoising has proved highly effective for removing additive noise while to a large extent maintaining image details. The algorithm performs denoising by averaging each pixel with other pixels that have similar characteristics in the image. This letter considers the real and imaginary parts of complex speech spectrogram each as a separate image and presents a modified NL-means algorithm to them for denoising to improve the noise robustness of speech recognition. Recognition results on a noisy speech database show that the proposed method is superior to classical methods such as spectral subtraction.

In this paper, the temporal correlation of speech is exploited in front-end feature extraction, client-based error recovery, and server-based error concealment (EC) for distributed speech recognition. First, the paper investigates a half frame rate (HFR) front-end that uses double frame shifting at the client side. At the server side, each HFR feature vector is duplicated to construct a full frame rate (FFR) feature sequence. This HFR front-end gives comparable performance to the FFR front-end but contains only half the FFR features. Second, different arrangements of the other half of the FFR features creates a set of error recovery techniques encompassing multiple description coding and interleaving schemes where interleaving has the advantage of not introducing a delay when there are no transmission errors. Third, a subvector-based EC technique is presented where error detection and concealment is conducted at the subvector level as opposed to conventional techniques where an entire vector is replaced even though only a single bit error occurs. The subvector EC is further combined with weighted Viterbi decoding. Encouraging recognition results are observed for the proposed techniques. Lastly, to understand the effects of applying various EC techniques, this paper introduces three approaches consisting of speech feature, dynamic programming distance, and hidden Markov model state duration comparison.

Condition-dependent training strategy divides a training database into a number of clusters, each corresponding to a noise condition and subsequently trains a hidden Markov model (HMM) set for each cluster. This paper investigates and compares a number of condition-dependent training strategies in order to achieve a better understanding of the effects on automatic speech recogntion (ASR) performance as caused by a splitting of the training databases. Also, the relationship between mismatches in signal-to-noise ratio (SNR) is analyzed. The results show that a splitting of the training material in terms of both noise type and SNR value is advantageous compared to previously used methods, and that training of only a limited number of HMM sets is sufficient for each noise type for robustly handling of SNR mismatches. This leads to the introduction of an SNR and noise classification-based training strategy (SNT-SNC). Better ASR performance is obtained on test material containing data from known noise types as compared to either multicondition training or noise-type dependent training strategies. The computational complexity of the SNT-SNC framework is kept low by choosing only one HMM set for recognition. The HMM set is chosen on the basis of results from noise classification and SNR value estimations. However, compared to other strategies, the SNT-SNC framework shows lower performance for unknown noise types. This problem is partly overcome by introducing a number of model and feature domain techniques. Experiments using both artificially corrupted and real-world noisy speech databases are conducted and demonstrate the effectiveness of these methods.

Compared to multi-condition training (MTR), condition-dependent training generates multiple acoustic hidden Markov model sets each identified by a noisy environment and is known to perform substantially better for known noise types (included in training) while worse for unknown (untrained) noise types. This paper attempts to bridge the performance gap between known and unknown noise types by introducing a Minimum Mean-Square Error (MMSE) noise-type based compensation algorithm. On the basis of a modified Vector Taylor Series and the measurement of feature reliability as well as noise similarity, the MMSE estimation adapts the test features corrupted by the unknown noise type to the corresponding features corrupted by the known noise type. This method significantly improves the recognition performance for unknown noise types while maintaining the good performance for known noise types. Furthermore, in order to benefit directly from MTR, a model interpolation strategy is investigated which combines the MTR and the condition-dependent model sets. Both good performance and low computational cost are achieved by only interpolating the mixtures of each condition-dependent model state with the least weighted mixture in the corresponding MTR model state. The overall system gives promising results.

Robustness against transmission errors is one of the primary barriers to the widespread application of automatic speech recognition (ASR) in mobile communications. We have previously proposed a subvector based error concealment (EC) method that conducts error detection and mitigation in the feature-domain at the subvector level. This paper presents a weighted Viterbi decoding (WVD) algorithm that works in the model domain for counteracting unreliable features generated by the subvector based EC. The reliability of each feature is estimated during the process of subvector based EC and is used by the WVD for modifying the observation probability of the feature. Recognition experiments are conducted on the Aurora 2 database corrupted by GSM error pattern EP3. Combining the WVD and the subvector EC achieves 70 % and 24% performance improvement as compared to the ETSI-DSR standard and the subvector based EC, respectively. Index Terms: distributed speech recognition, error concealment, split vector quantization, weighted Viterbi

The introduction of ubiquitous computing and networking has fostered automatic speech recognition (ASR) systems of a distributed nature. The major challenge in deploying ubiquitous ASR is that the operating environments may change rapidly leaving the ASR system very vulnerable. This paper deals with the concept of making ASR systems context-aware with the aim of improving robustness against varying conditions such as dynamic network constraints and environmental noise. To fully benefit from a variety of networks with different characteristics, a number of distributed speech recognition (DSR) schemes are presented each of which is applicable to a specific network context. To increase ASR system robustness in varying environmental noise context, a multiple-model framework for noise-robust ASR is presented where multiple HMM model sets are trained, one for each noise type and each specific signal-to-noise ratio (SNR) that characterise the noise context. Experimental results show that the performance of ASR is largely improved by exploiting the context information. 1.

This paper presents a multiple-model framework for noise-robust speech recognition. In this framework, multiple HMM model sets are trained- each identified by a noise type and a specific Signal-to-Noise Ratio (SNR) value. This, however, does not increase the computational complexity of the recognition process since only one model set is selected according to the noise classification and SNR estimation. The optimal number of model sets is first identified on the basis of the Aurora 2 database. With only three model sets for each noise type, the framework shows superior performance to Multi-style TRaining (MTR) when testing on known noise types but lower performance on unknown noise types. To overcome this drawback, a modified Jacobian method is proposed to adapt the selected HMM models to the test environment. Furthermore, given the fact that MTR often gives relatively stable performance for unknown noise types, a combined technique is applied in which interpolation between the MTR and the adapted models is performed. This combined technique gives more than 24 % performance improvement as compared to MTR. 1.

In this paper a half frame-rate (HFR) front-end is investigated for distributed speech recognition (DSR). The work is inspired from the need for low bit-rate and is justified by the redundancies known to exist in full frame-rate (FFR) features. At the client-side in the DSR architecture, implementation of the HFR is carried out by using double frame shifting as compared to the FFR resulting in the achievement of half the bit rate. At the server-side, each HFR feature vector is repeated once to construct the FFR features and no changes are therefore required in the recognition back-end. It is experimentally justified that the performance achieved by HFR is comparable to FFR and that repetition of each HFR feature vector is critical for the HFR front-end to maintain the performance. Motivated by the effectiveness of HFR, a number of additional FFR-based DSR schemes are further presented. Finally, this paper introduces an adaptive multi-frame-rate scheme in which the DSR system adapts to the characteristics of the transmission channel by switching between HFR and the FFR-based schemes. This multi-frame-rate scheme is found to be superior to the basic FFR

Conventional error concealment (EC) algorithms for distributed speech recognition (DSR) share a common characteristic namely the fact of conducting EC at the vector (or frame) level. This strategy, however, fails to effectively exploit the error-free fraction left within erroneous vectors where a substantial number of subvectors often are error-free. This paper proposes a novel EC approach for DSR encoded by split vector quantization (SVQ) where the detected erroneous vectors are submitted to a further analysis at the subvector level. Specifically, a data consistency test is applied to each erroneous vector to identify inconsistent subvectors. Only inconsistent subvectors are replaced by their nearest neighbouring consistent subvectors whereas consistent subvectors are kept untouched. Experimental results demonstrate that the proposed algorithm in terms of recognition accuracy is superior to conventional EC methods having almost the same complexity and resource requirement.

This paper presents the basic rationale behind the development and testing of a multimodal communication aid especially designed for people suffering from global aphasia, and thus having severe expressive difficulties. The principle of the aid is to trigger patient associations by presenting various multimodal representations of communicative expressions. The aid can in this way be seen as a conceptual continuation of previous research within the field of communication aids based on uni-modal (pictorial) representations of communicative expressions. As patients suffering from global aphasia seldom have identical symptoms, the focus of this paper is placed on the development of a highly dedicated communication aid adaptive to the individual patients’ needs. The paper investigates whether or not such a highly dedicated communication aid based on multimodal representations of communicative expressions can be used to support patients with global aphasia in communicating by means of short sentences with their surroundings. Only a limited evaluation is carried out, and as such no statistically significant results are obtained. The tests however indicate that the aid is capable of supporting a global aphasia patient in participating in conversations based on short sentences which otherwise would be impossible without the use of a communication aid.

Mobile communication presents a number of challenges to speech technology such as the limited resources available in the terminals in addition to the bandwidth constraints and the errors occurring in transmissions over mobile networks. These challenges need to be solved before automatic speech recognition (ASR) is ready for widespread use in the context of personal communication environments. This paper gives an overview of the problems inherent in the recently developed network based ASR with an emphasis on the robustness issues that are highly influenced by network degradations. The paper further presents a number of transmission error protection and concealment schemes that are evaluated in a number of ASR experiments encompassing a range of typical real-environment transmission errors. 1.

In standard Spectral Subtraction (SS), Half-Wave Rectification SS (HWR-SS) is normally applied to avoid negative values in the Power Spectral Density (PSD) that occur mainly due to inaccurate noise estimation caused by a Voice Activity Detector (VAD). In this paper analyses show that, given accurate noise estimation, the phase relationship between speech and noise becomes the dominant cause of the negative values. FullWave Rectification based SS (FWR-SS) combined with Instantaneous Noise Estimation (INE) is therefore proposed to be applied instead of VAD based HWR-SS as it is better capable of maintaining the speech information in those negative values. It is also shown in the paper that FWR-SS provides optimum orthogonality between the estimated noise and speech signals. The INE method proposed in this paper is Likelihood Controlled Instantaneous Noise Estimation (LCINE), which combines long-term statistical characteristics of noise resulting from a VAD with a method of short-term INE. The combination of FWR-SS and LCINE is computationally efficient and shows a 51% error rate reduction on the Aurora 2 database in comparison to the basic Aurora front-end provided by ETSI [1].

This paper presents research on two aspects of distributed speech recognition (DSR) in the presence of channel transmission errors in wireless network environments. The first is on experiments with a frame-based channel error protection scheme, where in previous research we reported results from experiments using randomly distributed bit-errors. This paper presents results from experiments using three additional, more realistic error distributions: burst-like packet loss, GSM error patterns and UMTS statistics. The second is on exploiting the knowledge about channel transmission errors for the purpose of optimising the Out-of-Vocabulary (OOV) detection. Transmission errors influence the acoustic likelihood, and therefore affect the optimal threshold setting for discrimination between In-Vocabulary (IV) words and OOV words. An OOV-detection method is proposed in which the estimated Frame-Error-Rate (FER) is used to adjust the discrimination threshold. Results from experiments are reported over a range of transmission errors.

A multi-stream speech recogniser is based on the combination of multiple feature streams each containing complementary information. In the past, multi-stream research has typically focused on systems that use a single feature extraction method. This heritage from conventional speech recognisers is an unnecessary restriction and both psychoacoustic and phonetic knowledge strongly motivate the use of heterogeneous features. In this paper we investigate how heterogeneous processing can be used in two different multi-stream configurations: first, a system where each stream handles a different frequency region of the speech (a multi-band recogniser) and, second a multi-stream recogniser where each stream handles the full frequency region. For each type of system we compare the performance using both homogeneous and heterogeneous processing. We demonstrate that the use of heterogeneous information significantly improves the clean speech recognition performance motivating us to continue exploring more specifically designed stream processing.

An important aspect of noise robustness of automatic speech recognisers (ASR) is the proper handling of non-speech acoustic events. The present paper describes further improvements of an already existing reference recogniser towards achieving such kind of robustness. The reference recogniser applied is the COST 249 SpeechDat reference recogniser, which is a fully automatic, language-independent training procedure for building a phonetic recogniser (http://www.telenor.no/fou/prosjekter/taletek/refrec). The reference recogniser relies on the HTK toolkit and a SpeechDat (II) compatible database, and is designed to serve as a reference system in multilingual speech recognition research. The paper describes version 0.96 of the reference recogniser which take into account labelled non-speech acoustic events during training and provides robustness against these during testing. Results are presented on small and medium vocabulary recognition for six languages.

In automatic speech recognition (ASR) systems immunity to additive noise may either be applied at the preprocessing stage or at the pattern matching stage. The Feature Selective Modeling (FSM) approach suggested in this paper is applied in the pattern matching stage, but in contrast to most existing methods, it is optimized on a model basis such that noise robust and phonetically descriptive parameters of a particular model can be set in focus. For sonorant sounds this is done by marking the lowest n mean values of each HMM density function as being sensitive to noise in a log filterbank representation. The noise robustness is obtained by de-emphasizing the marked feature dimensions. Two different methods for de-emphasizing mean value masking and dimensional reduction are presented and experimentally compared to the PMC-algorithm [2].

Successful and flexible establishment of a spoken dialogue system requires the integration of dialogue design, dialogue management and spoken language processing into a common system environment. The research and development reported in this paper give details from the establishment of such an environment and its necessary development tools and devices as well as preliminary results from testing the integral continuous speech recogniser component on an artificial CADapplication. The application is characterised by a vocabulary containing names of geometrical objects, words by which these objects may be moved, changed in size, coloured etc, such that speech activated manipulation of the objects may be observed on a display. The interactive functionality is guided by reproductive speech output telling the user if, for instance, he/she is asking the system to perform illegal actions as seen from the pragmatic/semantic component of the dialogue system. The continuous speech recogniser pa...