End-to-end (E2E) Spoken Language Understanding (SLU) systems infer structured information directly from the speech signal using a single model. Due to the success of virtual assistants and the increasing demand for speech interfaces, these architectures are being actively researched for their potential to improve system performance by exploiting acoustic information and avoiding the cascading errors of traditional architectures. However, these systems require large amounts of specific, well-labelled speech data for training, which is expensive to obtain even in English, where the number of public audio datasets for SLU is limited. In this paper, we release the FalAI dataset, the largest public SLU dataset in terms of hours (250 hours), recordings (260,000) and participants (over 10,000), which is also the first SLU dataset in Galician and the first to be obtained in a low-resource scenario. Furthermore, we present new measures of complexity for the text corpora, the strategies followed for the design, collection and validation of the dataset, and we define splits for noisy audio, hesitant audio and audio where the sentence has changed but the structured information is preserved. These novel splits provide a unique resource for testing SLU systems in challenging, real-world scenarios.

This paper addresses the challenge of integrating low-resource languages into multilingual automatic speech recognition (ASR) systems. We introduce a novel application of weighted cross-entropy, typically used for unbalanced datasets, to facilitate the integration of low-resource languages into pre-trained multilingual ASR models within the context of continual multilingual learning. We fine-tune the Whisper multilingual ASR model on five high-resource languages and one low-resource language, employing language-weighted dynamic cross-entropy and data augmentation. The results show a remarkable 6.69% word error rate (WER) reduction for the low-resource language compared to the fine-tuned model without applying our approach, and a 48.86% WER reduction compared to the original Whisper model. In addition, our approach yields an average WER reduction of 3.29% across the six languages, showing no degradation for the high-resource languages.

There is an urgent need for new methods that improve the management and treatment of Major Depressive Disorder (MDD). Speech has long been regarded as a promising digital marker in this regard, with many works highlighting that speech changes associated with MDD can be captured through machine learning models. Typically, findings are based on cross-sectional data, with little work exploring the advantages of personalization in building more robust and reliable models. This work assesses the strengths of different combinations of speech representations and machine learning models, in personalized and generalized settings in a two-class depression severity classification paradigm. Key results on a longitudinal dataset highlight the benefits of personalization. Our strongest performing model set-up utilized self-supervised learning features and convolutional neural network (CNN) and long short-term memory (LSTM) back-end.

Depression is a common mental disorder that is usually addressed by outpatient treatments that favour patients’ inclusion in the society. This raises the need for tools to remotely monitor the emotional state of the patients, which can be carried out via telephone or the Internet using speech processing approaches. However, these strategies lead to privacy concerns caused by the transmission of the patients’ speech and its subsequent storage in servers. The use of speech de‐identification to protect the privacy of these patients seems straightforward, but the influence of this procedure in the manifestation of the disease in the patients’ speech has not been addressed yet. Hence, this study evaluates the performance of an automatic depression level estimation system when dealing with original and de‐identified speech, in order to analyse the influence of the de‐identification procedure in the detection of depression. Two de‐identification approaches based on voice transformation via frequency warping and amplitude scaling are assessed, which can be applied to any speaker without additional training. Experiments carried out in the framework of the audio/visual emotion challenge 2014 show that the proposed de‐identification approaches achieve promising de‐identification results at the expense of a slight degradation of depression detection.

Cross-lingual query-by-example spoken term detection (QbE STD) has caught the attention of speech researchers, as it makes it possible to develop systems for low-resource languages, in which the available amount of labelled data makes the training of automatic speech recognition approaches prohibitive. The use of phonetic posteriorgrams for speech representation combined with dynamic time warping search is a widely used approach for this task, but little attention has been focused in the suitability of a set of phonetic units to represent speech information spoken in a different language. This paper proposes a technique for estimating the relevance of phonetic units aiming at selecting the most suitable ones for a given target language. Experiments in a Spanish database using phoneme posteriorgrams in Czech, English, Hungarian and Russian proved the validity of the proposed method, as QbE STD performance was enhanced by reducing the set of phonetic units.

Best language recognition performance is commonly obtained by fusing the scores of several heterogeneous systems. Regardless the fusion approach, it is assumed that different systems may contribute complementary information, either because they are developed on different datasets, or because they use different features or different modeling approaches. Most authors apply fusion as a final resource for improving performance based on an existing set of systems. Though relative performance gains decrease as larger sets of systems are considered, best performance is usually attained by fusing all the available systems, which may lead to high computational costs. In this paper, we aim to discover which technologies combine the best through fusion and to analyse the factors (data, features, modeling methodologies, etc.) that may explain such a good performance. Results are presented and discussed for a number of systems provided by the participating sites and the organizing team of the Albayzin 2010 Language Recognition Evaluation. We hope the conclusions of this work help research groups make better decisions in developing language recognition technology.

In the last few years the number of systems and devices that use voice based interaction has grown significantly. For a continued use of these systems the interface must be reliable and pleasant in order to provide an optimal user experience. However there are currently very few studies that try to evaluate how good is a voice when the application is a speech based interface. In this paper we present a new automatic voice pleasantness classification system based on prosodic and acoustic patterns of voice preference. Our study is based on a multi-language database composed by female voices. In the objective performance evaluation the system achieved a 7.3% error rate

The recent developments on Hidden Markov Models (HMM) based speech synthesis showed that this is a promising technology fully capable of competing with other established techniques. However some issues still lack a solution. Several authors report an over-smoothing phenomenon on both time and frequencies which decreases naturalness and sometimes intelligibility. In this work we present a new vowel intelligibility enhancement algorithm that uses a discrete Kalman filter (DKF) for tracking frame based parameters. The inter-frame correlations are modelled by an autoregressive structure which provides an underlying time frame dependency and can improve time-frequency resolution. The system's performance has been evaluated using objective and subjective tests and the proposed methodology has led to improved results.

Reliable speaker segmentation is critical in many applications in the speech processing domain. In this paper, we extend our earlier formulation for false alarm reduction in a typical state-of-art speaker segmentation system. Specifically, we present two novel strategies for reducing the false alarm rate with a minimal impact on the true speaker change detection rate. One of the new strategies rejects, given a discard probability, those changes that are suspicious of being false alarms because of their low ΔBIC value; and the other one assumes that the occurrence of changes constitute a Poisson process, so changes will be discarded with a probability that follows a Poisson cumulative density function. Our experiments show the improvements obtained with each false alarm reduction approach using the Spanish Parliament Sessions defined for the 2006 TC-STAR Automatic Speech Recognition evaluation campaign.

Speaker segmentation systems usually have problems detecting short segments, which causes the number of deletions to be high and therefore harming the performance of the system. This is a complication when it comes to segmenting multimedia information such as movies and TV shows, where dialogs among characters are very common. In this paper a modification of the BIC algorithm is presented, which will reduce remarkably the number of deletions without causing an increase in the number of false alarms. This modification, referred to as MultiBIC, assumes that two change-points are present in a window of data, while conventional BIC approach supposes that there is just one. This causes the system to notice when there is more than one change-point in a window, finding shorter segments than traditional BIC.

A new multimodal biometric database designed and acquired within the framework of the European BioSecure Network of Excellence is presented. It is comprised of more than 600 individuals acquired simultaneously in three scenarios: 1) over the Internet, 2) in an office environment with desktop PC, and 3) in indoor/outdoor environments with mobile portable hardware. The three scenarios include a common part of audio/video data. Also, signature and fingerprint data have been acquired both with desktop PC and mobile portable hardware. Additionally, hand and iris data were acquired in the second scenario using desktop PC. Acquisition has been conducted by 11 European institutions. Additional features of the BioSecure Multimodal Database (BMDB) are: two acquisition sessions, several sensors in certain modalities, balanced gender and age distributions, multimodal realistic scenarios with simple and quick tasks per modality, cross-European diversity, availability of demographic data, and compatibility with other multimodal databases. The novel acquisition conditions of the BMDB allow us to perform new challenging research and evaluation of either monomodal or multimodal biometric systems, as in the recent BioSecure Multimodal Evaluation campaign. A description of this campaign including baseline results of individual modalities from the new database is also given. The database is expected to be available for research purposes through the BioSecure Association during 2008.

Despite of the advances in the e-learning domain during the last decades, there is a lack of suitable mechanism to carry out assessment with the appropriate measures to avoid cheating. Current LMSs do not provide the needed features to check that the intended student is taking the online exam by himself, or even to know if he has spent the whole session time in front of the computer. This paper presents a web-based application that offers biometric authentication based on face-recognition. This application, which can be easily integrated in currently available LMSs, is able to use face-recognition during access control, tracking and assessment. Thus it is possible to improve security during critical phases in the learning process (e.g. assessment).

The paper describes our current work on speech-to-text transcription for recordings in two languages. The experimental framework consists of television news shows in Galician and, to some extent, Spanish languages. A priori language detection is avoided so a bilingual speech recognition system has been developed and its performance is presented. Better results are obtained when speaker and speech style is taken into account through adaptation of both acoustic and language models. Special attention must be paid to the limited resources available in the experimental framework.

In the framework of the EU-funded Project LC-STAR, a set of Language Resources (LR) for all the Speech to Speech Translation components (Speech recognition, Machine Translation and Speech Synthesis) was developed. This paper deals with the development of bilingual corpora in Spanish, US English and Catalan. The corpora were obtained from spontaneous dialogues in one of these three languages which were translated to the other two languages. The paper describes the translation methodology, specific problems of translating spontaneous dialogues to be used for MT training, formats and the validation criteria.

To assess the correctness of a recognizer output in any instance of a dialogue is a complex task that has been studied thoroughly during the past decade. Its importance relays on the need for robust dialogue systems, capable of dealing with difficulties inherent to human-machine communications: user errors and corrections, speech recognizer errors, error recovery techniques, etc. In this paper, we present a novel approach to the problem of deciding what the user has said. We use confidence measures derived from low level knowledge sources (acoustic and linguistic information) and generated in parallel from several topic-adapted speech recognizers. Each recognizer is aimed to the recognition of a particular topic, and confidence measures are compared through the use of a classifier that lead to a most probable solution. This approach shows to be specially suited for difficult topics, such as proper names or confirmations, which are highly meaningful for error correction tasks. These topics present high error rates when using an application-wide speech recognizer, but recognition correction is greatly enhanced through the use of parallel recognizers. Moreover, the use of topic-adapted recognizers seems to help also in the identification of the user intention and in the detection of outof-application utterances. 1.

This paper describes a large scale experiment in which eight research institutions have tested their audio partitioning and labeling algorithms on the same data, a multi-lingual database of news broadcasts, using the same evaluation tools and protocols. The experiments have provide more insight in the cross-lingual robustness of the methods and they have demonstrated that by further collaborating in thedomains of speaker change detection and speaker clustering it should be possible to achieve further technological progress in the near future.

In distributed speech recognition, speech feature vectors are obtained at the client side, and transmitted to the remote server for recognition. In this paper, we investigate the robustness of the remote recognizer against the inherent packet loss in an Internet communication. In the decoding process, we propose to apply techniques used for "missing data" problems. The idea is to use a simple approach of error concealment to recover the non-received speech frames, and then to consider these recovered speech frames as not completely reliable. Thus, at recognition stage, our recognition engine uses a weighted (or soft decision) Viterbi algorithm in order to take into account the reliability of the recovered speech frames. Results on Aurora databases show that the proposed approach provides good recognition performance over a wide range of network conditions.

The paper deals with the task of audio segmentation in TV broadcast news. A multimedia approach for this purpose, by means of audio and video processing, is proposed. Thus, the segmentation system is composed by two differentiated parts: one analyzes the audio stream, and is based on the well-known Bayesian information criterion (BIC), whereas the other part extracts useful information from the video stream to improve the performance of BIC. An investigation of parameters involved in BIC formulation is also accomplished, in order to achieve the best results possible in our experimental framework: the database Transcrigal-DB. The final system provides significative improvements in both overall performance and robustness.

In this paper we present a fast method to implement a language model (LM) look-ahead algorithm in a Viterbi-based, single-lexical-tree speech recognizer. We have used three different mechanisms to speed up the calculation: a cache memory attached to each node or the network, a pre-calculation of the probabilities of the active contexts, and an organization of the LM using perfect hash. These enhancements make it possible to use the full trigram LM to compute the look-ahead with better overall results, both in terms of recognition rate and computation time, than using a unigram or bigram based approximation.

We present results on the performance of Distributed Speech Recognition operating over simulated IP networks. ETSI AURORA front-end running at client nodes extracts the speech parameters, packetizes and sends them as real-time IP traffic to a remote recognizer based on Continuous Density Hidden Markov Models. The experimental framework is the ETSI STQ-AURORA Project Database 2.0. The impact of transmission over IP networks is modeled by (1) random losses, (2) losses generated by a Gilbert model and (3) network simulations. Results show that random losses and moderately bursty losses do not significantly affect the recognition performance. Strongly bursty packet losses, as those generated by real-time and Web traffic competing over a network bottleneck, instead, can have a very negative impact on recognition performance, indicating that DSR over the Internet, to be successful, requires high levels of Quality of Service.

Abstract In this paper, we describe the design procedure for a wirelesscommunication interactive voice response system. The applica-tion must work in a very noisy environment which has imposedmany design constraints. Wewilladdress the sensible aspects ofthree components of the application: the voice activity detector(VAD),the automatic speech recognition (ASR)system, and theconfidence measure (CM) determination. In order to get a sat-isfactory product, it has been necessary to reduce the importantmismatch between available linguistic and acoustic resourcesand the operational environment. Adaptation techniques for theacoustic models of the speech recognition system have provento be effective to speed up the application deployment time. 1. Introduction It is widely accepted that speech technology is nowadays ma-ture enough for the deployment of practical, commercial ap-plications in the real-world. In the last years, many servicesand products have shown up specially in the telecommunicationarena[1]. Also, many resources as speech and text databases,recognition engines, text-to-speech engines are available to thedevelopers for building up applications and products [2]. Nev-ertheless, the concourse of highly specialized and well-trainedpersonnel is required in order to get a quality product. Even af-ter a careful design of the project specifications and an adequateselection of linguistic and software components, an error-and-trial strategy must be conducted in order to tune and debug allthe components. This is even more delicate when there is a sig-nificantmismatch between theavailable linguisticresources andthe actual acoustic and linguistic environment. This is the caseof the project we will describe in this paper. The applicationitself, a command-control system, is simple in nature, but theacoustic environment (a very noisy lumberyard), the transmis-sion channel (a half-duplex radio channel), and the two possiblelanguages (Galician and Spanish) are factors that make very dif-ficult the transfer of technology from off-the-shelf componentsto a complete product.A state-of-the-art system may perform poorly when the testdata are collected under a totally different environmental con-dition. Regarding to the possible mismatches, both linguisticand acoustic mismatches might occur. A linguistic mismatchis mainly caused by incomplete task specification, inadequateknowledge representations, and insufficient training data. Onthe other hand, an acoustic mismatch between training and op-erational conditions arises, being in this case changes in trans-ducers, channel, speaker environment, background noise, etc.We will address how some techniques must be applied inorder to improve the performance and ergonomy of the appli-cation. We have selected three subsystems that required of spe-cial attention: the voice activity detector (VAD), the automaticspeech recognition (ASR) system, and the confidence measure(CM) determination. Before describing the most relevant as-pect of each of them, we will show an overview of the wholeapplication. Some conclusions will be drawn at the end of thepaper.

We investigated how hoarseness effects perceived hypernasality ratings in order to make a rating scale for hoarse voices. Thirty stimuli, of which 18 were target stimuli (three each for voices with and without hoarseness at three levels of hypernasality), and of which 12 were foil stimuli. These voices were listened to by four experienced speech pathologists. They were asked to rate hypernasality on 5-point scale. Perceived hypernasality decreased with hoarseness for a severely hypernasal voice, but results varied among the listeners for voices with no and moderate hypernasality.

In this paper we present a novel technique for combining a Speaker Verification System with an Utterance Verification System in a Speaker Authentication system over the telephone. Speaker Verification consists in accepting or rejecting the claimed identity of a speaker by processing samples of his/her voice. Usually, these systems are based on HMM's that try to represent the characteristics of the talkers' vocal tracts [1]. Utterance Verification systems make use of a set of speakerindependent speech models to recognize a certain utterance and decide whether a speaker has uttered it or not. If the utterances consist of passwords, this can be used for identity verification purposes [2][3]. Up to now, both techniques have been used separately. This paper is focused on the problem of how to combine these two sources of information. A new architecture is presented to join an utterance verification system and a speaker verification system in order to improve the performance in a textdependent speaker verification task.

Text-to-speech (T-T-S) systems based on the concatenation of speech units need a prosodic modification algorithm to adjust the prosodic features of the stored speech units to the desired output values. We discuss the application of a sinusoidal shape-invariant model to a T-T-S system for Spanish, paying special attention to the concatenation issues and phase treatment. The resulting speech waveform resembles the waveform of its contributory units, without sounding reverberant as in other sinusoidal implementations.

The authors describe the main topics related to a real-time implementation of the Proposed Federal Standard 1016 (PFS-1016) 4800 bps CELP (code excited linear prediction) voice coder together with some alternatives to extend real-time CELP schemes over a wide range of bit rates (8000, 6500, and 2400 bps are considered). Several procedures are evaluated in order to select the options in PFS-1016 that provide the highest quality for an implementation based on a single AT&T DSP32C chip. From the basic CELP scheme, higher and lower bit rates are derived after evaluation of the efficiency obtained for the different elements in this speech coding algorithm.>

The main results from work on analysis techniques and quantization procedures for real-time implementation of the Proposed Federal Standard 1016 4800-b/s voice coder developed by the US DoD and AT&T Bell Laboratories are presented. Reduced search procedures are described for the adaptive and stochastic codebooks, two basic functions in code-excited linear predictive coders (CELPs), where great computational effort is required. An efficient procedure for the joint computation and quantization of the line spectrum pair (LSP) is also proposed. Details about a preliminary real-time implementation on a commercial board with a single DSP32C processor are described.>

A speech coder is presented which combines vector excitation coding (VXC) with frequency-domain representations so as to obtain a high-quality efficient scheme at 4.8 kb/s based on the use of different coding strategies and bit allocations for different frames of speech and different frequency bands. Attention is focused on the representation of voiced sounds and on multiband procedures for representing the short-time Fourier transform. The proposed scheme starts with a phonetically based frame segmentation. Multiband long-term prediction is included only when necessary, and random excitation in voiced sounds is spectrally shaped by using a linear predictive coding envelope from the long-term prediction error.>

The authors discuss various possible synthesis procedures for frequency-domain vector-excited coding (VXC). The algorithm, called vector adaptive transform coding (VATC), is a two-stage vector quantizer for the short-time Fourier transform (STFT) of speech. One stage represented by an adaptive codebook based on long-term prediction and the other based by a random codebook. The authors focus on the differences between time and frequency VXC, stressing the importance of the long-term prediction. A fast algorithm for long-term prediction is introduced, and different strategies for noise reduction are proposed. VATC is presented as a particular case of a general vector linear prediction (VLP) scheme. Following this approach, a coder based on a self-spectral shaping procedure is proposed. This coder has the potential of producing telephonic-quality speech at rates lower than 4.8 kb/s.>