We propose a Speech-to-Text Translation (S2TT) approach that integrates phoneme representations into a Chain-of-Thought (CoT) framework to improve translation in low-resource and zero-resource settings. By introducing phoneme recognition as an intermediate step, we enhance cross-lingual transfer, enabling translation even for languages with no labeled speech data. Our system builds on a multilingual LLM, which we extend to process speech and phonemes. Training follows a curriculum learning strategy that progressively introduces more complex tasks. Experiments on multilingual S2TT benchmarks show that phoneme-augmented CoT improves translation quality in low-resource conditions and enables zero-resource translation, while slightly impacting high-resource performance. Despite this trade-off, our findings demonstrate that phoneme-based CoT is a promising step toward making S2TT more accessible across diverse languages.

Code-switching (CS), the alternating use of two or more languages, challenges automatic speech recognition (ASR) due to scarce training data and linguistic similarities. The lack of dedicated CS datasets limits ASR performance, as most models rely on monolingual or mixed-language corpora that fail to reflect real-world CS patterns. This issue is critical in multilingual societies where CS occurs in informal and formal settings. A key example is Catalan-Spanish CS, widely used in media and parliamentary speeches. In this work, we improve ASR for Catalan-Spanish CS by exploring three strategies: (1) generating synthetic CS data, (2) concatenating monolingual audio, and (3) leveraging real CS data with language tokens. We extract CS data from Catalan speech corpora and fine-tune OpenAI’s Whisper models, making them available on Hugging Face. Results show that combining a modest amount of synthetic CS data with the dominant language token yields the best transcription performance.

Mamba, a state space model-based architecture, is emerging as a strong alternative to Transformer models, showing equal or superior performance in sequence generation, including speech. However, analyses have focused mainly on highresource scenarios. This paper explores Mamba’s potential in ASR for low-resource scenarios. We compare the Transformerbased Conformer and its state-space counterpart, ConMamba, across nine languages with varying training data. Our results show that ConMamba achieves similar WER to Conformer for short-context inputs but significantly improves performance on long-context inputs, reducing WER by up to 50% on average. Additionally, ConMamba enhances efficiency, requiring 40–45% less training time, using 50% less memory, and accelerating inference by 63–70%, making it a more effective ASR solution across different data availability scenarios.

The aim of this paper is to investigate the benefit of combining both language and acoustic modelling for speaker diarization. Although conventional systems only use acoustic features, in some scenarios linguistic data contain high discriminative speaker information, even more reliable than the acoustic ones. In this study we analyze how an appropriate fusion of both kind of features is able to obtain good results in these cases. The proposed system is based on an iterative algorithm where a LSTM network is used as a speaker classifier. The network is fed with character-level word embeddings and a GMM based acoustic score created with the output labels from previous iterations. The presented algorithm has been evaluated in a Call-Center database, which is composed of telephone interview audios. The combination of acoustic features and linguistic content shows a 84.29% improvement in terms of a word-level DER as compared to a HMM/VB baseline system. The results of this study confirms that linguistic content can be efficiently used for some speaker recognition tasks.

Abstract In the context of object detection, sliding-window classifiers and single-shot convolutional neural network (CNN) meta-architectures typically yield multiple overlapping candidate windows with similar high scores around the true location of a particular object. Non-maximum suppression (NMS) is the process of selecting a single representative candidate within this cluster of detections, so as to obtain a unique detection per object appearing on a given picture. In this paper, we present a highly scalable NMS algorithm for embedded graphics processing unit (GPU) architectures that is designed from scratch to handle workloads featuring thousands of simultaneous detections on a given picture. Our kernels are directly applicable to other sequential NMS algorithms such as FeatureNMS, Soft-NMS or AdaptiveNMS that share the inner workings of the classic greedy NMS method. The obtained performance results show that our parallel NMS algorithm is capable of clustering 1024 simultaneous detected objects per frame in roughly 1 ms on both Tegra X1 and Tegra X2 on-die GPUs, while taking 2 ms on Tegra K1. Furthermore, our proposed parallel greedy NMS algorithm yields a 14–40x speed up when compared to state-of-the-art NMS methods that require learning a CNN from annotated data.

Most state-of-the-art Deep Learning systems for text-independent speaker verification are based on speaker embedding extractors. These architectures are commonly composed of a feature extractor front-end together with a pooling layer to encode variable-length utterances into fixed-length speaker vectors. In this paper we present Double Multi-Head Attention (MHA) pooling, which extends our previous approach based on Self MHA. An additional self attention layer is added to the pooling layer that summarizes the context vectors produced by MHA into a unique speaker representation. This method enhances the pooling mechanism by giving weights to the information captured for each head and it results in creating more discriminative speaker embeddings. We have evaluated our approach with the VoxCeleb2 dataset. Our results show 6.09% and 5.23% relative improvement in terms of EER compared to Self Attention pooling and Self MHA, respectively. According to the obtained results, Double MHA has shown to be an excellent approach to efficiently select the most relevant features captured by the CNN-based front-ends from the speech signal.

Probabilistic Linear Discriminant Analysis (PLDA) is the most efficient backend for i-vectors. However, it requires labeled background data which can be difficult to access in practice. Unlike PLDA, cosine scoring avoids speaker-labels at the cost of degrading the performance. In this work, we propose a post processing of i-vectors using a Deep Neural Network (DNN) to transform i-vectors into a new speaker vector representation. The DNN will be trained using i-vectors that are similar to the training i-vectors. These similar i-vectors will be selected in an unsupervised manner. Using the new vector representation, we will score the experimental trials using cosine scoring. The evaluation was performed on the speaker verification trials of VoxCeleb-1 database. The experiments have shown that with the help of the similar i-vectors the new vectors become more discriminative than the original i-vectors. The new vectors have gained a relative improvement of 53% in terms of EER, compared to the conventional i-vector/PLDA system, but without using speaker labels.

Speaker labeled background data is an essential requirement for most state-of-the-art approaches in speaker recognition, e.g., xvectors and i-vector/PLDA. However, in reality it is difficult to access large amount of labeled data. In this work, we propose siamese networks for speaker verification without using speaker labels. We propose two different siamese networks having two and three branches, respectively, where each branch is a CNN encoder. Since the goal is to avoid speaker labels, we propose to generate the training pairs in an unsupervised manner. The client samples are selected within one database according to highest cosine scores with the anchor in i-vector space. The impostor samples are selected in the same way but from another database. Our double-branch siamese performs binary classification using cross entropy loss during training. In testing phase, we obtain speaker verification scores directly from its output layer. Whereas, our triple-branch siamese is trained to learn speaker embeddings using triplet loss. During testing, we extract speaker embeddings from its output layer, which are scored in the experiments using cosine scoring. The evaluation is performed on VoxCeleb-1 database, which show that using the proposed unsupervised systems, solely or in fusion, the results get closer to supervised baseline

The computing power of mobile devices limits the end-user applications in terms of storage size, processing, memory and energy consumption. These limitations motivate researchers for the design of more efficient deep models. On the other hand, self-attention networks based on Transformer architecture have attracted remarkable interests due to their high parallelization capabilities and strong performance on a variety of Natural Language Processing (NLP) applications. Inspired by the Transformer, we propose a tandem Self-Attention Encoding and Pooling (SAEP) mechanism to obtain a discriminative speaker embedding given non-fixed length speech utterances. SAEP is a stack of identical blocks solely relied on self-attention and position-wise feed-forward networks to create vector representation of speakers. This approach encodes short-term speaker spectral features into speaker embeddings to be used in text-independent speaker verification. We have evaluated this approach on both VoxCeleb1 & 2 datasets. The proposed architecture is able to outperform the baseline x-vector, and shows competitive performance to some other benchmarks based on convolutions, with a significant reduction in model size. It employs 94%, 95%, and 73% less parameters compared to ResNet-34, ResNet-50, and x-vector, respectively. This indicates that the proposed fully attention based architecture is more efficient in extracting time-invariant features from speaker utterances.

Most state-of-the-art Deep Learning (DL) approaches for speaker recognition work on a short utterance level. Given the speech signal, these algorithms extract a sequence of speaker embeddings from short segments and those are averaged to obtain an utterance level speaker representation. In this work we propose the use of an attention mechanism to obtain a discriminative speaker embedding given non fixed length speech utterances. Our system is based on a Convolutional Neural Network (CNN) that encodes short-term speaker features from the spectrogram and a self multi-head attention model that maps these representations into a long-term speaker embedding. The attention model that we propose produces multiple alignments from different subsegments of the CNN encoded states over the sequence. Hence this mechanism works as a pooling layer which decides the most discriminative features over the sequence to obtain an utterance level representation. We have tested this approach for the verification task for the VoxCeleb1 dataset. The results show that self multi-head attention outperforms both temporal and statistical pooling methods with a 18\% of relative EER. Obtained results show a 58\% relative improvement in EER compared to i-vector+PLDA.

In the last years, i-vectors followed by cosine or PLDA scoringtechniques were the state-of-the-art approach in speaker veri-fication. PLDA requires labeled background data, and thereexists a significant performance gap between the two scoringtechniques. In this work, we propose to reduce this gap by us-ing an autoencoder to transform i-vector into a new speaker vec-tor representation, which will be referred to as ae-vector. Theautoencoder will be trained to reconstruct neighbor i-vectors in-stead of the same training i-vectors, as usual. These neighbori-vectors will be selected in an unsupervised manner accordingto the highest cosine scores to the training i-vectors. The evalua-tion is performed on the speaker verification trials of VoxCeleb-1 database. The experiments show that our proposed ae-vectorsgain a relative improvement of 42% in terms of EER comparedto the conventional i-vectors using cosine scoring, which fillsthe performance gap between cosine and PLDA scoring tech-niques by 92%, but without using speaker labels

Over the last few years, i-vectors have been the state-of-the-art technique in speaker recognition. Recent advances in Deep Learning (DL) technology have improved the quality of i-vectors but the DL techniques in use are computationally expensive and need phonetically labeled background data. The aim of this work is to develop an efficient alternative vector representation of speech by keeping the computational cost as low as possible and avoiding phonetic labels, which are not always accessible. The proposed vectors will be based on both Gaussian Mixture Models (GMM) and Restricted Boltzmann Machines (RBM) and will be referred to as GMM–RBM vectors. The role of RBM is to learn the total speaker and session variability among background GMM supervectors. This RBM, which will be referred to as Universal RBM (URBM), will then be used to transform unseen supervectors to the proposed low dimensional vectors. The use of different activation functions for training the URBM and different transformation functions for extracting the proposed vectors are investigated. At the end, a variant of Rectified Linear Units (ReLU) which is referred to as variable ReLU (VReLU) is proposed. Experiments on the core test condition 5 of NIST SRE 2010 show that comparable results with conventional i-vectors are achieved with a clearly lower computational load in the vector extraction process.

This paper presents a new speaker change detection system based on Long Short-Term Memory (LSTM) neural networks using acoustic data and linguistic content. Language modelling is combined with two different Joint Factor Analysis (JFA) acoustic approaches: i-vectors and speaker factors. Both of them are compared with a baseline algorithm that uses cosine distance to detect speaker turn changes. LSTM neural networks with both linguistic and acoustic features have been able to produce a robust speaker segmentation. The experimental results show that our proposal clearly outperforms the baseline system.

The lack of labeled background data makes a big performance gap between cosine and Probabilistic Linear Discriminant Analysis (PLDA) scoring baseline techniques for i-vectors in speaker recognition. Although there are some unsupervised clustering techniques to estimate the labels, they cannot accurately predict the true labels and they also assume that there are several samples from the same speaker in the background data that could not be true in reality. In this paper, the authors make use of Deep Learning (DL) to fill this performance gap given unlabeled background data. To this goal, the authors have proposed an impostor selection algorithm and a universal model adaptation process in a hybrid system based on deep belief networks and deep neural networks to discriminatively model each target speaker. In order to have more insight into the behavior of DL techniques in both single- and multisession speaker enrollment tasks, some experiments have been carried out in this paper in both scenarios. Experiments on National Institute of Standards and Technology 2014 i-vector challenge show that 46% of this performance gap, in terms of minimum of the decision cost function, is filled by the proposed DL-based system. Furthermore, the score combination of the proposed DL-based system and PLDA with estimated labels covers 79% of this gap.

With the emergence of GPU computing, deep neural networks have become a widely used technique for advancing research in the field of image and speech processing. In the context of object and event detection, sliding-window classifiers require to choose the best among all positively discriminated candidate windows. In this paper, we introduce the first GPU-based non-maximum suppression (NMS) algorithm for embedded GPU architectures. The obtained results show that the proposed parallel algorithm reduces the NMS latency by a wide margin when compared to CPUs, even clocking the GPU at 50% of its maximum frequency on an NVIDIA Tegra K1. In this paper, we show results for object detection in images. The proposed technique is directly applicable to speech segmentation tasks such as speaker diarization.

This paper is focused on the application of the Language Identification (LID) technology for intelligent vehicles. We cope with short sentences or words spoken in moving cars in four languages: English, Spanish, German, and Finnish. As the response time of the LID system is crucial for user acceptance in this particular task, speech signals of different durations with total average of 3.8s are analyzed. In this paper, the authors propose the use of Deep Neural Networks (DNN) to model effectively the i-vector space of languages. Both raw i-vectors and session variability compensated i-vectors are evaluated as input vectors to DNNs. The performance of the proposed DNN architecture is compared with both conventional GMM-UBM and i-vector/LDA systems considering the effect of durations of signals. It is shown that the signals with durations between 2 and 3s meet the requirements of this application, i.e., high accuracy and fast decision, in which the proposed DNN architecture outperforms GMM-UBM and i-vector/LDA systems by 37% and 28%, respectively.

i-vector modeling techniques have been successfully used for speaker clustering task recently. In this work, we propose the extraction of i-vectors from short-and long-term speech features, and the fusion of their PLDA scores within the frame of speaker diarization. Two sets of i-vectors are first extracted from short-term spectral and longterm\nvoice-quality, prosodic and glottal to noise excitation ratio (GNE) features. Then, the PLDA scores of these two ivectors are fused for speaker clustering task. Experiments have been carried out on single and multiple site scenario test sets of Augmented Multi-party Interaction (AMI) corpus. Experimental results show that i-vector based PLDA speaker clustering technique provides a significant diarization error rate (DER) improvement than GMM based BIC\nclustering technique.

The use of Restricted Boltzmann Machines (RBM) is proposed in this paper as a non-linear transformation of GMM supervectors for speaker recognition. It will be shown that the RBM transformation will increase the discrimination power of raw GMM supervectors for speaker recognition. The experimental results on the core test condition of the NIST SRE 2006 corpus show that the proposed RBM supervectors will achieve a comparable performance to i-vectors. Furthermore, the combination of RBM supevectors and i-vectors in the score level improves the performance of the i-vector approach by more than 10% in terms of EER.

Jitter and shimmer voice-quality measurements have been successfully used to detect voice pathologies and classify different speaking styles. In this paper, we investigate the usefulness of jitter and shimmer voice measurements in the framework of the speaker diarization task. The combination of jitter and shimmer voice-quality features with the long-term prosodic and short-term spectral features is explored in a subset of the Augmented Multi-party Interaction (AMI) corpus, a multi-party and spontaneous speech set of recordings. The best results have been obtained by fusing the voice-quality features with the prosodic ones at the feature level, and then fusing them with the spectral features at the score level. Experimental results show more than 20% relative DER improvement compared to the spectral baseline system.

The use of Deep Belief Networks (DBNs) is proposed in this paper to model discriminatively target and impostor i-vectors in a speaker verification task. The authors propose to adapt the network parameters of each speaker from a background model, which will be referred to as Universal DBN (UDBN). It is also suggested to backpropagate class errors up to only one layer for few iterations before to train the network. Additionally, an impostor selection method is introduced which helps the DBN to outperform the cosine distance classifier. The evaluation is performed on the core test condition of the NIST SRE 2006 corpora, and it is shown that 10% and 8% relative improvements of EER and minDCF can be achieved, respectively.

The goal of face detection is to determine the presence of faces in arbitrary images, along with their locations and dimensions. As it happens with any graphics workloads, these algorithms benefit from data-level parallelism. Existing parallelization efforts strictly focus on mapping different divide and conquer strategies into multicore CPUs and GPUs. However, even the most advanced single-chip many-core processors to date are still struggling to effectively handle real-time face detection under high-definition video workloads. To address this challenge, face detection algorithms typically avoid computations by dynamically evaluating a boosted cascade of classifiers. Unfortunately, this technique yields a low ALU occupancy in architectures such as GPUs, which heavily rely on large SIMD widths for maximizing data-level parallelism. In this paper we present several techniques to increase the performance of the cascade evaluation kernel, which is the most resource-intensive part of the face detection pipeline. Particularly, the usage of concurrent kernel execution in combination with cascades generated with the Gentle Boost algorithm solves the problem of GPU underutilization, and achieves a 5X speedup in 1080p videos on average over the fastest known implementations, while slightly improving the accuracy. Finally, we also studied the parallelization of the cascade training process and its scalability under SMP platforms. The proposed parallelization strategy exploits both task and data-level parallelism and achieves a 3.5X speedup over single-threaded implementations.

This work presents a novel two-step algorithm to estimate the\norientation of speakers in a smart-room environment equipped\nwith microphone arrays. First the position of the speaker is\nestimated by the SRP-PHAT algorithm, and the time delay of\narrival for each microphone pair with respect to the detected\nposition is computed. In the second step, the value of the cross-\ncorrelation at the estimated time delay is used as the fundamen-\ntal characteristic from where to derive the speaker orientation. The proposed method performs consistently better than other state-of-the-art acoustic techniques with a purposely recorded database and the CLEAR head pose database.

The handling of overlapping speech in the context of speaker diarisation attracted in recent years the interest of the scientific community, since speaker overlap was identified as one of the factors degrading the performance of conventional diarisation systems. In this study, the authors are discussing the possibility of using long-term prosodic features for the detection of overlapping speech, which is subsequently employed in speaker diarisation to improve the baseline system. The most relevant subset from the set of candidate prosodic features is determined in two steps. First, a ranking according to minimal-redundancy-maximal-relevance criterion is obtained, and then a hill-climbing wrapper strategy is applied for determining the optimal number of prosodic features, which should accompany short-term spectral features for overlap detection. In experiments on the augmented multi-party interaction (AMI) meeting distant-channel data, the authors show that the addition of prosodic features decreased overlap detection error. Detected overlap segments were used in speaker diarisation to recover missed speech by assigning multiple speaker labels and to increase the purity of speaker clusters. Improvements of the baseline diarisation system are reported in both single- and multi-site data conditions. However, the extension of the diarisation system with TDOAs showed its incompatibility with the overlap exclusion technique.

Simultaneous speech poses a challenging problem for conventional speaker diarization systems. In meeting data, a substantial amount of missed speech error is due to speaker overlaps, since usually only one speaker label per segment is assigned. Furthermore, simultaneous speech included in training data can lead to corrupt speaker models and thus worse segmentation performance. In this paper, we propose the use of three spatial cross-correlation-based features together with spectral information for speaker overlap detection on distant microphones. Different microphone-pair data are fused by means of principal component analysis. We have obtained an improvement of the speaker diarization system over the baseline by discarding overlap segments from model training and assigning two speaker labels to them according to likelihoods in Viterbi decoding. In experiments conducted on the AMI Meeting corpus, we achieve a relative DER reduction of 11.2% and 17.0% for single- and multi-site data, respectively. The improvement of clustering with techniques such as beamforming and TDOA-feature stream also leads to a higher effectiveness of the overlap labeling algorithm. Preliminary experiments with NIST RT data show DER improvement on the RT'09 meeting recordings as well.

Overlapping speech is responsible for a certain amount of errors produced by tandard speaker diarization systems in meeting environment. We are investigating a set of prosody-based long-term features as a potential complement to our overlap detection\nsystem relying on short-term spectral parameters. The most relevant features are selected in a two-step process. They\nare firstly evaluated and sorted according to mRMR criterion and then the optimal number is determined by iterative wrapper\napproach. We show that the addition of prosodic features decreased overlap detection error. Detected overlap segments are used in speaker diarization to recover missed speech by assigning multiple speaker labels and to increase the purity of speaker clusters.

A substantial portion of errors of the conventional speaker\ndiarization systems on meeting data can be accounted to overlapped\nspeech. This paper proposes the use of several spatial\nfeatures to improve speech overlap detection on distant channel\nmicrophones. These spatial features are integrated into a\nspectral-based system by using principal component analysis\nand neural networks. Different overlap detection hypotheses\nare used to improve diarization performance with both overlap\nexclusion and overlap labeling. In experiments conducted\non AMI Meeting Corpus we demonstrate a relative DER improvement\nof 11.6% and 14.6% for single- and multi-site data,\nrespectively.

The detection of the acoustic events (AEs) that are naturally \nproduced in a meeting room may help to describe the human \nand social activity that takes place in it. When applied to \nspontaneous recordings, the detection of AEs from only audio \ninformation shows a large amount of errors, which are mostly due to temporal overlapping of sounds. In this paper, a system to detect and recognize AEs using both audio and video information is presented. A feature-level fusion strategy is used, and the structure of the HMM-GMM based system considers each class separately and uses a one-against-all strategy for training. Experime ntal AED results with a new and rather spontaneous dataset are presented which show the advantage of the proposed approach.

In multimodal fusion systems a normalization of the features or the scores is needed before the fusion process. In this work, in addition to the conventional methods, histogram equalization, which was recently introduced by the authors in multimodal systems, and Bi-Gaussian equalization, which takes into account the separate statistics of the genuine and impostor scores, and is introduced in this paper, are applied upon the scores in a multimodal SVM-based person verification system composed by prosodic, speech spectrum, and face information. Bi-Gaussian equalization has obtained the best results and outperform in more than a 23.25% the results obtained by Min-Max normalization.

Comunicació presentada a 9th Annual Conference of the International Speech Communication Association celebrada a Brisbane (Australia) del 22 al 26 de setembre de 2008.

This paper proposes an initialization for an agglomerative\nsystem applied to speaker diarization in the meeting environment.\nThe initialization is based on a previous clustering of the\ntemporal sequence generated by the estimation of the Time Delay\nof Arrival (TDOA) among pair of sensors. That initial clustering\nhas the purpose of obtaining initial classes with speaker\ninformation from a sole speaker. The aim is to ensure the purity\nof the initial segments based on the position of the speakers in\na meeting along time. The TDOA initialization was tested with\nthe dataset used in the RT07s evaluation where an improvement\nof the diariazation error rate is obtained with respect to the classical\nuniform initialization. The most of the experiments show\nthat the purity of the beginning segments leads to a better clustering\non the posterior hierarchical strategy based on cepstral\nfeatures.

This paper presents a novel approach to speaker orientation\nestimation in a SmartRoom environment equipped with\nmultiple microphones. The ratio between the high and low\nband energies (HLBR) received at each microphone has been\nshown in our previous work to be a potentially approach to estimate\nthe direction of the voice produced by a speaker. In this\nwork, for each microphone pair, a smoothed CPS phase is obtained\nby a proper windowing of the main peak of the crosscorrelation\nsequence estimated with the GCC-PHAT method,\nand a HLBR is computed from the processed CPS. The proposed\nmethod keeps the computational simplicity of the HLBR\nalgorithm while adding the robustness offered by the GCCPHAT\ntechnique. Experimental preliminary results were conducted\nover a database recorded purposely in the UPC Smart\nroom, and over the CLEAR head pose database. The proposed\nmethod performs consistently better than other state-of-the-art\ntechniques with both databases.

Accurate modeling of speaker clusters is important in the task of speaker diarization. Creating accurate models involves both selection of the model complexity and optimum training given the data. Using models with fixed complexity and trained using the standard EM algorithm poses a risk of overfitting, which can lead to a reduction in diarization performance. In this paper a technique proposed by the author to estimate the complexity of a model is combined with a novel training algorithm called "cross-validation EM" to control the number of training iterations. This combination leads to more robust speaker modeling and results in an increase in speaker diarization performance. Tests on the NIST RT (MDM) datasets for meetings show a relative improvement of 10.6% relative on the test set.

In the task of speaker diarization for meetings it has been shown in previous work that it is useful to use the time delay of arrival (TDOA) between the different audio channels in the meeting room as an extra source of information in addition to the acoustic features. When combining feature streams, we use a weight to control the relative contributions of the streams. In the past, this weight was determined using development data and the same weight value was applied to all meetings. In this paper we present a method for automatically determining the weight. A metric derived from the Bayesian information criterion (BIC) computed for each feature stream estimates the weight for each meeting on the initial clustering iteration and adapts its value throughout the diarization process. By using this technique we achieve a more robust system and up to 18.2% relative improvement over the method of tuning the weight on development data.

This paper presents a multimodal approach to head pose estimation and 3D gaze orientation of individuals in a SmartRoom environment equipped with multiple cameras and microphones. We first introduce the two monomodal approaches as reference. In video, we estimate head orientation from color information by exploiting spatial redundancy among cameras. Audio information is processed to estimate the direction of the voice produced by a speaker making use of the directivity characteristics of the head radiation pattern. Two multimodal information fusion schemes working at data and decision levels are analyzed in terms of accuracy and robustness of the estimation. Experimental results conducted over the CLEAR evaluation database are reported and the comparison of the proposed multimodal head pose estimation algorithms with the reference monomodal approaches proves the effectiveness of the proposed approach.

The head orientation of human speakers in a smart-room affects the quality of the signals recorded by far-field microphones, and consequently influences the performance of the technologies deployed based on those signals. Additionally, knowing the orientation in these environments can be useful for the development of several multimodal advanced services, for instance, in microphone network management. Consequently, head orientation estimation has recently become a growing interesting research topic. In this paper, we propose two different approaches to head orientation estimation on the basis of multimicrophone recordings: first, an approach based on the generalization of the well-known SRP-PHAT speaker localization algorithm, and second a new approach based on measurements of the ratio between the high and the low band speech energies. Promising results are obtained in both cases, with a generalized better performance of the algorithms based on speaker localization methods. Index Terms: head orientation estimation, microphone arrays, speaker tracking

Comunicació presentada a: 8th Annual Conference of the International Speech Communication Association a Antwerp (Belgium) celebrada del 27 al 31 d'agost de 2007.

When performing speaker diarization on recordings from meetings, multiple microphones of different qualities are usually available and distributed around the meeting room. Although several approaches have been proposed in recent years to take advantage of multiple microphones, they are either too computationally expensive and not easily scalable or they cannot outperform the simpler case of using the best single microphone. In this paper, the use of classic acoustic beamforming techniques is proposed together with several novel algorithms to create a complete frontend for speaker diarization in the meeting room domain. New techniques we are presenting include blind reference-channel selection, two-step time delay of arrival (TDOA) Viterbi postprocessing, and a dynamic output signal weighting algorithm, together with using such TDOA values in the diarization to complement the acoustic information. Tests on speaker diarization show a 25% relative improvement on the test set compared to using a single most centrally located microphone. Additional experimental results show improvements using these techniques in a speech recognition task.

When performing speaker diarization, it is common to use an agglomerative clustering approach where the acoustic data is first split in small pieces and then pairs are merged until reaching a stopping point. When using a purely agglomerative clustering technique, one cluster cannot be split into two. Therefore, errors caused by multiple speakers being assigned to one cluster can be common. Furthermore, clusters often contain non-speech frames, creating problems when deciding which two clusters to merge and when to stop the clustering. In this paper, we present two algorithms that aim to purify the clusters. The first assigns conflicting speech segments to a new cluster, and the second detects and eliminates non-speech frames when comparing two clusters. We show improvements of over 18% relative using three datasets from the most current rich transcription (RT) evaluations

Reliable measures of speaker positions are needed for computational perception of human activities taking place in a smartroom environment. In this work, it is described the development process and the experiments conducted in the design and implementation of an Audio Person Tracking system for smart-room environments. The proposed system is based on the SRP-PHAT algorithm, as it is known to perform robustly in most environmental conditions. Novelties proposed are aimed to enhance the accuracy of the system independently on the application scenario and to reduce the computational complexity. Index Terms: sound localization, source tracking, microphone arrays.

The task of speaker diarization consists of answering the question “Who spoke when?”. The most commonly used approach to speaker diarization is agglomerative clustering of multiple initial clusters. Even though the initial clustering is greatly modified by iterative cluster merging and possibly multiple resegmentations of the data, the initialization algorithm is a key module for system performance and robustness. In this paper we present a novel approach that obtains a desired initial number of clusters in three steps. It first computes possible speaker change points via a standard technique based on the Bayesian information criterion (BIC). It then classifies the resulting segments into ”friend” and ”enemy” groups to finally creates an initial set of clusters for the system. We test this algorithm with the dataset used in the RT05s evaluation, where we show a 13% Diarization error rate relative improvement and a 2.5% absolute cluster purity improvement with respect to our previous algorithm. Index Terms: Speaker diarization, speaker segmentation and clustering, clusters initialization, meetings indexing.

Jacobian Adaptation (JA) of the acoustic models is a fast adaptation technique that has been successfully used in both speech and speaker recognition systems. This technique adapts the acoustic models on the basis of the difference between the testing and the training noise conditions. For this reason, a noise reference of both the training and the testing phase is needed. In previous works, the noise conditions have been commonly supposed to be known, or estimated from the first part of the signals or using manually obtained speech and non-speech labels. However, in a real application the noise conditions are not generally known, it is not sure that the first part of the signal does not contain speech and manual labels are not usually available. In this work we propose to obtain the noise references by using continuous noise estimation methods, which are more appropriate for real applications. Several noise estimation methods are compared and the obtained results show that these techniques are effective for JA.

Comunicació presentada a: 9th International Conference on Spoken Language Processing; 17-21 de setembre de 2006 a Pittsburgh, Estats Units d'Amèrica

To realize the long-term goal of ubiquitous computing, technological advances in multi-channel acoustic analysis are needed in order to solve several basic problems, including speaker localization and tracking, speech activity detection (SAD) and distant-talking automatic speech recognition (ASR). The European Commission integrated project CHIL, “ Computers in the Human Interaction Loop”, aims to make significant advances in these three technologies. In this work, we report the results of our initial automatic source localization, speech activity detection, and speech recognition experiments on the CHIL seminar corpus, which is comprised of spontaneous speech collected by both near- and far-field microphones. In addition to the audio sensors, the seminars were also recorded by calibrated video cameras. This simultaneous audio-visual data capture enables the realistic evaluation of component technologies as was never possible with earlier data bases.

Reliable measures of speaker positions are needed for computational perception of human activities taking place in a smart-room environment. In this work, we investigate the effect of talkers head orientation on the accuracy of acoustical source localization techniques and its relation with the talker directivity pattern and room reverberation. Two different representative speaker localization techniques are assessed, steered response power and a crossing lines based method, in both cases on the basis of the estimated delays between pairs of microphones with the GCC-PHAT algorithm. A small database has been collected at the UPC’s smart room for evaluation. The results show how the localization error heavily depends on the head orientation, and also the fact that the space exploration based technique is much more robust to head orientation changes than the crossing lines technique, due to the way the contributions from the various microphones are combined.

In this paper, we propose some novel normalization and fusion techniques for biometric matching score level fusion in person verification. While conventional matching score level fusion methods use global score statistics, we consider in this work both genuine and impostor statistics separately. Performing a joint mean normalization of the separate monomodal scores, multimodal scores with less separate variance than the monomodal ones are obtained. Furthermore, a weighting method has been designed in order to minimize the variance sum of the separate multimodal statistics. This method obtains a minor sum of genuine and impostor variances for the multimodal biometric than that of the monomodal ones. The results obtained in speech and face scores fusion upon POLYCOST and XM2VTS databases show that the proposed normalization and fusion techniques provide better results than the conventional methods.

A new method to detect words that are likely to be confused by speech recognition systems is presented in this letter. A new dissimilarity measure between two words is calculated in two steps. First, the phonetic transcriptions of the words are aligned using only phonetic information. Two kinds of alignments are used: either with or without insertions and deletions. Second, the dissimilarity measure is calculated on the basis of the resulting alignment and acoustic information obtained from the hidden Markov models of the phones. In a classical false acceptance/false rejection framework, the equal error rate was measured to be less than 5%.

A robust adaptive beamforming method is presented in this paper for speech enhancement and speech recognition with microphone arrays. The proposal is based on a modification of the Generalized Sidelobe Canceller with adaptive blocking matrix and the use of a Wiener filter. Alternatively to most of the previous reported works based on microphone arrays with postfiltering, the new technique integrates the Wiener filter in the structure of the adaptive beamformer in a single stage. Experimental results show that the proposed integrated adaptive Wiener-filtering (IAW) beamformer usually is more robust to directional and ambient noises than conventional postfiltering of the beamformer output with a lower level of degradation. Speech recognition experiments which show improvements with the proposed beamformer are also reported. 1.

The amount of data usually determines the robustness of speaker models in speaker recognition. In this sense, it is convenient to set a model quality measure for every speaker and to classify models into different categories according to their quality level. We propose a new quality measure, which uses only data from clients, based on the number of training utterances that surpasses a certain threshold. If the desired quality is not high enough, the quality measure allows for the detection of non-representative utterances. Once selected, these utterances, considered as outliers, can be removed or better replaced by new ones coming from the same speaker. A database of 184 speakers in Spanish is used to obtain empirical results with connected digits. Our experiments removing outliers and replacing them by new utterances coming from the same speaker outperform the baseline experiments by 40%.

Jacobian Adaptation (JA) of the acoustic models is an efficient adaptation technique for robust speech recognition. Several improvements for the JA have been proposed in the last years, either to generalize the Jacobian linear transformation for the case of large noise mismatch between training and testing or to extend the adaptation to other degrading factors, like channel distortion and vocal tract length. However, the JA technique has only been used so far with the conventional mel-frequency cepstral coefficients (MFCC). In this paper, the JA technique is applied to an alternative type of features, the Frequency-Filtered (FF) spectral energies, resulting in a more computationally efficient approach. Furthermore, in experimental tests with the database Aurora1, this new approach has shown an improved recognition performance with respect to the Jacobian adaptation with MFCCs.

Decomposition of speech signals into simultaneous streams of periodic and aperiodic information has been successfully applied to speech analysis, enhancement, modification and recently recognition. This paper examines the effect of different weightings of the two streams in a conventional HMM system in digit recognition tests on the Aurora 2.0 database. Comparison of the results from using matched weights during training showed a small improvement of approximately 10% relative to unmatched ones, under clean test conditions. Principal component analysis of the covariation amongst the periodic and aperiodic features indicated that only 45 (51) of the 78 coefficients were required to account for 99% of the variance, for clean (multi-condition) training, which yielded an 18.4% (10.3%) absolute increase in accuracy with respect to the baseline. These findings provide further evidence of the potential for harmonically-decomposed streams to improve performance and substantially to enhance recognition accuracy in noise.

speech signal has shown to be attractive for noisy speech recognition because of both its high recognition performance with respect to the conventional LP in severe conditions of additive broad-band noise and its computational simplicity. However, the mel-cepstrum representation, which comes from a filter bank (FB) analysis, has become increasingly popular. Furthermore, hybrid techniques that combine filter-bank and LP analysis, have also been proposed. The aim of this paper is to gain some perspective of the merit of the autocorrelation, LP and FB based techniques in a real environment. 1.

In current speech recognition systems, speech is represented by a 2-D sequence of parameters that model the temporal evolution of the spectral envelope of speech. Linear transformation or filtering along both time and frequency axes of that 2-D sequence are used to enhance the discriminative ability and robustness of speech parameters in the HMM pattern-matching formalism. In this paper, we compared two recently reported approaches which operate on the sequence of logarithmically compressed mel-scaled filter-bank energies: the first approach - TIFFING (TIme and Frequency FilterING) - applies FIR filters to that 2-D sequence along both axes, while the second one - CTM (Cepstral Time Matrix) - uses the DCT to compute a set of parameters in the 2-D transformed domain. They are compared in several ways: (1) analytically, using Fourier transformation, (2) statistically and (3) performing recognition tests with clean and noisy speech.

The spectral parameters that result from filtering the frequency sequence of log mel-scaled filter-bank energies with a first or second order FIR filter have proved to be competitive for speech recognition. Recently, the authors have shown that this frequency filtering can approximately equalize the cepstrum variance enhancing the oscillations of the spectral envelope curve that are most effective for discrimination between speakers. Even better speaker identification results than using mel-cepstrum were observed on the TIMIT database, especially when white noise was added. In this paper, the hybridization of both linear prediction and filter-bank spectral analysis using either cepstral transformation or the alternative frequency filtering is explored for speaker verification. This combination, that had shown to be able to outperform the conventional techniques in clean and noisy word recognition, has yield good text-dependent speaker verification results on the new speaker-oriented telephone-line POL YCOST database.

Speech dynamic features are routinely used in current speech recognition systems in combination with short-term (static) spectral features. Although many existing speech recognition systems do not weight both kinds of features, it seems convenient to use some weighting in order to increase the recognition accuracy of the system. In the cases that this weighting is performed, it is manually tuned or it consists simply in compensating the variances. The aim of this paper is to propose a method to automatically estimate an optimum state-dependent stream weighting in a continuous density hidden Markov model (CDHMM) recognition system by means of a maximum-likelihood based training algorithm. Unlike other works, it is shown that simple constraints on the new weighting parameters permit to apply the maximum-likelihood criterion to this problem. Experimental results in speaker independent digit recognition show an important increase of recognition accuracy.

In this paper, two ways of obtaining more robust spectral parameters are explored. Firstly, an hybridization of both LP and filter-bank approaches is considered, which is capable of improving recognition results for both noisy and clean speech in CDHMM digit recognition. Secondly, better performance may also be achieved by replacing the cepstral coefficients by a recently proposed set of parameters located in the frequency domain which come from a simple filtering of the log band energies.

Recently, the set of spectral parameters of every speech frame that result from filtering the frequency sequence of mel-scaled filter-bank energies with a simple first-order high-pass FIR filter have proved to be an efficient speech representation in terms of both speech recognition rate and computational load. In this paper, we apply the same technique to speaker recognition. Frequency filtering approximately equalizes the cepstrum variance, enhancing the oscillations of the spectral envelope curve that are most effective for discriminating between speakers. In this way, even better speaker identification results than using conventional mel-cepstrum were observed in continuous observation Gaussian density HMM, especially in noisy conditions.

The article presents a robust representation of speech based on AR modeling of the causal part of the autocorrelation sequence. In noisy speech recognition, this new representation achieves better results than several other related techniques.

Cepstral coefficients are widely used in speech recognition. In this paper, we claim that they are not the best way of representing the spectral envelope, at least for some usual speech recognition systems. In fact, cepstrum has several disadvantages: poor physical meaning, need of transformation, and low capacity of adaptation to some recognition systems. In this paper, we propose a new representation that significantly outperforms both mel-cepstrum and LPC-cepstrum techniques in both recognition rate and computational cost. It consists of filtering the frequency sequence of filter-bank energies with an extremely simple filter that equalizes the variance of the cepstral coefficients. Excellent results of the new technique using a continuous observation density HMM recognition system and two very different recognition tasks, connected digits and phone recognition, are presented. 1.

We study some speech enhancement algorithms based on the iterative Wiener filtering method due to Lim-Oppenheim [2], where the AR spectral estimation of the speech is carried out using a second-order analysis. But in our algorithms we consider an AR estimation by means of cumulant analysis. This work extends some preceding papers due to the authors, where information of previous speech frames is taken to initiate speech AR modelling of the current frame. Two parameters are introduced to dessign Wiener filter at first iteration of this iterative algorithm. These parameters are the Interframe Factor (IF) and the Previous Frame Iteration (PFI). A detailed study of them shows they allow a very important noise suppression after processing only first iteration of this algorithm, without any appreciable increase of distortion. Finally, the simplest cumulant-based algorithm is applied to Speech Recognition and some preliminary results are presented.

The performance of the existing speech recognition systems degrades rapidly in the presence of background noise. The OSALPC (one-sided autocorrelation linear predictive coding) representation of the speech signal has shown to be attractive for speech recognition because of its simplicity and its high recognition performance with respect to the standard LPC in severe conditions of additive white noise. The aim of this paper is twofold: (1) to show that OSALPC also achieves good performance in a case of real noisy speech (in a car environment), and (2) to explore its combination with several robust similarity measuring techniques, showing that its performance improves by using cepstral liftering, dynamic features and multilabeling.>

The OSALPC (One-Sided Autocorrelation Linear Predictive Coding) representation of the speech signal has shown to be attractive for speech recognition because of its simplicity and its high recognition performance with respect to the standard LPC in severe noisy conditions. In this paper the OSALPC technique is applied to the problem of speaker identification in noisy conditions. As shown with experimental results, using additive white noise, that technique also achieves much better results than both LPC and mel-cepstrum parameterizations in this task.

Some Speech Enhancement algorithms based on the iterative Wiener filtering Method due to L1m-Oppenheim [2] are presented. In the original Lim-Oppenheim algorithm, speech AR estimation is carried out using classic second-order analysis, but our algorithms consider a more robust AR modelling. Two different strategies of speech AR estimation are presented and both estimators are trying to see as less amount of noise as possible. First one uses a previous One-Sided Autocorrelation computation, that is a pole-preserving function, and the actual SNR m the second-order LPC analysis is increased. Second one combines advantages of Higher-Order Statistics [1] with a linear combination of AR coefficients, belonging to two consecutive overlapped frames, to assess a less disturbed speech estimation.

The performance of existing speech recognition systems degrades rapidly in the presence of background noise when training and testing cannot be done under the same ambient conditions. The aim of this paper is to propose the application of a simple multilabeling method, instead of the standard vector quantization -so called labeling-, as the front end for a speech recognizer based on the Vector Quantization (VQ) and Hidden Markov Models (HMM) approaches in order to increase its robustness to noise. Furthermore, not only cepstrum but also other features such as energy and dynamic parameters are evaluated and quantized independently in the multilabeling stage to represent more accurately characteristics of speech. The result of this process is a multiple multilabeling. Experimental results in the presence of additive white noise and car environment clearly demonstrate its good performance in isolated word recognition in noisy environments.

Speech recognition in noisy environments remains an unsolved problem even in the case of isolated word recognition with small vocabularies. Recently, several techniques have been proposed to alleviate this problem. Concretely, two closely related parameterization techniques based on an AR modelling in the autocorrelation domain called SMC [1] and OSALPC [2] have shown good results using speech contaminated by additive white noise. The aim of this paper is twofold: to compare several techniques based on an AR modelling in the autocorrelation domain, including SMC and OSALPC, and to find the optimum model order and cepstral liftering for noisy conditions.

A novel cepstral function, the cepstrum (CEP) of the one-sided autocorrelation sequence (COSA), is presented and applied to pitch determination of speech signals. This pitch determination algorithm (PDA) starts from the autocorrelation sequence in lieu of the speech signal. Although the COSA pitch determination algorithm does not improve the performance of the autocorrelation-with-center-clipping and CEP algorithms in quasiperiodic speech frames, it significantly reduces their pitch-period errors at transitional speech segments as well as in speech signals contaminated by noise. The PDA's better performance is based on its accuracy at nonstationary segments of speech signals and its noise capability.>

Peer Reviewed