As a young and emerging field in social human–robot interaction (HRI), semantic-free utterances (SFUs) research has been receiving attention over the last decade. SFUs are an auditory interaction means for machines that allow emotion and intent expression, which are composed of vocalizations and sounds without semantic content or language dependence. Currently, SFUs are most commonly utilized in animation movies (e.g., R2-D2, WALL-E, Despicable Me), cartoons (e.g., “Teletubbies,” “Morph,” “La Linea”), and computer games (e.g., The Sims) and hold significant potential for applications in HRI. SFUs are categorized under four general types: Gibberish Speech (GS), Non-Linguistic Utterances (NLUs), Musical Utterances (MU), and Paralinguistic Utterances (PU). By introducing the concept of SFUs and bringing multiple sets of studies in social HRI that have never been analyzed jointly before, this article addresses the need for a comprehensive study of the existing literature for SFUs. It outlines the current grand challenges, open questions, and provides guidelines for future researchers considering to utilize SFU in social HRI.

We present a framework for combination aware AU intensity recognition. It includes a feature extraction approach that can handle small head movements which does not require face alignment. A three layered structure is used for the AU classification. The first layer is dedicated to independent AU recognition, and the second layer incorporates AU combination knowledge. At a third layer, AU dynamics are handled based on variable duration semi-Markov model. The first two layers are modeled using extreme learning machines (ELMs). ELMs have equal performance to support vector machines but are computationally more efficient, and can handle multi-class classification directly. Moreover, they include feature selection via manifold regularization. We show that the proposed layered classification scheme can improve results by considering AU combinations as well as intensity recognition.

This paper studies body-conducted speech for noise robust speech processing purposes. As body-conducted speech is typically limited in bandwidth, signal processing is required to obtain a signal that is both high in quality and low in noise. We propose an algorithm that first equalizes the body-conducted speech using filters obtained from a pre-defined filter set and subsequently fuses this equalized signal with a noisy conventional microphone signal using an optimal clean speech amplitude and phase estimator. We evaluated the proposed equalization and fusion technique using a combination of a conventional close-talk and a throat microphone. Subjective listening tests show that the proposed method successfully fuses the speech quality of the conventional signal and the noise robustness of the throat microphone signal. The listening tests also indicate that the inclusion of the body-conducted signal can improve single-channel speech enhancement methods, while a calculated set of objective signal quality measures confirm these observations.

A common approach in visual speech synthesis is the use of visemes as atomic units of speech. In this paper, phonemebased and viseme-based audiovisual speech synthesis techniques are compared in order to explore the balancing between data availability and an improved audiovisual coherence for synthesis optimization. A technique for automatic viseme clustering is described and it is compared to the standardized viseme set described in MPEG-4. Both objective and subjective testing indicated that a phoneme-based approach leads to better synthesis results. In addition, the test results improve when more different visemes are defined. This raises some questions on the widely applied viseme-based approach. It appears that a many-to-one phoneme-to-viseme mapping is not capable of describing all subtle details of the visual speech information. In addition, with viseme-based synthesis the perceived synthesis quality is affected by the loss of audiovisual coherence in the synthetic speech.

In this paper, we propose an approach to convert acoustic speech to video realistic mouth animation based on an articulatory dynamic Bayesian network model with constrained asynchrony (AF_AVDBN). Conditional probability distributions are defined to control the asynchronies between the articulators such as lips, tongue and glottis/velum. An EM-based conversion algorithm is also presented to learn the optimal visual features given an auditory input and the trained AF_AVDBN parameters. In the training of the AF_AVDBN models, downsampled YUV spatial frequency features of the interpolated mouth image sequences are extracted as visual features. For reproducing the mouth animation sequence, from the learned visual features, a spatial upsampling and a temporal downsampling are applied. Both qualitative and quantitative results show that the proposed method is capable of producing more natural and realistic mouth animations, and the accuracy is further improved compared to the state of the art multi-stream Hidden Markov Model (MSHMM) and articulatory DBN model without asynchrony constraint (AF_DBN).

We propose an audiovisual source separation algorithm for speech signals. In our proposed algorithm we first extract the time segments with low activity of the mouth region from synchronous video recordings. An automatically selected optimal classifier is used to detect silent intervals in these instants of low visual mouth activity. Then, the source separation problem is formulated and solved for the entire signal duration. Our approach was tested on two challenging speech corpora with two speakers and two microphones, namely in the first corpus separate source signals were mixed in a simulated room, and the second corpus contains recorded conversations. The results are promising on both corpora: with the visual silence detector the performance of the source separation algorithm, measured by the signal to noise inference ratio increases.

This paper presents a mouth animation construction method based on the DBN models with articulatory features (AF_AVDBN), in which the articulatory features of lips, tongue, glottis/velum can be asynchronous within a maximum asynchrony constraint to describe the speech production process more reasonably. Given an audio input and the trained AF_AVDBN models, the optimal visual feature learning algorithm is deduced based on the Maximum Likelihood Estimation criterion. The learned visual features are then used to construct the mouth images for the input speech. Objective and subjective evaluations on the mouth animations of 110 speech sentences show that the learned visual features from the AF_AVDBN models track the real visual features very closely, and the constructed mouth images from the AF_AVDBN models are very much like the real ones.

We present a harmonic-plus-noise modelling (HNM) strategy in the context of corpus-based text-to-speech (TTS) synthesis, in which whole speech phonemes are modelled in their integrity, contrary to the traditional frame-based approach. The pitch and amplitude trajectories of each phoneme are modelled with a low-order DCT expansion. The parameter analysis algorithm is to a large extent aided and guided by the pitch contours, and by the phonetic annotation and segmentation information that is available in any TTS system. The major advantages of our model are: few parameter interpolation points during synthesis (one per phoneme), flexible time and pitch modifications, and a reduction in the number of model parameters which is favourable for low bit rate coding in TTS for embedded applications. Listening tests on TTS sentences have shown that very natural speech can be obtained, despite the compactness of the signal representation.

In this paper we present a study of speech pauses at three different speaking rates, based on the analysis of four hours of read speech in six European languages. Our results confirm earlier observations that the logarithmic duration of the pauses can be well approximated by a bi-Gaussian distribution. We found this also to be true at slow and fast speaking rates. Our analysis further shows that, as far as the long speech pauses are concerned, similar strategies are used in all languages considered. For speaking slowly, speakers increase the total amount of pauses and they use a wider range of pause durations. Overall, there appeared to be no striking change in the average, nor in the variance of the distribution of the pause durations. For speaking rapidly, speakers decrease the amount of pauses used and they refrain from using the longest pauses that occur in their normal speech. Overall, this results in a lower average and a smaller variance of the pause durations.

Signal requantization to reduce the word-length of an audio stream introduces distortions. Noise shaping can be applied in combination with a psychoacoustic model in order to make requantization distortions minimally audible. The psychoacoustically optimal noise shaping curve depends on the time-varying characteristics of the input signal. Therefore, the noise shaping filter coefficients are to be computed and updated on a regular basis. In this paper, we present a least squares theory for optimal noise shaping of audio signals. It provides a shorter and more straightforward proof of known properties, and in contrast with the standard theory, it does show how noise shaping filters that attain the theoretical optimum can be designed in practice.

Signal requantization to reduce the word-length of an audio stream introduces distortions. Noise shaping can be applied in combination with a psychoacoustic model in order to make requantization distortions minimally audible. The psychoacoustically optimal noise shaping curve depends on the time-varying characteristics of the input signal. Therefore, the noise shaping filter coefficients are to be computed and updated on a regular basis. In this paper, we present a least squares theory for optimal noise shaping of audio signals. It provides shorter and more straightforward proof of known properties, and in contrast with the standard theory, it does show how noise shaping filters that attain the theoretical optimum can be designed in practice.

While a traditional sinusoidal model is capable of representing audio segments, a sum of exponentially damped sinusoids is more efficient to model the transient segments that are readily found in audio signals. In this paper, Total Least Squares (TLS) algorithms are applied to automatically extract the modeling parameters in the Exponential Sinusoidal Model (ESM). In order to turn the SNR . optimization criterion of these TLS algorithms into a perceptual modeling strategy we incorporate the psycho-acoustic model of MPEG 1 - Layer 1 into a subband TLS-ESM scheme. This allows us to model each subband in accordance with its perceptual relevance. Informal listening tests confirm that perceptual ESM achieves the same perceived quality as plain ESM while using substantially less components.

Signal Subspace (SS) based speech enhancement techniques obtain significant additive-noise reduction by altering the singular value spectrum of the speech observation matrix. Among the class of different possible SS weighting strategies, the Minimum Variance (MV) estimation method substantially increases the speech recognition accuracy in additive noise environments, outperforming the widely used Spectral Subtraction methods. However, these SS approaches are developed as pure speech enhancement techniques, and it is still unknown how effective they are for noise robust speech recognition. In this respect, we present the idea of `optimal SS weighting' for speech recognition systems, and we illustrate in detail that the MV estimation closely approximates this optimum. We applied the SS weighting methods to a LV-CSR task with noisy data (10 dB SNR), and obtained relative reductions in Word Error Rate of more than 60%.

We describe how Total Least Squares (TLS) algorithms can be applied as a powerful and efficient modelling tool for wideband speech. A detailed description in both time domain and frequency domain illustrates how the modelling functions - damped sinusoids - naturally synthesise non-stationary signals. Straightforward implementations of TLS applied to fullband speech are known to be computationally hard and they can suffer from numerical sensitivity. In this paper we introduce a subband approach, which leads to a significant reduction of the computational load with an enhanced numerical stability. Moreover, it enables to control the distribution of the TLS components over the spectral range of the input signal such that perceptual criteria can be incorporated in the modelling scheme. We also address the scalability of our design from smallband speech to high quality audio, and provide evidence for the existence of coupled components in TLS modelled segments.

The paper considers a prototype for automatic postsynchronization that consists of two basic components. As a first step, dynamic time warping is applied to compute the time-correspondence between an original utterance and an utterance that serves as the timing reference signal. In a second step, a time-scaling algorithm modifies the time structure of the original utterance accordingly. Informal diagnostic evaluation has shown that good results are obtained if the similarity between the acoustic-phonetic contents of the utterances is high. Possible ways for improving robustness against acoustic-phonetic differences, such as those that result from different coarticulation, are suggested.

Given a set of utterances that have been spoken by two speakers, the problem of voice conversion consists in constructing some sort of transformation that can be applied on any utterance of the first speaker and makes the result appear to have been spoken by the second speaker. Conversion rules for the vocal tract spectral characteristics, which are usually trained on individual subsets of a partition of the spectral feature space, still do not reach the desired level of accuracy. The paper presents an analysis of the problem.

A synchronization criterion for overlap-add time-scale modification is derived through a least squares estimation of the modified short-time Fourier transform. Based on this finding, a structural time-domain framework for time-scale modification is described. One efficient variant, which was called the Waveform Similarity based Overlap-Add (WSOLA) method, produces high quality output when applied to speech, but can even be applied successfully to a broader class of signals, including multiple voices together and musical instruments. Fine-tuning the synchronization criterion, without affecting the high quality that is obtained, can make the computational cost very low, revealing the versatile possibilities for on-line operation.

It is shown that a recently proposed technique for high-quality waveform manipulation can be formulated as a pitch-excited vocoder. This waveform vocoder produces high-quality speech over a wide range of prosodic modifications, showing that natural sounding speech can be produced using an impulse driven linear synthesis model. In a pilot experiment, waveform vocoding techniques were applied on the LPC (linear predictive coding) residue to investigate the relative importance of amplitude and phase in the synthesis of male and female voices. It was found that the amplitude information contributed more to speech quality than phase information, and that, for male voices, amplitude information alone was sufficient to make the synthetic speech quality almost indistinguishable from that of natural speech.>

A brief review of the basics of cepstral deconvolution is followed by a refined model for short-time cepstra of voiced speech. The model provides a better understanding of the nature of short-time cepstra and of the heuristics of deconvolution. It brings to light a number of properties which are important for the successful application of short-time cepstral deconvolution to voiced speech. It allows a set of rules to be derived which can be used for the optimization of cepstral deconvolution systems.>

In its nature, the autocorrelation function (ACF) is well suited to periodicity detection. As exemplified by the SIFT algorithm, and discussed by Sondhi in a classic paper, the success of ACF pitch detection is very much dependent on the availability of an efficient preprocessor. Often, the combination of low pass Filtering and Uniform Sign Clipping (FUSC) is chosen as a preprocessor because it yields an algorithm which is easily implemented in real-time. A disadvantage of using FUSC is that one does not obtain a reliable V-UV decision from the ACF alone. Also, FUSC algorithms often fail when successive pitch pulses vary widely in amplitude (at the onset and termination of voicing, or when the vocal cords of the speaker vibrate anomalously). Adaptive Non-Uniform Sign Clipping (ANUSC) is introduced as a means of coping with these problems while achieving the same computational advantages as with FUSC.

In speech synthesis, a digital filter is used to generate samples of the acoustic speech waveform. Because of the time-varying nature of speech, filter parameters are updated at frame boundaries. This results in a transient perturbation of synthesized speech whenever parameter values exhibit a large frame-to-frame variation. From the computational point of view, such perturbations are caused by a mismatch which is introduced between the updated filter parameters on the one hand, and the energy contained in the autoregressive part of filtermemory on the other. We propose to use a modified-superposition synthesis strategy, which allows the contribution of this energy as it dies away in the present frame to be computed from the non-updated parameterset. In this way, essentially transient-free speech is obtained. Applications include a robust cascade-type synthesizer for formant vocoding which is immune to formant labeling errors.