The identity design of artificial agents carries growing ethical, psychological, and cultural weight, as ubiquitous language models and diverse robotic forms are blended into everyday use. However, structured approaches to designing coherent and interpretable artificial identities remain limited. To address urgent challenges in artificial identity design, including harmful stereotypes and deceptive practices, we introduce the Identity Design (ID) Framework and an accompanying research agenda. Drawing on emerging work on artificial identity in human-robot interaction and taking an interdisciplinary perspective, we propose twelve design principles across three levels: individual (recognisability, behavioural consistency, identity continuity, memory, persistent goals), group (membership signalling, social alignment, role clarity), and societal (benevolence, artificiality, social justice, transparency). The research agenda outlines open questions around the operationalisation and measurement of identity, social dynamics, and ethical considerations for identity design. Together, they lay the groundwork for future research and responsible practice in robotic, virtual, and multi-embodied agents.

As social robots become integral to diverse interaction scenarios, their design must effectively engage users emotionally while accurately conveying their functional capabilities through coherent design. While previous research in adaptive human-robot interaction has predominantly focused on enabling robots to learn and adapt to users' dynamic behavior, less attention has been paid to robots' affordances. This study addresses this gap by investigating how users' preferences for a social robot's affordances, such as look and voice, vary across different social roles and scenarios. Using the Stereotype Content Model's dimensions of warmth and competence, we conducted a quantitative survey with Likert-scale measures to assess preferences for affordance designs in general and context-dependent use cases. Our results show a medium to low preference for humanlike affordances when the robot's intended use is unspecified. While human likeness positively correlates with perceived warmth and competence, these social perceptions do not significantly drive preferences for humanlike features. Instead, preferences are primarily influenced by robots' intended situational roles, which differ from the stereotypical occupational roles in human society. These results highlight the importance of context-specific adaptive affordance design, which should focus on aligning robot characteristics with contextual expectations of social attributes.

Communication impairments are prevalent among a significant proportion of individuals. Methods of Augmentative and Alternative Communication (AAC) can support people with speech disorders (PwSD) to some extent, but AAC users encounter substantial difficulties when engaging in open-domain social interactions, especially involving multiple participants. This is mainly due to the significant communication rate gap between typical speakers and AAC users. Large Language Models (LLM) offer a solution by providing predictions of the next words or sentences. This work analyses refining the prediction capabilities of Masked Language Models (MLM) for AAC users by performing layer-wise analysis specifically for word prediction on an AAC corpus. Experiments show that fine-tuning only specific low-performing LLM layers leads to better results than fine-tuning of the entire model. Fine-tuning of specific layers of a Robust Bidirectional Encoder Representations from Transformers (RoBERTa) model outperforms other tested models; for qualitative evaluation and informal prototype AAC device testing. Fine-tuning the word predictions in an AAC context results in approx. 20% increase in average communication rate (across different communication scenarios) to input speed of approx. 30 words per minute (WPM).

Every six years (since 1997), a survey has been conducted at the IEEE workshop on Automatic Speech Recognition and Understanding (ASRU). The aim has been to gain an insight into the research community's perspective on the 'progress and prospects' for spoken language technology. These surveys have been based on a set of 'statements' describing possible scenarios, and respondents are asked to estimate the year (in the future or in the past) when each given scenario might be realised. A number of the statements have appeared in multiple surveys, hence it has been possible to track changes in opinion over time. This paper presents the combined results from five such surveys, the most recent of which was conducted at ASRU-2021. The latest results reveal a dramatic increase in optimism.

With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot’s current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with.

Generative Adversarial Networks (GANs) are machine learning networks based around creating synthetic data. Voice Conversion (VC) is a subset of voice translation that involves translating the paralinguistic features of a source speaker to a target speaker while preserving the linguistic information. The aim of non-parallel conditional GANs for VC is to translate an acoustic speech feature sequence from one domain to another without the use of paired data. In the study reported here, we investigated the interpretability of state-of-the-art implementations of non-parallel GANs in the domain of VC. We show that the learned representations in the repeating layers of a particular GAN architecture remain close to their original random initialised parameters, demonstrating that it is the number of repeating layers that is more responsible for the quality of the output. We also analysed the learned representations of a model trained on one particular dataset when used during transfer learning on another dataset. This showed extremely high levels of similarity across the entire network. Together, these results provide new insight into how the learned representations of deep generative networks change during learning and the importance in the number of layers.

Speech emotion recognition is essential for obtaining emotional intelligence which affects the understanding of context and meaning of speech. The fundamental challenges of speech emotion recognition from a machine learning standpoint is to extract patterns which carry maximum correlation with the emotion information encoded in this signal, and to be as insensitive as possible to other types of information carried by speech. In this paper, a novel recurrent residual temporal context modelling framework is proposed. The framework includes mixture of multi-view attention smoothing and high dimensional feature projection for context expansion and learning feature representations. The framework is designed to be robust to changes in speaker and other distortions, and it provides state-of-the-art results for speech emotion recognition. Performance of the proposed approach is compared with a wide range of current architectures in a standard 4-class classification task on the widely used IEMOCAP corpus. A significant improvement of 4% unweighted accuracy over state-of-the-art systems is observed. Additionally, the attention vectors have been aligned with the input segments and plotted at two different attention levels to demonstrate the effectiveness.

Speech emotion recognition (SER) is a requisite for emotional intelligence that affects the understanding of speech. One of the most crucial tasks is to obtain patterns having a maximum correlation for the emotion classification task from the speech signal while being invariant to the changes in frequency, time and other external distortions. Therefore, learning emotional contextual feature representation independent of speaker and environment is essential. In this paper, a novel spatiotemporal context modelling framework for robust SER is proposed to learn feature representation by using acoustic context expansion with high dimensional feature projection. The framework uses a deep convolutional neural network (CNN) and self-attention network. The CNNs combine spatiotemporal features. The attention network produces high dimensional task-specific features and combines these features for context modelling, which altogether provides a state-of-the-art technique for classifying the extracted patterns for speech emotion. Speech emotion is a categorical perception representing discrete sensory events. The proposed approach is compared with a wide range of architectures on the RAVDESS and IEMOCAP corpora for 8-class and 4-class emotion classification tasks and remarkable gain over state-of-the-art systems are obtained, absolutely 15%, 10% respectively.

Emotion recognition from speech plays a significant role in adding emotional intelligence to machines and making human-machine interaction more natural. One of the key challenges from machine learning standpoint is to extract patterns which bear maximum correlation with the emotion information encoded in this signal while being as insensitive as possible to other types of information carried by speech. In this paper, we propose a novel temporal modelling framework for robust emotion classification using bidirectional long short-term memory network (BLSTM), CNN and Capsule networks. The BLSTM deals with the temporal dynamics of the speech signal by effectively representing forward/backward contextual information while the CNN along with the dynamic routing of the Capsule net learn temporal clusters which altogether provide a state-of-the-art technique for classifying the extracted patterns. The proposed approach was compared with a wide range of architectures on the FAU-Aibo and RAVDESS corpora and remarkable gain over state-of-the-art systems were obtained. For FAO-Aibo and RAVDESS 77.6% and 56.2% accuracy was achieved, respectively, which is 3% and 14% (absolute) higher than the best-reported result for the respective tasks.

The term 'phoneme' lies at the heart of speech science and technology, and yet it is not clear that the research community fully appreciates its meaning and implications. In particular, it is suspected that many researchers use the term in a casual sense to refer to the sounds of speech, rather than as a well defined abstract concept. If true, this means that some sections of the community may be missing an opportunity to understand and exploit the implications of this important psychological phenomenon. Here we review the correct meaning of the term 'phoneme' and report the results of an investigation into its use/misuse in the accepted papers at INTERSPEECH-2018. It is confirmed that a significant proportion of the community (i) may not be aware of the critical difference between `phonetic' and 'phonemic' levels of description, (ii) may not fully understand the significance of 'phonemic contrast', and as a consequence, (iii) consistently misuse the term 'phoneme'. These findings are discussed, and recommendations are made as to how this situation might be mitigated.

Despite increasing awareness of Alexa’s potential as an educational tool, there remains a limited scope for Alexa skills to accommodate the features required for effective language learning. This paper describes an investigation into implementing ‘spaced-repetition’, a non-trivial feature of flashcard-based learning, through the development of an Alexa skill called ‘Japanese Flashcards’. Here we show that existing Alexa development features such as skill persistence allow for the effective implementation of spaced-repetition and suggest a heuristic adaptation of the spaced-repetition model that is appropriate for use with voice assistants (VAs). We also highlight areas of the Alexa development process that limit the facilitation of language learning, namely the lack of multilingual speech recognition, and offer solutions to these current limitations. Overall, the investigation shows that Alexa can successfully facilitate simple L2-L1 flashcard-based language learning and highlights the potential for Alexa to be used as a sophisticated and effective language learning tool.

Vehicle logo recognition is an important part of vehicle identification in intelligent transportation systems. State-of-the-art vehicle logo recognition approaches use automatically learned features from Convolutional Neural Networks (CNNs). However, CNNs do not perform well when images are rotated and very noisy. This paper proposes an image recognition framework with a capsule network. A capsule is a group of neurons, whose length can represent the existence probability of an entity or part of an entity. The orientation of a capsule contains information about the instantiation parameters such as positions and orientations. Capsules are learned by a routing process, which is more effective than the pooling process in CNNs. This paper, for the first time, develops a capsule learning framework in the field of intelligent transportation systems. By testing with the largest publicly available vehicle logo dataset, the proposed framework gives a quick solution and achieves the highest accuracy (100%) on this dataset. The learning capsules have been tested with different image changes such as rotation and occlusion. Image degradations including blurring and noise effects are also considered, and the proposed framework has proven to be superior to CNNs.

Sign language is a visually oriented, natural, nonverbal communication medium. Having shared similar linguistic properties with its respective spoken language, it consists of a set of gestures, postures and facial expressions. Though, sign language is a mode of communication between deaf people, most other people do not know sign language interpretations. Therefore, it would be constructive if we can translate the sign postures artificially. In this paper, a capsule-based deep neural network sign posture translator for an American Sign Language (ASL) fingerspelling (posture), has been presented. The performance validation shows that the approach can successfully identify sign language, with accuracy like 99%. Unlike previous neural network approaches, which mainly used fine-tuning and transfer learning from pre-trained models, the developed capsule network architecture does not require a pre-trained model. The framework uses a capsule network with adaptive pooling which is the key to its high accuracy. The framework is not limited to sign language understanding, but it has scope for non-verbal communication in Human-Robot Interaction (HRI) also.

Studies on recognising unspoken speech with the use of electroencephalographic (EEG) signals vary in their designs. The participants are either asked to imagine unspoken speech within a specific time frame, or alternatively indicate the start and end of the imagined speech. Optimizing the length and training size of imagined speech is important to improve the rate and speed of recognizing unspoken speech in on-line applications. In this study, we recorded EEG data when the participants performed unspoken speech of five words using two technologies: (1) marking the start and end of the trial by using mouse clicks and (2) performing the imagination in a four-second fixed time window. Four classifiers were trained in all experiment parts: support vector machine, naive bayes, random forest, and linear discriminate analysis. The results show that the best time frame is 3.5-4 seconds length. Moreover, the increase in training size improve the average classification accuracy. However, this improvement becomes slight between 125-175 total training trials. The training data can be recorded in parts, however, the required training size should be increased to have better classification accuracy. In all analysis parts, random forest classifier shows better results among the other classifiers.

Emotions, and emotional expression, have a broad influence on social interactions and are thus a key factor to consider in developing social robots. This study examined the impact of life-like affective facial expressions, in the humanoid robot Zeno, on children’s behaviour and attitudes towards the robot. Results indicate that robot expressions have mixed effects depending on participant gender. Male participants interacting with a responsive facially expressive robot showed a positive affective response and indicated greater liking towards the robot, compared to those interacting with the same robot maintaining a neutral expression. Female participants showed no marked difference across the conditions. We discuss the broader implications of these findings in terms of gender differences in human–robot interaction, noting the importance of the gender appearance in robots (in this case, male) and in relation to advancing the understanding of how interactions with expressive robots could lead to task-appropriate symbiotic relationships.

As part of an ongoing research into extracting mission-critical information from Search and Rescue speech communications, a corpus of unscripted, goal-oriented, two-party spoken conversations has been designed and collected. The Sheffield Search and Rescue (SSAR) corpus comprises about 12 hours of data from 96 conversations by 24 native speakers of British English with a southern accent. Each conversation is about a collaborative task of exploring and estimating a simulated indoor environment. The task has carefully been designed to have a quantitative measure for the amount of exchanged information about the discourse subject. SSAR includes different layers of annotations which should be of interest to researchers in a wide range of human/human conversation understanding as well as automatic speech recognition. It also provides an amount of data for analysis of multiple parallel conversations around a single subject. The SSAR corpus is available for research purposes.

Copyright © 2017 ISCA. To help people who have lost their voice following total laryngectomy, we present a speech restoration system that produces audible speech from articulator movement. The speech articulators are monitored by sensing changes in magnetic field caused by movements of small magnets attached to the lips and tongue. Then, articulator movement is mapped to a sequence of speech parameter vectors using a transformation learned from simultaneous recordings of speech and articulatory data. In this work, this transformation is performed using a type of recurrent neural network (RNN) with fixed latency, which is suitable for realtime processing. The system is evaluated on a phoneticallyrich database with simultaneous recordings of speech and articulatory data made by non-impaired subjects. Experimental results show that our RNN-based mapping obtains more accurate speech reconstructions (evaluated using objective quality metrics and a listening test) than articulatory-to-acoustic mappings using Gaussian mixture models (GMMs) or deep neural networks (DNNs). Moreover, our fixed-latency RNN architecture provides comparable performance to an utterance-level batch mapping using bidirectional RNNs (BiRNNs).

This paper describes a technique that generates speech acoustics from articulator movements. Our motivation is to help people who can no longer speak following laryngectomy, a procedure that is carried out tens of thousands of times per year in the Western world. Our method for sensing articulator movement, permanent magnetic articulography, relies on small, unobtrusive magnets attached to the lips and tongue. Changes in magnetic field caused by magnet movements are sensed and form the input to a process that is trained to estimate speech acoustics. In the experiments reported here this “Direct Synthesis” technique is developed for normal speakers, with glued-on magnets, allowing us to train with parallel sensor and acoustic data. We describe three machine learning techniques for this task, based on Gaussian mixture models, deep neural networks, and recurrent neural networks (RNNs). We evaluate our techniques with objective acoustic distortion measures and subjective listening tests over spoken sentences read from novels (the CMU Arctic corpus). Our results show that the best performing technique is a bidirectional RNN (BiRNN), which employs both past and future contexts to predict the acoustics from the sensor data. BiRNNs are not suitable for synthesis in real time but fixed-lag RNNs give similar results and, because they only look a little way into the future, overcome this problem. Listening tests show that the speech produced by this method has a natural quality that preserves the identity of the speaker. Furthermore, we obtain up to 92% intelligibility on the challenging CMU Arctic material. To our knowledge, these are the best results obtained for a silent-speech system without a restricted vocabulary and with an unobtrusive device that delivers audio in close to real time. This work promises to lead to a technology that truly will give people whose larynx has been removed their voices back.

International audience

In our research, we explore possible solutions for extracting valuable information about first responders’ (FR) location from speech communication channels during crisis response. Fine-grained identification of fundamental units of meaning (e. g. sentences, named entities and dialogue acts) is sensitive to high error rate in automatic transcriptions of noisy speech. However, looking from a topic-based perspective and utilizing text vectorization techniques such as Latent Dirichlet Allocation (LDA) make this more robust to such errors. In this paper, the location estimation problem is framed as a topic segmentation task on FRs’ spoken reports about their observations and actions. Identifying the changes in the content of a report over time is an indication that the speaker has moved from one particular location to another. This provides an estimation about the location of the speaker. A goal-oriented human/human conversational speech corpus was collected based on an abstract communication model between FR and task leader during a search process in a simulation environment. Results show the effectiveness of a topic-based approach and especially low sensitivity of the LDA-based method to the highly imperfect automatic transcriptions.

This paper investigates the potential of a silent speech interface (SSI) based on Permanent Magnetic Articulography (PMA) to be used in applications involving unconstrained, phonetically rich speech. In previous work the SSI was evaluated on isolatedword and connected-digits recognition tasks with promising results. Furthermore, it was shown that PMA data is enough to distinguish between minimal pairs of consonants with the same manner and place of articulation but different voicing. The study presented in this paper extends previous work by investigating to what extent information about the speech production process can be recovered from PMA data. In particular, the three main aspects of speech production are investigated here: voicing, place of articulation and manner of articulation. The results show that PMA achieves comparable accuracy to using the audio signal for discriminating the place of articulation of consonants, but it provides little information regarding the voicing and manner of articulation.

This paper presents a computational model of human speech production based on the hypothesis that low energy attractors for a human speech production system can be identified, and that interpolation/extrapolation along the key dimension of hypo/hyper-articulation can be motivated by energetic considerations of phonetic contrast. An HMM-based speech synthesiser along with continuous adaptation of its statistical models was used to implement the model. Two adaptation methods were proposed for vowel and consonant models and their effectiveness was tested by showing that such hypo/hyper-articulation control can manipulate successfully the intelligibility of synthetic speech in noise. Objective evaluations with the ANSI Speech Intelligibility Index indicate that intelligibility in various types of noise is effectively controlled. In particular, in the hyper-articulation transforms, the improvement with respect to unadapted speech is above 25 %.

In previous publications, a silent speech interface based on permanent-magnetic articulography (PMA) has been introduced and evaluated using standard automatic speech recognition techniques. However, word recognition is a task that is computationally expensive and introduces a significant time delay between speech articulation and generation of the acoustic signal. This paper investigates a direct synthesis approach where control parameters for parametric speech synthesis are generated directly from the sensor data of the silent speech interface, without an intermediate lexical representation. Users of such a device would not be tied to the limited vocabulary of a word-based recogniser and could therefore express themselves more freely. This paper presents a feasibility study that investigates whether it is possible to infer speech synthesis parameters from PMA sensor data

Cross-language speech recognition often assumes a certain amount of knowledge about the target language. However, there are hundreds of languages where not even the phoneme inven-tory is known. In the work reported here, phone recognisers are evaluated on a cross-language task with minimum target knowl-edge. A phonetic distance measure is introduced for the evalua-tion, allowing a distance to be calculated between any utterance of any language. This has a number of spin-off applications such as allophone detection, a phone-based ROVER approach to recognition, and cross-language forced alignment. Results show that some of these novel approaches will be of immediate use in characterising languages where there is little phonological knowledge. Index Terms: universal phone recognition, cross-language speech recognition, forced alignment, under-resourced lan-guages 1.

In 1997, and again in 2003, the author was invited to conduct a survey at the IEEE workshop on ‘Automatic Speech Recognition and Understanding ’ (ASRU) in which attendees were offered a set of statements about putative future events relating to progress in various aspects of speech technology R&D. The task of the respondents was to assign a date to each possible event. The 1997 and 2003 results were published at INTERSPEECH 2005 in Lisbon. Six years later, the author was invited by the organisers of ASRU’2009 to repeat the survey for a third time, and this paper presents the combined results from all three 1997, 2003 and 2009 surveys. The overall conclusion is that, over the twelve year period progress is perceived as slow, and the future appears to be generally no nearer than it has been in the past. However, on a positive note, the 2009 survey confirmed that the market for speech technology applications on mobile devices would be highly attractive over the next ten or so years. Index Terms: speech recognition, speech synthesis, survey of progress, future predictions

\n Contains fulltext :\n 86046.pdf (Publisher’s version ) (Open Access)\n

This paper reports on isolated word recognition experiments using a novel silent speech interface. The interface consist of magnetic pellets that are fixed to relevant speech articulators, and a set of magnetic field sensors that measure changes in the overall magnetic field created by these pellets during speech. The reported experiments demonstrate the effectiveness of this technique and show the suitability of the system, even at its early stages of development, for small vocabulary speech recognition.

This paper describes an attention-gating recurrent self-organising map approach for emergent speech representation. Inspired by evidence from human cognitive processing, the architecture combines two main neural components. The first component, the attention-gating mechanism, uses actor–critic learning to perform selective attention towards speech. Through this selective attention approach, the attention-gating mechanism controls access to working memory processing. The second component, the recurrent self-organising map memory, develops a temporal-distributed representation of speech using phone-like structures. Representing speech in terms of phonetic features in an emergent self-organised fashion, according to research on child cognitive development, recreates the approach found in infants. Using this representational approach, in a fashion similar to infants, should improve the performance of automatic recognition systems through aiding speech segmentation and fast word learning.

\n Contains fulltext :\n 76399.pdf (author's version ) (Open Access)\n

In order to avoid global parameter settings which are locally suboptimal, this paper argues for the inclusion of more knowledge (in particular procedural knowledge) into automatic speech recognition (ASR) systems. Two related fields provide inspiration for this new perspective: (a) ‘cognitive architectures’ indicate how experience with related problems can give rise to more (expert) knowledge, and (b) ‘case-based reasoning’ provides an extended framework which is relevant to any similarity-based recognition systems. The outcome of this analysis is a proposal for a new approach termed ‘Case-Based ASR’.

There has been considerable debate over the existence of the ‘vocabulary spurt ’ phenomenon- an apparent acceleration in word learning that is commonly said to occur in children around the age of 18 months. This paper presents an investigation into modelling the phenomenon using data from almost 1800 children. The results indicate that the acquisition of a receptive/productive lexicon can be quite adequately modelled as a single growth function with an ecologically well founded and cognitively plausible interpretation. Hence it is concluded that there is little evidence for the vocabulary spurt phenomenon as a separable aspect of language acquisition. Index Terms: child language acquisition, lexical development, vocabulary size

In this paper, we discuss a computational model that is able to detect and build word-like representations on the basis of sensory input. The model is designed and tested with a further aim to investigate how infants may learn to communicate by means of spoken language. The computational model makes use of a memory, a perception module, and the concept of 'learning drive'. Learning takes place within a communicative loop between a 'caregiver' and the 'learner'. Experiments carried out on three European languages with different genetic background (Finnish, Swedish, and Dutch) show that a robust word representation can be learned in using less than 100 acoustic tokens (examples) of that word. The model is inspired by the memory structure that is assumed functional for human cognitive processing.

Natural human speech contains a large amount of variation which poses difficulties for contemporary applications of speech technology. Following Lindblom's theory of hyper- and hypospeech, variation in speech is governed by a general principle, which is to optimise the ratio of information throughput to the energy invested. Even though such a relation is often implicitly assumed by researchers, it has never been quantified in terms of actual energy consumption. This is due to the difficulty of measuring speech energetics in humans. This paper introduces an animatronic model of the vocal tract – ‘AnTon’ – which has been designed to quantify energy relations in human speech production. Experiments are reported that demonstrate the animatronic tongue's ability to imitate basic human speech gestures.

This paper introduces a new model for automatic speech recognition (ASR) called TEMM- Temporal Episodic Memory Model. TEMM is derived from a simulation of human episodic memory called MINERVA2, and it not only overcomes the inability of MINERVA2 to use temporal sequence for recognition flexibly, but it also employs a prediction mechanism as an additional source of information. The performance of TEMM on an ASR task is compared to state-of-the-art HMM/GMM baseline systems, and a first analysis shows both promising results and a need to further stabilise the consistency of the output of the new model.

It is well known that utterances convey a great deal of information about the speaker in addition to their semantic content. One such type of information consists of cues to the speaker's personality traits, the most fundamental dimension of variation between humans. Recent work explores the automatic detection of other types of pragmatic variation in text and conversation, such as emotion, deception, speaker charisma, dominance, point of view, subjectivity, opinion and sentiment. Personality affects these other aspects of linguistic production, and thus personality recognition may be useful for these tasks, in addition to many other potential applications. However, to date, there is little work on the automatic recognition of personality traits. This article reports experimental results for recognition of all Big Five personality traits, in both conversation and text, utilising both self and observer ratings of personality. While other work reports classification results, we experiment with classification, regression and ranking models. For each model, we analyse the effect of different feature sets on accuracy. Results show that for some traits, any type of statistical model performs significantly better than the baseline, but ranking models perform best overall. We also present an experiment suggesting that ranking models are more accurate than multi-class classifiers for modelling personality. In addition, recognition models trained on observed personality perform better than models trained using self-reports, and the optimal feature set depends on the personality trait. A qualitative analysis of the learned models confirms previous findings linking language and personality, while revealing many new linguistic markers.

Recent years have seen steady improvements in the quality and performance of speech-based human-machine interaction driven by a significant convergence in the methods and techniques employed. However, the quantity of training data required to improve state-of-the-art systems seems to be growing exponentially and performance appears to be asymptotic to a level that may be inadequate for many real-world applications. This suggests that there may be a fundamental flaw in the underlying architecture of contemporary systems, as well as a failure to capitalize on the combinatorial properties of human spoken language. This paper addresses these issues and presents a novel architecture for speech-based human-machine interaction inspired by recent findings in the neurobiology of living systems. Called PRESENCE-"PREdictive SENsorimotor Control and Emulation"-this new architecture blurs the distinction between the core components of a traditional spoken language dialogue system and instead focuses on a recursive hierarchical feedback control structure. Cooperative and communicative behavior emerges as a by-product of an architecture that is founded on a model of interaction in which the system has in mind the needs and intentions of a user and a user has in mind the needs and intentions of the system.

Abstract Speech technology is potentially of enormous benefit to people with physical disabilities. Applications of speech technology to e-inclusion are reviewed and described in the areas of access, control, communication and rehabilitation/therapy, with particular reference to speech technology developments for people with disordered speech. To be successful, applications should effectively take into account the needs of user groups and have the ability to adapt to the needs of individuals. This is a challenging area but effective progress can be made through multi-disciplinary research and development. 1. Introduction People with physical disabilities take advantage of a variety of methods to gain access to information technology and to electronic assistive technology for communication, mobility and daily living tasks. Many of these access methods are slow and can lead to frustration, a prime example being the use of switch-activated menu scanning (hereafter referred to as switch-scanning). Automatic Speech Recognition is potentially of enormous benefit to people with severe physical disabilities. The tremendous richness of human speech communication gives the user many degrees of freedom for control and input. The speed of speech recognition also gives it a potential advantage over other input methods commonly employed by physically disabled people. People with neurological conditions causing disability often have associated dysarthria, which is the most common acquired speech disorder affecting 170 per 100,000 population [1]. This may be developmental dysarthria such as that associated with cerebral palsy or acquired dysarthria associated with progressive neurological disease (e.g. Parkinson's disease, motor neurone disease, and multiple sclerosis) or sudden onset conditions such as/or head injury. In its severest form, dysarthric speech is unintelligible to others and may take the form of producing vocal utterances, rather than words recognisable to unfamiliar communication partners. The combination of speech and general physical disability can make it particularly problematic for people to interact in their environment and can severely limit independence and inclusion. This paper examines some of the areas for e-inclusion of people with disabilities that can benefit from the use of speech technology. These areas include access, control, communication, rehabilitation and therapy. In all areas, the use of speech technology for people with dysarthric speech is particularly examined and discussed.

This paper investigates a simulation of episodic memory known in the literature as ‘MINERVA 2’. MINERVA 2 is a computational multiple-trace memory model that successfully predicts basic findings from the schema-abstraction literature. This model has been implemented and tested on a simple ASR task using vowel formant data taken from the Peterson & Barney database. Recognition results are compared to a number of state-of-the-art pattern classifiers, and it is shown that the episodic model achieves the best performance.

In 1997 the author conducted a survey at the IEEE workshop on ‘Automatic Speech Recognition and Understanding’ (ASRU) in which attendees were offered a set of twelve putative future events to which they were asked to assign a date. Six years later at ASRU’2003, the author repeated the survey with the addition of eight additional items. This paper presents the combined results from both surveys.

ATA mining is concerned with the science, technology, and engineering of discovering patterns and extracting potentially useful or interesting information automatically or semi-automatically from data. Data mining was introduced in the 1990s and has deep roots in the fields of statistics, artificial intelligence, and machine learning. With the advent of inexpensive storage space and faster processing over the past decade or so, data mining research has started to penetrate new grounds in areas of speech and audio processing as well as spoken language dialog. It has been fueled by the influx of audio data that are becoming more widely available from a variety of multimedia sources including webcasts, conversations, music, meetings, voice messages, lectures, television, and radio. Algorithmic advances in automatic speech recognition have also been a major, enabling technology behind the growth in data mining. Current state-of-the-art, large-vocabulary, continuous speech recognizers are now trained on a record amount of data—several hundreds of millions of words and thousands of hours of speech. Pioneering research in robust speech processing, large-scale discriminative training, finite state automata, and statistical hidden Markov modeling have resulted in real-time recognizers that are able to transcribe spontaneous speech with a word accuracy exceeding 85%. With this level of accuracy, the technology is now highly attractive for a variety of speech mining applications. Speech mining research includes many ways of applying machine learning, speech processing, and language processing algorithms to benefit and serve commercial applications. It also raises and addresses several new and interesting fundamental research challenges in the areas of prediction, search, explanation, learning, and language understanding. These basic challenges are becoming increasingly important in revolutionizing business processes by providing essential sales and marketing information about services, customers, and product offerings. They are also enabling a new class of learning systems to be created that can infer knowledge and trends automatically from data, analyze and report application performance, and adapt and improve over time with minimal or zero human involvement. Effective techniques for mining speech, audio, and dialog data can impact numerous business and government applications. The technology for monitoring conversational speech to discover patterns, capture useful trends, and generate alarms is essential for intelligence and law enforcement organizations as well as for enhancing call center operation. It is useful for an

An often-cited advantage of automatic speech recognition (ASR) is that it is ‘fast’; it is quite easy for a person to speak at several hundred words a minute, well above the rates that are possible using other modes of data entry. However, in order to conduct a fair comparison between alternative data entry methods, it is necessary to consider not the input rate per se, but the rate at which it is possible to enter information that is fully correct. This paper describes a model for predicting the relative success of alternative method of data entry in terms of the effective ‘throughput’ that is achievable taking into account typical input data entry rates, error rates and error correction times. Results are presented for the entry of both conventional and SMS-style text.

Since the introduction of hidden Markov modelling there has been an increasing emphasis on data-driven approaches to automatic speech recognition. This derives from the fact that systems trained on substantial corpora readily outperform those that rely on more phonetic or linguistic priors. Similarly, extra training data almost always results in a reduction in word error rate- “there's no data like more data”. However, despite this progress, contemporary systems are not able to fulfill the requirements demanded by many potential applications, and performance is still significantly short of the capabilities exhibited by human listeners. For these reasons, the R&D community continues to call for even greater quantities of data in order to train their systems. This paper addresses the issue of just how much data might be required in order to bring the performance of an automatic speech recognition system up to that of a human listener. 1.

Significant progress has taken place in ‘Spoken Language Output ’ (SLO) R&D, yet there is still some way to go before it becomes a ubiquitous and widely deployed technology. This paper reviews the challenges facing SLO, using ‘Technology Roadmapping ’ (TRM) to identify market drivers and future product concepts. It concludes with a summary of the behaviours that will be required in future SLO systems. 1.

Recent years have seen dramatic developments in the capabilities and applications of spoken language technology. From a few niche applications for a range of expensive solutions, the field has developed to the point where keenly-priced products have swept the awards at consumer electronics shows. Speech recognisers has reached the high-street store, and a significant proportion of the developed world's population has now been exposed to the possibility of controlling one's computer, or creating a document, by voice.

Despite the significant theoretical and practical advances that have been made in automatic speech recognition, relatively little effort has been devoted to the evaluation of speech in an interactive multi-modal application interface. This paper introduces a general methodology for assessing speech-based systems and concludes with a proposal for a test scenario which focuses on the understanding component of a spoken language system.

Although the possibility of asynchrony between different components of the speech spectrum has been acknowledged, its potential effect on automatic speech recogniser performance has only recently been studied. This paper presents the results of continuous speech recognition experiments in which such asynchrony is accommodated using a variant of HMM decomposition. The paper begins with an investigation of the effects of partitioning the speech spectrum explicitly into subbands. Asynchrony between these sub-bands is then accommodated, resulting in a significant decrease in word errors. The same decomposition technique has previously been used successfully to compensate for asynchrony between the two input streams in an audiovisual speech recognition system.

Dialogue move recognition is taken as being representative of a class of spoken language applications where inference about high level semantic meaning is required from lower level acoustic, phonetic or word based features.Topic identication is another such application.In the particular case of inference from words, the multinomial distribution is shown to be inadequate for modelling word frequencies, and the multivariate Poisson is a more reasonable choice.Zipf's law is used to model a prior distribution.This more rigorous mathematical formulation is shown to improve dialogue move classication both subjectively and quantitatively.

This paper compares two approaches to context-sensitive phoneme-level hidden Markov modelling for vocabulary-independent automatic speech recognition. The first is an existing method based on the use of binary decision trees to identify equivalence classes of contexts which induce the same effect on the acoustic realisation of a given phoneme. The second is a novel method, called context adaptive phone modelling, which is based on the use of 'context-independent generalised phones-in-context'. In the first method equivalence classes of contexts are derived from a direct analysis of the acoustic patterns, whereas the second approach utilises a symbolic transcription of the training corpus. The paper presents an experimental and methodological comparison of the two methods.>

Research on continuous-speech recognition using phoneme-level hidden Markov models (HMMs) is described. The aim of the project is automatic recognition of spoken airborne reconnaissance mission (ARM) reports. The evolution of the ARM system from a simple baseline system to its current configuration is described, and a considerable number of experimental results are included. Work on alternative approaches to modeling contextual effects and on improved duration modeling is described.>

The problem of automatic speech recognition in the presence of interfering signals and noise with statistical characteristics ranging from stationary to fast changing and impulsive is discussed. A technique of signal decomposition using hidden Markov models is described. This is a generalization of conventional hidden Markov modeling that provides an optimal method of decomposing simultaneous processes. The technique exploits the ability of hidden Markov models to model dynamically varying signals in order to accommodate concurrent processes, including interfering signals as complex as speech. This form of signal decomposition has wide implications for signal separation in general and improved speech modeling in particular. The application of decomposition to the problem of recognition of speech contaminated with noise is emphasized.>

The information produced by a straightforward auditory model, like the original speech signal, is subject to extreme local variability and is not suitable for direct presentation to speech recognition algorithms. Therefore, an analysis is required which will make the important features of the neural firing patterns explicit, but which will describe them concisely and without disruption by known characteristics of speech. Several attempts have been made to develop such an analysis, however, such analyses ignore significant aspects of neural firing patterns, and consequently information is lost (usually that associated with fine temporal structure). The authors describe a 'reduced auditory representation', which retains more information, and presents the results of preliminary recognition experiments using this data.>

A preliminary theoretical and experimental examination is made of three noise compensation techniques. The three techniques are those due to: Klatt (1976); Bridle et al (1984); and Holmes & Sedgwick (1986). The first two of these techniques have been re-interpreted for use within a hidden Markov model based recogniser. A description is given of how this was done, together with a discussion on some implementation considerations. Experimental results are given for the performance of the algorithms at various signal-to-noise ratios. The principles of recognition in noise are discussed from an implementation point of view and it is shown how the three techniques can be viewed as variations on a single theme.>

Semi-Markov models have been proposed as a mechanism for overcoming some of the limitations inherent in first-order Markov modelling of speech signals. Results have been presented which show that these models provide an appropriate framework for modelling durational structure and can lead to significant improvements in recognition accuracy.

Whole-word pattern matching using dynamic time-warping (DTW) has achieved considerable success as an algorithm for automatic speech recognition. However, the performance of such an algorithm is ultimately limited by its inability to discriminate between similar sounding words. The problem arises because all differences between speech patterns are treated as being equally important, hence the algorithm is particularly susceptible to confusions caused by irrelevant differences. This paper presents an alternative DTW approach which is able to focus its attention on those parts of a speech pattern which serve to distinguish it from similar patterns. A network-type data structure is derived from reference speech patterns, and the separate paths through the network determine the regions where recognition takes place. Results indicate that discrimination between similar sounding words can be greatly improved.

Dynamic Time-Warping is one the most important tools available for overcoming timescale variability problems in Automatic Speech Recognition. One of the main problems associated with the technique is to constrain the behaviour of the algorithm in order to avoid unlikely timescale distortion. This paper describes techniques for incorporating information about timescale variability directly into the Dynamic Time-Warping process. Results are presented which show that these techniques can lead to considerable improvements in recognition accuracy, especially if the differences between word classes are mainly due to temporal structure.

Recent years have seen the emergence of dynamic programming as one of the most important tools available for overcoming temporal variability problems in automatic speech recognition. Considerable research effort has been devoted to the investigation of different dynamic programming algorithms, yet most work has been concerned with global, rather than local, constraints upon temporal variation. Clearly the likelihood of timescale distortion is not constant for the entire duration of an utterance; variability must be, at least to some extent, data dependent. It is therefore desirable that information to this effect should be made available to the dynamic time warping process. This paper describes a technique for obtaining an estimate of local timescale variability based on a bi-directional dynamic programming algorithm and basic fuzzy set theory. Results are presented which indicate that this technique will lead to improved discrimination, especially if the differences between classes are mainly due to temporal structure.

The problems of speech recognition and orthographic word correction have been greatly mitigated by the use of dynamic programming techniques for finding the distance between two finite sequences. This paper extends the technique into two dimensions, and presents an algorithm for finding the distance between two finite areas. Applications of the algorithm are suggested.