The most intuitive way to model a transaction in the financial world is through a Graph. Every transaction can be considered as an edge between two vertices, one of which is the paying party and another is the receiving party. Properties of these nodes and edges directly map to business problems in the financial world. The problem of detecting a fraudulent transaction can be considered as a property of the edge. The problem of money laundering can be considered as a path-detection in the Graph. The problem of a merchant going delinquent can be considered as the property of a node. While there are many such examples, the above help in realising the direct mapping of Graph properties with the financial problems in the real-world. This tutorial is based on the potential of using Graph Neural Network based Learning for solving business problems in the financial world.

Of many things, Covid-19 has provided a stark proof that uncertainty is real, and it is here to stay. Perhaps nothing is more sensitive to uncertainty than the Financial World. To couple with it, while Artificial Intelligence techniques are used to predict the future state of events, their performance is significantly impacted by disruptions not captured in the past. Unforeseen scenarios such as economy changes, variations in the customer behaviour, pandemics, recessions, and fraudulent transactions often result in unexpected behaviour of financial models, thus associating a level of uncertainty with them. It is thus imperative for the research community to explore, identify, analyze, and address such uncertainties in order to develop robust models applicable in real-world scenarios. To this effect, the International Workshop on Modelling Uncertainty in the Financial World 2021 (MUFin21) aims to bring academics and industry experts together to discuss on this important, timely and yet- unsolved area of modelling uncertainties in the financial world.

Profile images play an important role in partner selection in a matrimony or dating site. The hypothesis of this paper is that perceived beauty of a profile image is a subjective opinion based on who is viewing the image. We validate this hypothesis by showing that this subjective bias for attractiveness can be learnt from the sender-receiver image pairs. We train a Bi-channel CNN based deep architecture that incorporates the visual features of both users and learns the attractiveness of sender as perceived by the receiver. This network was trained and tested on 3.5 million image pairs and achieved an accuracy of 69% with images alone, thus proving that rather than the eye, beauty lies in the face of the beholder. When this network was used in conjunction with other profile features such as age, city and caste, it further improved the accuracy of the system by a 5% relative number.

Gaze based navigation with digital screens offer a hands-free and touchless interaction, which is often useful in providing a hygienic interaction experience in a public kiosk scenario. The goodness of such a navigation system depends not only on the accuracy of detecting the eye gaze but also on the ability to determine whether a user is interested in clicking a button or is just looking at the button. The time for which a user needs to gaze at a particular button before it is considered as a click action is called the dwell time. In this paper, we explore intelligent adjustment of dwell times, where mouse click events on the buttons of a given application are emulated with user gaze. A constant dwell-time for all buttons and for all users may not provide an efficient and intuitive interface. We thereby propose a model to dynamically adjust dwell-time values used to emulate user mouse click events, exploiting the user's experience with different portions of a given application. The adjustment happens at a per-user, per-button granularity, as a function of the user's (a) prior usage experience of the given button within the application and (b) Midas touch characteristics for the given button. We propose OptiDwell, inspired by the action-value method based solutions to the Multi-Armed Bandits problem, for dwell click time adaptation. We experiment OptiDwell using an interactive TV channel browsing interface application, constituting of a mix of text and image buttons, over 10 computer-savvy users generating over 9000 click tasks. We observe significant improvement of user comfort level over the sessions, quantified by (a) improved (reduced) dwell times and (b) reduced number of Midas touches in spite of faster dwell-clicks, as high as 10-fold reduction in the best case. Our work is useful for creating an interface, with accurate, fast and comfortable dwell-clicks for each interface element (e.g., buttons), and each user.

Gesture-based interaction provides a multitude of benefits to individuals with disabilities, for example, enhancing social, motor and cognitive skills. However, applications that encourage self-efficacy by promoting a life-skill through simulations of real world scenarios are largely missing. We explore the benefits of using a gesture-based application for individuals with developmental disabilities. The context is a special school in New Delhi, Nai Disha, where we designed and developed an application, Kirana, that integrates arithmetic and social interaction to teach purchasing of items from a local grocery store. In our study, 18 participants with developmental disabilities, previously unable to visit a grocery store, used Kirana for three weeks. Our results indicate that gesture-based applications can teach a life skill and enable self-efficacy for individuals with developmental disabilities by breaking down complex tasks that require social, mathematical and decision-making skills.

Kirana is a gesture-based application that simulates purchasing experience at a local Indian grocery store. It provides individuals with developmental difficulties a safe and controlled environment to explore the processes of purchasing an item: namely, deciding items to buy (decision making), calculating costs and balance (mathematical skills), and interactions via pointing (social interaction). Previous research has established that gesture-based interaction has the potential to enhance social, motor and cognitive skills for individuals with developmental disabilities. Currently, translating learnings from gesture-based virtual applications to real world scenarios is underexplored. Kirana is an approach to simulating practical real interactions, by breaking down complex tasks that require social, mathematical and decision-making skills, and encouraging self-efficacy. Skills learnt by purchasing items with Kirana, are potentially transferable to a real world store.

There exists mounting evidence in favor of computer supported autism interventions at the individual level. However, the potential benefits of using computer supported collaboration to encourage social interactions between individuals with autism and typically developed individuals are underexplored, particularly in developing regions. We present an exploratory study of a collaborative gesture-based application, Balloons. The application encourages joint attention, which is defined as the shared attention between two individuals towards the same object. Using mixed methods, we evaluated Balloons for three weeks in New Delhi with ten medium-low functioning autistic children. Our findings suggest that employing CSC interventions for children with autism in India provide (a) observable improvements in social interaction with typically developed peers, (b) the opportunity to customize and individualize intervention to cater to a large spectrum of children and (c) the potential opportunity of reducing fears of certain objects.

This paper lays the ground work for assistive navigation using wearable sensors and social sensors to foster situational awareness for the blind. Our system acquires social media messages to gauge the relevant aspects of an event and to create alerts. We propose social semantics that captures the parameters required for querying and reasoning an event-of-interest, such as what, where, who, when, severity, and action from the Internet of things, using an event summarization algorithm. Our approach integrates wearable sensors in the physical world to estimate user location based on metric and landmark localization. Streaming data from the cyber world are employed to provide awareness by summarizing the events around the user based on the situation awareness factor. It is illustrated using disaster and socialization event scenarios. Discovered local events are fed back using sound localization so that the user can actively participate in a social event or get early warning of any hazardous events. A feasibility evaluation of our proposed algorithm included comparing the output of the algorithm to ground truth, a survey with sighted participants about the algorithm output, and a sound localization user interface study with blind-folded sighted participants. Thus, our framework supports the navigation problem for the blind by combining the advantages of our real-time localization technologies so that the user is being made aware of the world, a necessity for independent travel.

We present a mobile phone based cricket game for improving the spoken English pronunciation of school children designed and tested within a specific socio-cultural context in rural India, the Mewat district of Haryana State. The development of the game concept was informed by a field study, which identified the cultural restrictions respected by the community as well community interests and motivations. The game was accessible using a low-end mobile phone and evaluated with a group of 63 students from classes 4 and 5 of a rural school. The results suggest that the cricket game can be effectively used to engage students and to improve their spoken English skills in the given setting. Further, the game serves as an informative example of how factors impacting the acceptability and appropriation of a technology in a particular setting can be taken into account from the very beginning of the design process.

Heavy penetration of mobile devices in rural areas enables access to information services to low-literate users. While connectivity solves a major issue, concepts such as browsing and searching are not intuitive to such users. This paper presents the design of a multimodal (speech+icons) voice browser for low-literate users in rural India. We conducted a field study to determine their current communication styles and preferences. Based on this, we designed three browsers, each influenced by a specific way of communicating -- persona based roles, storytelling and direct interaction. The prototypes were evaluated in a task-based study with 62 low-literate users and were compared with a baseline Interactive Voice Response (IVR) browser. Our results suggest that there is clear acceptance and understanding of the necessary concepts if the browser is designed using the design constructs of persona based roles and storytelling.

Building search engines that can respond to spoken queries with spoken content requires that the system not just be able to find useful responses, but also that it know when it has heard enough about what the user wants to be able to do so.This paper describes a simulation study with queries spoken by non-native speakers that suggests that indicates that finding relevant content is often possible within a half minute, and that combining features based on automatically recognized words with features designed for automated prediction of query difficulty can serve as a useful basis for predicting when that useful content has been found.

The SiMPE workshop series started in 2006 with the goal of enabling speech processing on mobile and embedded devices. The SiMPE 2012 workshop extended the notion of audio to non-speech "Sounds" and thus the expansion became "Speech and Sound". SiMPE 2010 and 2011 brought together researchers from the speech and the HCI communities. Speech User interaction in cars was a focus area in 2009. Multimodality got more attention in SiMPE 2008. In SiMPE 2007, the focus was on developing regions.

In this paper we introduce Acoustic Quick Response codes to facilitate sharing between Interactive Voice Response (IVR) service users. IVRs are telephone-based, and similar to the world wide web in many aspects, but currently lack support for content sharing. Our approach uses 'audio codes' to let people share their call positions, and allows callers to hold their normal (low-end) handsets together to synchronise. The technique uses remote generation and recognition of audio codes to ensure that sharing is possible on any type of phone without the need for textual literacy or an internet connection. We begin by exploring existing user needs for sharing, then evaluate the technical robustness of our audio-based design. We demonstrate the value of the approach for voice service users over several separate studies--including an eight-month extended field deployment--then conclude with a discussion of future possibilities for such scenarios.

Several issues arise with management of content that is generated in developing regions. Some result from linguistic diversity (as in India and Africa), some result from content being available only in forms that are more difficult to computationally manipulate (e.g., handwriting, speech, and legacy digital text in nonstandard encodings), some result from underinvestment in language resources for the languages of these regions, and some result from increased contact between cultures that have different views regarding the proper use of information and information artifacts. Such issues warrant focused attention if we are to optimally leverage information and knowledge management to the advantage of populations in developing regions. That is the purpose of this workshop.

In this paper, we describe the “Spoken Web Search” Task, which was held as part of the 2011 MediaEval benchmark campaign. The purpose of this task was to perform audio search with audio input in four languages, with very few resources being available in each language. The data was taken from “spoken web” material collected over mobile phone connections by IBM India. We present results from several independent systems, developed by five teams and using different approaches, compare them, and provide analysis and directions for future research.

Virtually all work on topic modeling has assumed that the topics are to be learned over a text-based document corpus. However, there exist important applications where topic models must be learned over an audio corpus of spoken language. Unfortunately, speech-to-text programs can have very low accuracy. We therefore propose a novel topic model for spoken language that incorporates a statistical model of speech-to-text software behavior. Crucially, our model exploits the uncertainty numbers returned by the software. Our ideas apply to any domain in which it would be useful to build a topic model over data in which uncertainties are explicitly represented.

Word clouds are extensively used to present a summary of the prominent words in a document on the World Wide Web. Such clouds give the user an idea about the content of the document. In this paper we present a mechanism to create and render an audio cloud for audio content. Such audio clouds are expected to provide a similar summary of the audio documents. They have wide applicability in various domains, especially for low-literate users who currently do not use the Internet but interact with audio-based systems.

Much of the mobile work by HCI researchers explores a future world populated by high-end devices and relatively affluent users. This paper turns to consider the hundreds of millions of people for whom such sophistication will not be realised for many years to come. In developing world contexts, people will continue to rely on voice-primary interactions due to both literacy and economic reasons. Here, we motivate research into how to accommodate advanced mobile interface techniques while overcoming the handset, data-connection and user limitations. As a first step we introduce TapBack: back-of-device taps to control a dialled-up, telephone-network-based voice service. We show how these audio gestures might be recognised over a standard telephone connection, via users' existing low-end devices. Further, in a longitudinal deployment, the techniques were made available on a live voice service used by rural Indian farmers. Data from the study illustrates the desire by users to adopt the approach and its potential extensions.

Spoken Web is an alternative Web for low-literacy users in the developing world. People can create audio content over phone and share on the Spoken Web. This enables easy creation of locally relevant content. Even on the World Wide Web in developed regions, the recent increase in traffic is due to the locally relevant content created on social networking sites. This paper argues that content search and ranking in the new scenario needs a re-look. The generic model of using in-links for ranking such content is not an appropriate measure of the content relevance in such a collaborative Web 2.0 world. This paper aims to bring the social context in Spoken Web ranking. We formulate a relationship function between the query-creator and the content-creator and use this as one measure of the content relevance to the user. The relationship function uses the geographical location of the two people and their prior browsing preferences as parameters to determine the relationship between the two users. Further we also determine the trustability of the content based on the content creator's acceptance measure by the social network. We use these two features in addition to the term-frequency - inverse-term-frequency match to rank the search results in context of the social network of the query-creator and provide a more specific and socially relevant result to the user.

The Spoken Web is a voice-based equivalent of the World Wide Web (WWW), developed by IBM Research Laboratory, India, primarily designed for rural and semi-urban people to provide information of value to them through their mobile or landline phones. This demonstration aims to present the use of VoiceSites for information creation and access for the rural population. VoiceSites can be accessed by calling a phone number and end-users can listen to the content over phone and navigate through voice or the phone keypad. While one demonstration is of creating a VoiceSite, the other is for browsing a multitude of VoiceSites that would also demonstrate conducting transactions across VoiceSites. We will also present the mechanism to search for content on VoiceSites. These demonstrations will help in presenting the concept of the Spoken Web, which we foresee, can have the same effect in the developing regions community which the WWW has done to the developed world over the last decade.

Information Technology (IT) has had significant impact on the society and has touched all aspects of our lives. Up and until now computers and expensive devices have fueled this growth. It has resulted in several benefits to the society. The challenge now is to take this success of IT to its next level where IT services can be accessed by the users in developing regions. The focus of the workshop in 2011 is to identify the alternative sources of intelligence and use them to ease the interaction process with information technology. We would like to explore the different modalities, their usage by the community, the intelligence that can be derived by the usage, and finally the design implications on the user interface. We would also like to explore ways in which people in developing regions would react to collaborative technologies and/or use collaborative interfaces that require community support to build knowledge bases (example Wikipedia) or to enable effective navigation of content and access to services.

"In all the wonderful worlds that writing opens, the spoken word still resides and lives. Written texts all have to be related somehow, directly or indirectly, to the world of sound, the natural habitat of language, to yield their meanings."

With the proliferation of pervasive devices and the increase in their processing capabilities, client-side speech processing has been emerging as a viable alternative. The SiMPE workshop series started in 2006 [5] with the goal of enabling speech processing on mobile and embedded devices to meet the challenges of pervasive environments (such as noise) and leveraging the context they offer (such as location). SiMPE 2010, the latest in the series brought together, very successfully, researchers from the speech and the HCI communities. We believe this is the beginning.

Spoken Web is a web of VoiceSites that can be accessed by a phone. The content in a VoiceSite is audio. Therefore Spoken Web provides an alternate to the World Wide Web (WWW) in developing regions where low Internet penetration and low literacy are barriers to accessing the conventional WWW. Searching of audio content in Spoken Web through an audio query-result interface presents two key challenges: indexing of audio content is not accurate, and the presentation of results in audio is sequential, and therefore cumbersome. In this paper, we apply the concepts of faceted search and browsing to the SpokenWeb search problem. We use the concepts of facets to index the meta-data associated with the audio content. We provide a mechanism to rank the facets based on the search results. We develop an interactive query interface that enables easy browsing of search results through the top ranked facets. To our knowledge, this is the first system to use the concepts of facets in audio search, and the first solution that provides an audio search for the rural population.

Delivering new technology solutions in a well established IT-educated consumer market largely depends on the usefulness and usability of the solution. However, this is very different from delivering ICT solutions in developing regions where there are multiple factors that affect technology adoption, which seem to be at least as important as the technology itself. This paper revolves around a case study of World Wide Telecom Web (WWTW), commonly known as Spoken Web, a voice based content-creation and dissemination platform. The study reflects different challenges in adoption and different ways of evolution of WWTW in the varied organizational, social and operational settings in rural India. The case study uses pilot deployments in four regions in India, each involving at least 500 users. All the four deployments had a different organizational, social and operational model. This paper presents the implications of these differences on the WWTW technology adoption. The study shows that organizations have a strong influence in the adoption, social settings play a major role in the evolution of the solution and operational model influences the quality of the solution.

With the proliferation of pervasive devices and the increase in their processing capabilities, client-side speech processing has been emerging as a viable alternative. The SiMPE workshop series started in 2006 [5] with the goal of enabling speech processing on mobile and embedded devices to meet the challenges of pervasive environments (such as noise) and leveraging the context they offer (such as location).

It is a common practice for us to skim textual content on a web page. While skimming, we usually skip words or phrases that are not of interest to us and we slow down our speed when the content seems to be of relevance to us. But when we listen to audio content, which is not persistent and is sequential, such skimming is not possible. In developing countries, cell-phone penetration is much higher than Internet penetration. Moreover, due to low literacy, voice is a convenient modality to access information. The skimming techniques are therefore more critical in the audio domain. In this paper, we describe the technique for navigating audio content while interacting with information systems in a client server environment, where a dumb phone is the client device. The paper presents techniques for skimming audio content and for placing markers in audio. The user studies conducted with 18 users for more than 1 month, in a live setting substantiates the usability and usefulness of the navigation techniques.

This paper describes our experiences deploying a recommender system for a mobile phone-based knowledge sharing application to farmers in rural India. Users of the system record questions and call back for answers left by other users and experts. We used collaborative filtering to derive relevant content for each user based on historical navigation patterns of the community. An empirical analysis of behavioral and interview data reveals key issues for future mobile recommender systems in developing regions of the world.

In this paper we present a study comparing speech and dialed input voice user interfaces for farmers in Gujarat, India. We ran a controlled, between-subjects experiment with 45 participants. We found that the task completion rates were significantly higher with dialed input, particularly for subjects under age 30 and those with less than an eighth grade education. Additionally, participants using dialed input demonstrated a significantly greater performance improvement from the first to final task, and reported less difficulty providing input to the system.

83% of the world population does not have access to Internet. Therefore there is a need for a simple and affordable interaction technology that can enable easy content creation and dissemination for this population. In this paper, we present the design, development and usage pattern of a VoiKiosk system that provides a voice-based kiosk solution for people in rural areas. This system is accessible by phone and thus meets the affordability and low literacy requirements. We present usability results gathered from usage by more than 900 villagers during four month of the on-field deployment of the system. The on-field experiments suggest the importance of locally created content in their own language for this population. The system provides interesting insights about the manner in which this community can create and manage information. Based on the use of the system in the four months, the VoiKiosk also suggests a mechanism to enable social networking for the rural population.

Many services taken for granted in the developed world are often missing from the developing countries. One typical example is that of map systems that form the basis of several location driven services. Its heavy reliance on content, provides a huge barrier towards building such systems. Further, in developing countries like India, the infrastructure typically has a history of unplanned development, leading to unorganized addresses and absence of standard naming conventions for roads. Detailed map systems such as online maps have only recently started becoming available but for major cities. Remote towns and villages remain out of reach till date. In this paper, we investigate a community-driven approach for creating maps in developing regions - following Web 2.0 principles, but not entirely relying on the existing Web. Our system, dubbed FOLKSOMAPS is an intelligent, community constructed map system, particularly architected with developing regions in mind. We present the design of FOLKSOMAPS, followed by an implementation of our proof-of-concept system. We present user studies aimed at understanding the uptake, usability and utility of FOLKSOMAPS. The results indicate a strong need for such a community-generated mapping ecosystem.

Websites in World Wide Web are primarily meant for visual consumption. Moreover, the wide variety of visual controls available make it harder to interpret the websites with screen readers. This problem of accessing information and services on the web escalates even further for visually impaired in developing regions since most are either semi-literate/illiterate or cannot afford computers and high-end phones with screen reading capability.In this paper, we present an alternate platform -- the World Wide Telecom Web (WWTW). WWTW is already being successfully deployed as a network of VoiceSites that can be created and accessed by a voice interaction over an ordinary phone. WWTW presents a whole new set of opportunities for delivering information and services to the visually impaired. We present a preliminary user study that leads us towards the belief that the Telecom Web can be the mainstream Web for blind users.

Information Technology has had significant impact on the society and has touched all aspects of our lives. So far, computers and expensive devices have fueled this growth. The challenge now is to take this success of IT to its next level where IT services can be accessed by masses. "Masses" here mean the people who (a) are not yet IT literate and/or (b) do not have the purchase power to use the current IT delivery mechanisms (PC centric model) and/or (c) do not find current IT solutions and services relevant to their life or business. Interestingly, a huge portion of the world's population falls in this category. To enable the IT access to such masses, this workshop focuses on easy-to-use and affordable, yet powerful, user interfaces that can be used by this population.

In the past, voice-based applications have been accessed using unintelligent telephone devices through Voice Browsers that reside on the server. The proliferation of pervasive devices and the increase in their processing capabilities, clientside speech processing has been emerging as a viable alternative. In SiMPE 2008, the third in the series, we will continue to explore the various possibilities and issues that arise while enabling speech processing on resource-constrained, possibly mobile devices.

In this paper we describe our experiences designing a voice interface in rural India. We outline our design process from initial contextual inquiry to a formal user evaluation, and use this discussion to motivate research guidelines for others designing voice interfaces in developing regions. Our three guidelines are to build around existing information systems, to iterate on the design through user testing, and to explore design alternatives through empirical analysis. We also share some practical lessons learned in designing, implementing, and evaluating information systems for developing regions in general.

Information Retrieval on the WWW is important because it is hard to find what one is looking for. There is a plethora of information available, and searching relevant information is a challenge. In the case of developing regions, we have the opposite problem: 1) Information availability of global markets is scarce. Most of the consumers and producers (of information as well as goods) are relegated to local markets in geographical vicinity. In order to reach wider markets, it is important for this local information to reach wider audiences. (Local information for global consumption LIG model). (2) At the same time, locally relevant information, such as delays in bus/train timings, mobile medical van schedule changes, electricity outage timings, is not easily available either. (Local information for local consumption LIL model). We introduce the term Information Uptrieval to address the reverse problem of acquiring, assimilating, aggregating and uploading global and local information that is relevant for developing regions to a platform that improves the reach of the information. While the WWW is an obvious example of one such platform, given the low internet penetration in such regions, we need to explore effective alternatives. Several innovative, but disconnected approaches have been attempted to address the information uptrieval problem, ranging from the use of DVDs (eSagu, http://www.esagu.in/esagu) through the use of wireless stations on motorcycles (First Mile Solutions, http://www.firstmilesolutions.com). Many of these have met with reasonable success in their pilot deployments.

As the number of telephony voice applications grow, there will be a need for a browser to surf the Web of interconnected voice applications (called as VoiceSites). These VoiceSites are accessed through a telephone over an audio channel. We present the concept and architecture of T-Web Browser, a World Wide Telecom Web browser that enables browsing the Web of voice applications through an ordinary phone. This browser will support rich browsing features such as history and bookmarking.

One of the several initiatives to bridge the digital divide in developing countries has been the deployment of information kiosks or knowledge centers in villages in rural parts of the country. These kiosks provide services ranging from email, chat and browsing to distance education programs, agricultural services and eGovernance services. A kiosk typically comprises of a computer with printer, web cam, multimedia system and Internet connectivity and is owned by a local entrepreneur. Moving away from the PC based kiosk model, we present an alternative platform to create and host such information kiosks in the telephony network. We call these as VoiKiosks and they are accessible through voice interaction over an ordinary phone call.

Various sectors in developing countries are typically dominated by the presence of a large number of small and microbusinesses that operate in an informal, unorganized manner. Many of these are single person run micro-businesses and cannot afford to buy and maintain their own IT infrastructure. For others, easy availability of cheap labour provides a convenient alternative even though it results in inefficiency, as little or no records are maintained, and only manual, paper-based processes are followed. This results in high response times for customers, no formal accountability and higher charges. For the businesses this translates to lower earnings and losses due to inefficiencies. In this paper, we look at few such micro-business segments and explore their current models of operation, while identifying existing inefficiencies and pain points. We build upon the findings and propose an approach for delivering benefits of IT solutions to such micro-business segments. Finally, we present technology that realizes the proposed approach in the specific context of two such segments.

In this paper, we present a system, called Sensei, for assessment of spoken English skills of call center agents. Sensei evaluates multiple parameters of spoken English skills, i.e., articulation of sounds, correctness of lexical stress in words and spoken grammar proficiency. Sensei provides an assessment test to be taken by a call center agent (or candidate) and generates score on each of the spoken English parameters as well as a combined score. It is implemented in the form of a web application so that it can be accessed through a web browser and doesn’t require any software to be installed at the client side. We describe how the individual parameters are assessed in Sensei using various speech processing techniques and the experiments conducted to evaluate these techniques. The performance is compared with assessment performed by human assessors. A correlation of 0.8 is obtained between overall score generated by Sensei and human assessors on a real life test dataset of 243 candidates which compares well with the corresponding human-to-human correlation of 0.91.

Utterance classification is an important task in spoken-dialog systems. The response of the system is dependent on category assigned to the speaker's utterance by the classifier. However, often the input speech is spontaneous and noisy which results in high word error rates. This results in unsatisfactory system performance. In this paper we describe a method to improve the natural language call classification task using statistical machine translation (SMT). We utilize the translation model in SMT to capture the relation between truth and the ASR transcribed text. The model is trained using the human transcribed text and the ASR transcribed text. During deployment SMT is used to sanitize the ASR transcribed text. Our experiments with IBM model 2 shows significant improvement in call classification accuracy.

Wepresent a technique for two-stream processing of speech signals for emotion detection. The first stream recognises emotion from acoustic features while the second stream recognises emotion from the semantics of the conversation. A probabilistic measure is derived for each of the individual streams and the emotion category from the two streams is recognised. The output of the two streams is combined to generate a score for a particular emotion category. The confidence level of each stream is used to weigh the scores from the two streams while generating the final score. This technique is extremely significant for call-center data that have some semantics associated with the speech. Theproposed technique is evaluated on theLDC corpus and on the real-word call-center data. Experiments suggest that use of a two-stream process provides better results than the existing techniques of extracting emotion only from acoustic features. Index Terms: emotion detection, text analytics, speech recognition

WWW has made information accessible to computer users in various ways not imagined before. However; there is a huge pool ofpeople, especially in emerging economies, still untouched by this revolution and are either unaware of, or are unable or to join this bandwagon. Mobile phones are increasingly empowering the under-privileged to utilize data and services beyond the basic voice communication. However; factors such as high illiteracy rate, cost sensitivity, and user interface issues prevent these users from deriving benefits of available infrastructure and services. We have developed a novel system - VOISERV that enables ordinary telephone subscribers to create, deploy and offer their own customized voice-driven applications called Voic-eSites. The generated VoiceSites get hosted in the network for low cost of ownership and maintenance, and are integrated with advanced services available in the converged networks of today.

Traditionally, voice-based applications have been accessed using unintelligent telephone devices through Voice Browsers that reside on the server. The proliferation of pervasive devices and the increase in their processing capabilities, client-side speech processing is emerging as a viable alternative. This workshop will explore the various possibilities and issues that arise while enabling speech processing on resource-constrained, possibly mobile devices. The workshop will highlight the many open areas that require research attention, identify key problems that need to be addressed, and also discuss a few approaches for solving some of them - to build the next generation of conversational systems.

The proliferation of pervasive devices has stimulated the development of applications that support ubiquitous access via multiple modalities. Since the processing capabilities of pervasive devices differ vastly, device-specific application adaptation becomes essential. We address the problem of speech application adaptation by dialog call-flow reorganisation for pervasive devices with different memory constraints. Given an atomic dialog call-flow A and device memory size m, we present optimal deterministic algorithms, RESEQUENCE and BALANCE-TREE, which minimise the number of questions in the reorganised output call-flow A/sub m/. Algorithms MASQ and MATREE produce C/sub m/, minimally distant from input call-flow A/sub m/ while accommodating the memory constraint m. These two minimisation criteria are capable of capturing various usability requirements important in dialog call-flow design. The following observation forms the cornerstone of all the algorithms in this paper: Two grammars g/sub 1/ and g/sub 2/ comprising of |g/sub 1/| and |g/sub 2/| elements respectively can be merged into a single grammar g = g/sub 1/ /spl times/ g/sub 2/ having |g/sub 1/|/spl middot/|g/sub 2/| elements for the sequential case, and g = g/sub 1/ + g/sub 2/ having |g/sub 1/|+|g/sub 2/| elements for the tree case. Device-speciific considerations lead us to introduce the concept of an-characterisation of a call-flow, defined as the set of pairs {(m/sub i/,q/sub i/)| /spl isin/ N}, where q/sub i/ is the minimum number of questions required for memory size m/sub i/. Each call-flow has a unique, device-independent signature in its-characterisation - a measure of its adaptability. We present SAMVAAD, a system that implements these algorithms on call-flows authored in VXML containing SRGS grammars. The system was tested on an IBM voice browser using a sample airline reservation system call-flow reorganised for memories ranging from 64 MB to 210 KB. We ran an experiment with 14 users to obtain feedback on the usability of the adapted call-flows.

This paper describes a morphing-based audio driven facial animation system. Based on an incoming audio stream, a face image is animated with full lip synchronization and synthesized expressions. A novel scheme to implement a language independent system for audio-driven facial animation given a speech recognition system for just one language, in our case, English, is presented. The method presented here can also be used for text to audio-visual speech synthesis. Visemes in new expressions are synthesized to be able to generate animations with different facial expressions. An animation sequence using optical flow between visemes is constructed, given an incoming audio stream and still pictures of a face representing different visemes. The presented techniques give improved lip synchronization and naturalness to the animated video.

Speech driven lip synthesis is an interesting and important step toward human-computer interaction. An incoming speech signal is time aligned using a speech recognizer to generate a phonetic sequence which is then converted to the corresponding viseme sequence to be animated. We present a novel method for generation of the viseme sequence, which uses viseme based acoustic models, instead of the usual phone based acoustic models, to align the input speech signal. This results in higher accuracy and speed of the alignment procedure and allows a much simpler implementation of the speech driven lip synthesis system as it completely obviates the requirement of an acoustic unit to visual unit conversion. We show, through various experiments, that the proposed method results in about 53% relative improvement in classification accuracy and about 52% reduction in the time required to compute alignments.

Speech drive lip synthesis is an interesting and important step toward human-computer interaction. An incoming speech signal is time aligned using a speech recognizer to generate phonetic sequence, which is then converted to corresponding viseme sequence to be animated. In this paper, we present a novel method for generation of the viseme sequence, which uses viseme based acoustic models, instead of usual phone based acoustic models, to align the input speech signal. This results in higher accuracy and speed of the alignment procedure and allows a much simpler implementation of the speech driven lip synthesis system as it completely obviates the requirement of acoustic unit to visual unit conversion. We show through various experiments that the proposed method results in about 53% relative improvement in classification accuracy and about 52% reduction in time, required to compute alignments.

In this paper, we demonstrate a morphing based automated audio driven facial animation system. Based on an incoming audio stream, a face image is animated with full lip synchronization and expression. An animation sequence using optical flow between visemes is constructed, given an incoming audio stream and still pictures of a face speaking different visemes. Rules are formulated based on coarticulation and the duration of a viseme to control the continuity in terms of shape and extent of lip opening. In addition to this new viseme-expression combinations are synthesized to be able to generate animations with new facial expressions. Finally various applications of this system are discussed in the context of creating audio-visual reality. 1.

Orthogonal information present in the video signal associated with the audio helps in improving the accuracy of a speech recognition system. Audio-visual speech recognition involves extraction of both the audio as well as visual features from the input signal. Extraction of visual parameters is done by the recognition of speech dependent features from the video sequence. The paper uses geometrical features to describe the lip shapes. Curve-based active shape models are used to extract the geometry. These geometrically represented visual parameters are used along with the audio cepstral features to perform an audio-visual classification. It is shown that the bimodal system presented gives an improvement in the classification results over classification using only the audio features.

We present a method for building an initial phoneme model for training an HMM in a new language using an already trained recognition system in a base language. HMM based phoneme recognition systems are used to model the phonemes in most large vocabulary speech recognition tasks. Mappings between the phonetic spaces of the two languages are generated and are used to populate the phonetic space of the new language. The best possible alignment of the new language data is obtained and initial phone models are built on this labeled data. A classification experiment is performed in the new language to illustrate the goodness of initial phone models. Experiments are carried out with Hindi as the new language using an English language recognition system to derive the initial phone models for Hindi language. 1.

In a continuous speech recognition system it is important to model the context dependent variations in the pronunciations of phones. In this work we have attempted to build decision trees for modeling phonetic context-dependency in Hindi. The approach followed is to modify a decision tree built to model context-dependency in American English. The reason the decision trees turn out to be different are that the English and Hindi phoneme sets are not identical. Then even for identical phonemes, the context-dependency is different for the two languages. Linguistic-Phonetic knowledge of Hindi is used to modify the English phone set. Since the Hindi phone set being used is derived from the English phone set, the adaptation of the English tree to Hindi follows naturally. Though here the adaptation is from English to Hindi, the method may be applicable for adapting between any two languages. The decision tree is built using either Hindi data or English data labeled with the correct Hindi contexts. This procedure is discussed and the limitations of both the methods are described. 1.

Considers the problem of combining visual cues with audio signals for the purpose of improved automatic machine recognition of speech. Although significant progress has been made in the machine transcription of large-vocabulary continuous speech (LVCSR) over the last few years, the technology to date is most effective only under controlled conditions, such as low noise, speaker-dependent recognition, read speech (as opposed to conversational speech), etc. On the other hand, while augmenting the recognition of speech utterances with visual cues has attracted the attention of researchers over the last couple of years, most efforts in this domain can be considered to be only preliminary in the sense that, unlike LVCSR efforts, tasks have been limited to small vocabularies (e.g. commands, digits) and often to speaker-dependent training or isolated word speech, where word boundaries are artificially well-defined.