Secure user authentication on mobile phones is crucial, as they store highly sensitive information. Common approaches to authenticate a user on a mobile phone are based either on entering a PIN, a password, or drawing a pattern. However, these authentication methods are vulnerable to the shoulder surfing attack. The risk of this attack has increased since means for recording high-resolution videos are cheaply and widely accessible. If the attacker can videotape the authentication process, PINs, passwords, and patterns do not even provide the most basic level of security. In this project, we assessed the vulnerability of a magnetic gestural authentication method to the video-based shoulder surfing attack. We chose a scenario that is favourable to the attack-er. In a real world environment, we videotaped the interactions of four users performing magnetic signatures on a phone, in the presence of HD cameras from four different angles. We then recruited 22 participants and asked them to watch the videos and try to forge the signatures. The results revealed that with a certain threshold, i.e, th=1.67, none of the forging attacks was successful, whereas at this level all eligible login attempts were successfully recognized. The qualitative feedback also indicated that users found the magnetic gestural signature authentication method to be more secure than PIN-based and 2D signature methods.

Around Device Interaction (ADI) is recently introduced in the field of Human Computer Interaction (HCI)to provide touch less, more intuitive way of interaction using space beyond the physical boundary of the computing devices. In this paper, we introduce a new ADI input device called Pingu in the form factor of a fingering that allows users to interact with any nearby computing device with wireless connectivity in a ubiquitous environment. Fingering form factor is chosen for our prototype design, as it is socially acceptable and is commonly worn in everyday social contexts, and based on the previous research, the information entropy of interaction by fingers is greater than the entropy for any other parts of the human body. The current Pingu prototype is consisted of an extensive set of sensors, visual and vibrot actile feedback mechanisms with wireless connectivity that make it a unique input device for human-computer or human-human interaction in the form of gestures, tactile and touch. Its usage can range from advanced, tiny and novel gestural interaction with a variety of devices to mobile and networked sensing, and social computing. We present a few potential applications of Pingu such as social interaction, context recognition, in-car interaction, and physical activity analysis.

Playing musical instruments such as chordophones, percussions and keyboard types accompany with harmonic interaction of player's hand with the instruments. In this work, we present a novel approach that enables the user to imitate the music playing gestures around mobile devices. In our approach, touch-less gestures, which change magnetic field around the device, are employed for interaction. The activity of playing an instrument can be transparently pursued by moving a tiny magnet in hand around new generation of mobile phones equipped with embedded digital compass (magnetic sensor). The phonation intentions of the user can be simulated on the mobile device by capturing the gestural pattern using magnetic sensor. The proposed method allows digital imitation of a broad number of instruments while still being able to sense musical hits and relative plectrum gestures. It provides a framework for extending interaction space with music applications beyond physical boundaries of small mobile devices, and to 3D space around the device. This can allow for a more natural, comfortable and flexible interaction. We present several mobile music applications developed based on the proposed method for Apple iPhone 3GS.

Models for the prediction of user judgments from interaction data can be used in different contexts such as system quality assessment, monitoring of deployed systems, or as a reward function in learned dialog managers. Such models still show a considerable lack with respect to their generalizability [6]. This paper specifically addresses this issue. We propose to use a Mixture of Experts approach for cross-database predictions. In Mixture of Experts, several classifiers are trained on subsets of the data showing specific characteristics. Predictions of each expert model are combined for the overall prediction result. We show that such an approach can improve the cross-database prediction accuracy. Index Terms: user judgments, quality prediction, evaluation, spoken dialog systems

The effect of masking due to fluctuating sources on speech intelligibility is a phenomenon difficult to predict. Intelligibility scores vary with the efficiency of the energetic masking while the linguistic content of the message and listener’s cognitive performances add to the general incertitude that peaks for the case of masking speech. The present contribution proposes a signal-based assessment of the energetic masking at the sentence level. A mapping onto the scale of the speech intelligibility index is established for stationary noise. Predictions are quantitatively compared with the results of an intelligibility test for speech-modulated noise. The model is independent of voices similarities and semantic features, two important sources of informational masking. Index Terms: speech intelligibility, automatic speech

In this paper, we propose a system and methodology for using mobile phones for monitoring physical activities of a user, and its applications in assisting elderly or people with need for special care and monitoring. The method is based on processing acceleration data provided by accelerometers integrated in new mobile phones. As the mobile phone is carried regularly by the user, the acceleration pattern can deliver information related to pattern of physical activities the user is engaged in. This information can be sent to a monitoring server, analyzed and presented as different health related factors for assistance, monitoring and healthcare purposes.

In this work, we present a system and methodology for using mobile phones for monitoring physical activities of a user, and its applications in assisting elderly and people with need for special care and monitoring. The method is based on processing acceleration data provided by accelerometers integrated in mobile phones. This information is sent to a monitoring server, analyzed and presented as different health related factors for assistance, monitoring and healthcare purposes. A monitoring agent can use a desktop application to observe pattern of physical activities of several users in a live manner, and receive warnings in case of unexpected physical conditions. The data can be also stored offline for longer term analysis of physical behaviour and health. The desktop application also provides different options for managing, browsing, and searching activity related data.

In this work, we present a method for controlling call alert functionality in mobile phones. It has happened for almost everybody experiencing a situation that call alert functionality is not proper for actual ambient context, leading to missing a phone call or disturbing others by a loud ring. In this work, we use audio and physical movement analysis to distinguish between different situations in which a mobile phone may ring, and adjust the call alert functionality accordingly. Considering the fact that mobile phones are usually carried in a pocket or bag, capturing ambient audio is not usually practically perfect. The novelty in our work is using information about physical movements of user of mobile device in addition to analysis of ambient audio. Analysis of user movements is based on information captured by acceleration sensors integrated in mobile phone. The call alert functionality is then adjusted based on a combination of ambient audio level and physical activities of user.

In this work, we present a new technique for efficient use of 3D space around a mobile device for interaction with the device. Around Device Interaction (ADI) enables extending interaction space of small mobile and tangible devices beyond their physical boundary. Our proposed method is based on using compass (magnetic field) sensor integrated in mobile devices (e.g. iPhone 3GS, G1 Android). In this method, a properly shaped permanent magnet (e.g. in the shape of a rod, pen or a ring) is used for interaction. The user makes coarse gestures in the 3D space around the device using the magnet. Movement of the magnet affects the magnetic field sensed by the compass sensor integrated in the device. The temporal pattern of the gesture is then used as a basis for sending different interaction commands to the mobile device. Zooming, turning pages, accepting/rejecting calls, clicking items, controlling a music player, and game interaction are some example use cases. The proposed method does not impose changes in hardware specifications of the mobile device, and unlike optical methods is not limited by occlusion problems.

We present a new technique based on using embedded compass (magnetic) sensor for efficient use of 3D space around a mobile device for interaction with the device. Around Device Interaction (ADI) enables extending interaction space of small mobile and tangible devices beyond their physical boundary. Our proposed method is based on using compass (magnetic field) sensor integrated in new mobile devices (e.g. iPhone 3GS, G1/2 Android). In this method, a properly shaped permanent magnet (e.g. a rod, pen or a ring) is used for interaction. The user makes coarse gestures in 3D space around the device using the magnet. Movement of the magnet affects magnetic field sensed by the compass sensor integrated in the device. The temporal pattern of the gesture is then used as a basis for sending different interaction commands to the mobile device. The proposed method does not impose changes in hardware and physical specifications of the mobile device, and unlike optical methods is not limited by occlusion problems. Therefore, it allows for efficient use of 3D space around device, including back of device. Zooming, turning pages, accepting/rejecting calls, clicking items, controlling a music player, and mobile game interaction are some example use cases. Initial evaluation of our algorithm using a prototype application developed for iPhone shows convincing gesture classification results.

In this work, we present a new approach for text (mainly digit) entry based on digit shaped gestures created in 3D space around a mobile device. Some new mobile devices such as Apple iPhone 3GS and Google Android are equipped with magnetic (compass) sensor. The main idea is to influence the magnetic sensor using a magnet taken in hand. The user draws (writes) digits in the 3D space around the device using the magnet taken in hand. Movement of the magnet changes temporal pattern of magnetic field around the device which is sensed and registered by the magnetic (compass) sensor. The registered pattern is then compared against already recorded templates for different digits. Such a text (digit) entry approach can be especially useful for small mobile devices in which it is hard to operate small buttons or touch screen. Using our technique, the text entry space extends beyond physical boundaries of the device. A demonstrator for this approach is implemented on Apple iPhone 3GS platform. It demonstrates registering a few templates for different digits, and recognizing digits written in the space around the device

Using phone posterior probabilities has been increasingly explored for improving automatic speech recognition (ASR) systems. In this paper, we propose two approaches for hierarchically enhancing these phone posteriors, by integrating long acoustic context, as well as phonetic and lexical knowledge. In the first approach, phone posteriors estimated with a multilayer perceptron (MLP), are used as emission probabilities in hidden Markov model (HMM) forward-backward recursions. This yields new enhanced posterior estimates integrating HMM topological constraints (encoding specific phonetic and lexical knowledge), and context. In the second approach, temporal contexts of the regular MLP posteriors are postprocessed by a secondary MLP, in order to learn inter- and intra-dependencies between the phone posteriors. These dependencies are phonetic knowledge. The learned knowledge is integrated in the posterior estimation during the inference (forward pass) of the second MLP, resulting in enhanced phone posteriors. We investigate the use of the enhanced posteriors in hybrid HMM/artificial neural network (ANN) and Tandem configurations. We propose using the enhanced posteriors as replacement, or as complementary evidences to the regular MLP posteriors. The proposed methods have been tested on different small and large vocabulary databases, always resulting in consistent improvements in frame, phone, and word recognition rates.

In this demo, we present an application for mobile phones which can monitor physical activities of users and detect unexpected emergency situations such as a sudden fall or accident. Upon detection of such an event, the mobile phone can inform a designated center (by automatically calling or sending message) about the incident and its location. This can facilitate and speed up recovery and help process especially if the user is alone or the accident has happened in a deserted place. Such an application can be particularly useful for elderly people or people with physical and movement disabilities. The application operates based on analysis of user movements using data provided by accelerometers integrated in mobile phones.

In this demo, we present an application for mobile phones which can analyse audio context and issue tactile or visual alerts if an audio event happens. This application can be useful especially for deaf or hard of hearing people to be alerted of audio events happening around them. The audio context analysis algorithm captures data using mobile phone's microphone and checks for changes in audio activities around the user. If such a change happens and also some other circumstances are met, the application issues visual or vibro-tactile alerts (by vibrating mobile phone) proportional to the change in audio context. This informs the user about an event. The functionality of this algorithm can be further enhanced by analysis of user movements.

This paper describes a system to detect angry vs. non-angry utterances of children who are engaged in dialog with an Aibo robot dog. The system was submitted to the Interspeech2009 Emotion Challenge evaluation. The speech data consist of short utterances of the children’s speech, and the proposed system is designed to detect anger in each given chunk. Frame-based cepstral features, prosodic and acoustic features as well as glottal excitation features are extracted automatically, reduced in dimensionality and classified by means of an artificial neural network and a support vector machine. An automatic speech recognizer transcribes the words in an utterance and yields a separate classification based on the degree of emotional salience of the words. Late fusion is applied to make a final decision on anger vs. non-anger of the utterance. Preliminary results show 75.9% unweighted average recall on the training data and 67.6 % on the test set. Index Terms: speech processing, meta-data extraction, emotion recognition, evaluation

In this paper, we propose a method for detecting physical shock by a mobile phone and its applications in security and emergency scenarios. We use acceleration signal provided by accelerometers integrated in a mobile phone to detect shocks. The shock is identified by analysis of acceleration pattern using statistical signal processing methods. We propose to use such a shock detection application for increasing security of data access in a mobile phone, as well as identifying scenarios which user of mobile phone experiences risky emergency events.

Phone posteriors has recently quite often used (as additional features or as local scores) to improve state-of-the-art automatic speech recognition (ASR) systems. Usually, better phone posterior estimates yield better ASR performance. In the present paper we present some initial, yet promising, work towards hierarchically improving these phone posteriors, by implicitly integrating phonetic and lexical knowledge. In the approach investigated here, phone posteriors estimated with a multilayer perceptron (MLP) and short (9 frames) temporal context, are used as input to a second MLP, spanning a longer temporal context (e.g. 19 frames of posteriors) and trained to refine the phone posterior estimates. The rationale behind this is that at the output of every MLP, the information stream is getting simpler (converging to a sequence of binary posterior vectors), and can thus be further processed (using a simpler classifier) by looking at a larger temporal window. Longer term dependencies can be interpreted as phonetic, sub-lexical and lexical knowledge. The resulting enhanced posteriors can then be used for phone and word recognition, in the same way as regular phone posteriors, in hybrid HMM/ANN or Tandem systems. The proposed method has been tested on TIMIT, OGI Numbers and Conversational Telephone Speech (CTS) databases, always resulting in consistent and significant improvements in both phone and word recognition rates.

In this paper, we present a method for integrating possible prior knowledge (such as phonetic and lexical knowledge), as well as acoustic context (e.g., the whole utterance) in the phone posterior estimation, and we propose to use the obtained posteriors as complementary posterior features in Tandem ASR configuration. These posteriors are estimated based on HMM state posterior probability definition (typically used in standard HMMs training). In this way, by integrating the appropriate prior knowledge and context, we enhance the estimation of phone posteriors. These new posteriors are called ?in-context? or HMM posteriors. We combine these posteriors as complementary evidences with the posteriors estimated from a Multi Layer Percep- tron (MLP), and use the combined evidence as features for training and inference in Tandem configuration. This approach has improved the performance, as compared to using only MLP estimated posteriors as features in Tandem, on OGI Numbers , Conversational Telephone speech (CTS), and Wall Street Journal (WSJ) databases.

Over the years, sophisticated techniques for utilizing the prior knowledge in the form of text-derived language model and in pronunciation lexicon evolved. However, their use has an undesirable effect: unexpected lexical items (words) in the phrase are replaced by acoustically acceptable in-vocabulary items [1]. This is the major source of error since the replacement often introduces additional errors [2, 3]. Improving the machine ability to handle these unexpected words would considerably increase the utility of speech recognition technology.

In this paper, we present initial investigations towards boosting posterior probability based speech recognition systems by estimating more informative posteriors taking into account acoustic context (e.g., the whole utterance), as well as possible prior information (such as phonetic and lexical knowledge). These posteriors are estimated based on HMM state posterior probability definition (typically used in standard HMMs training). This approach provides a new, principled, theoretical framework for hierarchical estimation/use of more informative posteriors integrating appropriate context and prior knowledge. In the present work, we used the resulting posteriors as local scores for decoding. On the OGI numbers database, this resulted in significant performance improvement, compared to using MLP estimated posteriors for decoding (hybrid HMM/ANN approach) for clean and more specially for noisy speech. The system is also shown to be much less sensitive to tuning factors (such as phone deletion penalty, language model scaling) compared to the standard HMM/ANN and HMM/GMM systems, thus practically it does not need to be tuned to achieve the best possible performance

In this paper, we propose a new posterior based scoring approach for keyword and non keyword (garbage) elements. The estimation of these scores is based on HMM state posterior probability definition, taking into account long contextual information and the prior knowledge (e.g. keyword model topology). The state posteriors are then integrated into keyword and garbage posteriors for every frame. These posteriors are used to make a decision on detection of the keyword at each frame. The frame level decisions are then accumulated (in this case, by counting) to make a global decision on having the keyword in the utterance. In this way, the contribution of possible outliers are minimized, as opposed to the conventional Viterbi decoding approach which accumulates likelihoods. Experiments on keywords from the Conversational Telephone Speech (CTS) and Numbers'95 databases are reported. Results show that the new scoring approach leads to better trade off between true and false alarms compared to the Viterbi decoding approach, while also providing the possibility to precalculate keyword specific spotting thresholds related to the length of the keywords.

In this paper we propose a methodology for selecting the most discriminative features in a set for online signature verification. We expose the difference in the definition of class between signature verification and other pattern recognition tasks, and extend the classical Fisher ratio to make it more robust to the small sample sizes typically found when dealing with global features and client enrollment time constraints for signature verification systems. We apply our methodology to global and local features extracted from a 50-users database, and find that our criterion agrees better with classifier error rates for local features than for global features. We discuss the possibility of performing feature selection without having forgery data available.

Local state or phone posterior probabilities are often investigated as local scores (e.g., hybrid HMM/ANN systems) or as transformed acoustic features (e.g., “Tandem”) to improve speech recognition systems. In this paper, we present initial results towards boosting these approaches by improving posterior estimates, using acoustic context (e.g., as available in the whole utterance), as well as possible prior information (such as topological constraints). In the present work, the enhanced posterior distribution is associated with the “gamma ” distribution typically used in standard HMMs training, and estimated from local likelihoods (GMM) or local posteriors (ANN). This approach results in a family of new HMM based systems, where only posterior probabilities are used, while also providing a new, principled, approach towards a hierarchical use/integration of these posteriors, from the frame level up to the phone and word levels, and integrating the appropriate context and prior knowledge in each level. In the present work, we used the resulting posteriors as local scores in a Viterbi decoder. On the OGI Numbers’95 database, this resulted in improved recognition performance, compared to a state-of-the-art hybrid HMM/ANN system. 1.