Misinformation can spread rapidly in everyday conversation, where pausing to verify is not always possible. We envision a wearable system that bridges the timing gap between hearing a claim and forming a judgment. It uses ambient listening to detect verifiable claims, performs rapid web verification, and provides a subtle haptic nudge with a glanceable overview. A controlled study (N=34) simulated this approach and tested against a no-support baseline. Results show that instant, body-integrated feedback significantly improved real-time truth discernment and increased verification activity compared to unsupported fact-checking. However, it also introduced over-reliance when the system made errors, i.e. failed to flag false claims or flagged true claims as false. We contribute empirical evidence of improved discernment alongside insights into trust, effort, and user–system tensions in verification wearables.

Current scene perception tools for Blind and Low Vision (BLV) individuals rely on spoken descriptions but lack engaging representations of visually pleasing distant environmental landscapes (Vista spaces). Our proposed Scene2Audio framework generates comprehensible and enjoyable nonverbal audio using generative models informed by psychoacoustics, and principles of scene audio composition. Through a user study with 11 BLV participants, we found that combining the Scene2Audio sounds with speech creates a better experience than speech alone, as the sound effects complement the speech making the scene easier to imagine. A mobile app “in-the-wild” study with 7 BLV users for more than a week further showed the potential of Scene2Audio in enhancing outdoor scene experiences. Our work bridges the gap between visual and auditory scene perception by moving beyond purely descriptive aids, addressing the aesthetic needs of BLV users.

Gastric interoception influences eating behavior and emotions, making its modulation valuable for healthcare and human-computer-interaction applications. However, whether gastric interoception can be modulated noninvasively in humans remains unclear. While previous research indicates that abdominal-sound-driven haptic feedback resembles gut sensations, its impact on gastric feelings and gastric interoceptive behavior is unknown. We conducted three experiments totalling 55 participants to investigate how gut-sound-driven audio-haptic feedback applied to the stomach (1) affects user’s feelings (2) influences perception of hunger and satiety levels and (3) influences gastric interoceptive behavior, quantified with Water Load Test-II. Results revealed that audio-haptic feedback patterns (a) induced the feelings of hunger, fullness, thirst, stomach upset, (b) increased hunger level, and (c) significantly increased volumes of ingested water. This work provides the first evidence that audio-haptic stimulation can alter gastric interoceptive behavior, motivating the use of noninvasive methods to influence users’ feelings and behaviors in future applications.

Stress and poor sleep remain pervasive challenges in modern life, yet traditional relaxation practices such as pranayama (breathing exercises) require guidance, discipline, and environments that are often difficult to sustain. VR–based relaxation tools have emerged as alternatives, but their abrupt immersion into fully virtual environments can feel disruptive and misaligned with the gradual nature of meditative practices. To address this gap, we collaborated with pranayama practitioners in a co-design process to develop Zenflow, an MR system that blends subtle visuals and breathing cues to gradually transform the user’s surroundings into a restorative virtual space. We evaluated the system in a 3 week within-subjects study (N=12), comparing traditional Pranayama with two variations of Zenflow. Results show that Zenflow transition design significantly improved self-reported sleep quality and objective measures of stress and sleep. Our work contributes design insights and evidence that gradual environmental transition can improve MR systems for stress management.

Drones have gained traction as a versatile form of assistive robots for Blind and Low Vision (BLV) people. Nonetheless, novel interaction techniques are required to enable BLV people to communicate with drones naturally. In this work, we built an LLM-powered assistive drone for BLV users. We leverage an LLM to translate high-level user goals to step-by-step instructions for the drone and to extract visual information from the images. Through a formative study with BLV users (N=9), we identified envisioned use cases and desired interaction modalities. Then, we took a participatory and iterative approach to build a prototype, incorporating feedback received from 3 BLV users, as well as 5 domain experts. Finally, we conducted a user study with an additional 6 BLV participants to evaluate the iterated prototype, and received positive feedback. This work is contributing to a growing body of research on harnessing the power of LLMs to build a more inclusive world.

Always-on sensing of AI applications on AR glasses makes traditional permission techniques inefficient for context-dependent private visual data within home environments. Home presents a challenging privacy context due to massive sensitive objects and the intimate nature of daily routines. We propose VisGuardian, a fine-grained content-based visual permission technique for AR glasses. VisGuardian features a group-based control mechanism that enables users to efficiently manage permissions for multiple private objects. VisGuardian detects objects using YOLO and adopts a pre-classified schema to group them. By selecting a single object, users can obscure groups of related objects based on criteria including privacy sensitivity, object category, or spatial proximity. A technical evaluation shows VisGuardian achieves mAP50 of 0.6704 with only 14.0 ms latency and a 1.7% increase in battery consumption per hour. Furthermore, a user study (N=24) comparing VisGuardian to slider-based and object-based baselines found it to be significantly faster for setting permissions and was preferred by users for its efficiency, effectiveness, and ease of use.

To support seamless mixed reality (MR) interactions without obstructing natural touch, we introduce PhantomFolds, a nail-mounted device featuring two linear resonant actuators (LRAs) at the lateral nail folds that provide spatio-tactile feedback for in-air as well as on-surface interactions in MR. In three studies, we investigate the perception of spatio-tactile feedback produced by PhantomFolds. Our results show that (1) PhantomFolds can leverage the funneling illusion to produce spatio-tactile feedback at the finger for in-air as well as on-surface interactions, (2) tactile feedback produced with PhantomFolds can successfully increase the perceived realism in MR applications without impeding user interaction, (3) phantom sensations, are perceived more localized when users touch a surface, and (4) while participants could not consistently feel touch illusions below the finger when touching a surface as described by prior work, our results indicate that a per-user calibration could increase the success rate of this effect in the future.

We introduce Prompt-to-Touch, a proof-of-concept of a multi-step pipeline that generates haptic effects based on the textual description of a haptic experience. Our pipeline first translates the haptic effect description to a sound effect description using our Foley-Interpreter component. It then uses a text-to-audio model to generate a sound effect from the sound description. Afterwards, the sound effect is converted into a perceivable haptic effect using our Dynamic-Audio-Processor . Finally, the haptic effect is post-processed to compensate for actuator-specific frequency response characteristics. We validate our concept in two preliminary human evaluation studies (n = 20, n = 10) and a technical analysis. Our results indicate that our pipeline can generate effects to enhance immersive multimedia experiences, abstract desktop/XR interactions, and social communication applications. They also still reveal significant potential for further improvement. Our pipeline could be used to develop future text-driven haptic design and automation tools. We provide open-source code to support future extensions.

International audience

Sound morphing is the process of gradually and smoothly transforming one sound into another to generate novel and perceptually hybrid sounds that simultaneously resemble both. Recently, diffusion-based text-to-audio models have produced high-quality sounds using text prompts. However, granularly controlling the semantics of the sound, which is necessary for morphing, can be challenging using text. In this paper, we propose MorphFader, a controllable method for morphing sounds generated by disparate prompts using text-to-audio models. By intercepting and interpolating the components of the cross-attention layers within the diffusion process, we can create smooth morphs between sounds generated by different text prompts. Using both objective metrics and perceptual listening tests, we demonstrate the ability of our method to granularly control the semantics in the sound and generate smooth morphs.

In today's digital world, computing education offers critical opportunities, yet systemic inequities exclude under-represented communities, especially in rural, under-resourced regions. Early engagement is vital for building interest in computing careers and achieving equitable participation. Recent work has shown that the use of sensor-enabled tools and block-based programming can improve engagement and self-efficacy for students from under-represented groups, but these findings lack replication in diverse, resource-constrained settings. This study addresses this gap by implementing sensor-based programming workshops with rural students in Sri Lanka. Replicating methods from the literature, we conduct a between-group study (sensor vs. non-sensor) using Scratch and real-time environmental sensors. We found that students in both groups reported significantly higher confidence in programming in Scratch after the workshop. In addition, average changes in both self-efficacy and outcome expectancy were higher in the experimental (sensor) group than in the control (non-sensor) group, mirroring trends observed in the original study being replicated. We also found that using the sensors helped to enhance creativity and inspired some students to express an interest in information and communications technology (ICT) careers, supporting the value of such hands-on activities in building programming confidence among under-represented groups.

Unmanned Aerial Vehicles (UAVs) or drones, are increasingly used in search and rescue missions to detect human presence. Existing systems primarily leverage vision-based methods which are prone to fail under low-visibility or occlusion. Drone-based audio perception offers promise but suffers from extreme ego-noise that masks sounds indicating human presence. Existing datasets are either limited in diversity or synthetic, lacking real acoustic interactions, and there are no standardized setups for drone audition. To this end, we present DroneAudioset (The dataset is publicly available at https://huggingface.co/datasets/ahlab-drone-project/DroneAudioSet/ under the MIT license), a comprehensive drone audition dataset featuring 23.5 hours of annotated recordings, covering a wide range of signal-to-noise ratios (SNRs) from -57.2 dB to -2.5 dB, across various drone types, throttles, microphone configurations as well as environments. The dataset enables development and systematic evaluation of noise suppression and classification methods for human-presence detection under challenging conditions, while also informing practical design considerations for drone audition systems, such as microphone placement trade-offs, and development of drone noise-aware audio processing. This dataset is an important step towards enabling design and deployment of drone-audition systems.

Productive Failure (PF) is a learning approach where students initially tackle novel problems targeting concepts they have not yet learned, followed by a consolidation phase where these concepts are taught. Recent application in STEM disciplines suggests that PF can help learners develop more robust conceptual knowledge. However, empirical validation of PF for programming education remains under-explored. In this paper, we investigate the use of PF to teach Python lists to undergraduate students with limited prior programming experience. We designed a novel PF-based learning activity that incorporated the unobtrusive collection of real-time heart-rate data from consumer-grade wearable sensors. This sensor data was used both to make the learning activity engaging and to infer cognitive load. We evaluated our approach with 20 participants, half of whom were taught Python concepts using Direct Instruction (DI), and the other half with PF. We found that although there was no difference in initial learning outcomes between the groups, students who followed the PF approach showed better knowledge retention and performance on delayed but similar tasks. In addition, physiological measurements indicated that these students also exhibited a larger decrease in cognitive load during their tasks after instruction. Our findings suggest that PF-based approaches may lead to more robust learning, and that future work should investigate similar activities at scale across a range of concepts.

Designing effective haptic feedback is challenging due to the subjective nature of touch and the fact that multiple people cannot easily share and evaluate touch experiences. In this work, we propose CoTacs, a collaborative haptic toolkit to address the challenges of haptic feedback design by enabling designers to explore haptic experiences together using group collaboration and allowing them to refine ideas quickly through early feedback during the design process. Two design sessions with five collaborators each show that CoTacs enables users to design, evaluate, refine and explore haptic feedback together. Participants leveraged group feedback to improve their designs and inspire creative ideas for the second collaboration round. Our work demonstrates how collaborative haptic toolkits can enable synergistic effects through interactions between the designers, benefiting haptic feedback design while still exposing several limitations. We discuss different opportunities for the design of future collaborative haptic toolkits and haptic group collaboration techniques.

Autistic children may often experience challenges in mastering daily living skills crucial for their independence and well-being. This study introduces “Tidd,” an augmented tabletop interactive system designed to assist autistic children in practising daily living skills in an engaging and physically interactive environment. We conducted a user study in a medical rehabilitation centre and an integrated kindergarten. Seventeen autistic children aged three to five years used Tidd in training sessions covering two vital skills: bed-making and dressing. Progress was evaluated through task completion and progress tracking, observational data for children, and therapist qualitative feedback. Therapists reported that Tidd was beneficial in maintaining the children’s attention and enhancing their motivation. Observational data further suggested increased engagement and decreased frustration during tasks. This study with Tidd highlights the potential of tabletop interaction to support therapists in training autistic children to learn daily living skills. • Tidd: AR tabletop device aiding autistic children in bed-making and dressing. • Improved task accuracy and engagement in study with 17 autistic children. • Therapists validated Tidd’s practicality and appeal in autism interventions.

Programming has become a highly sought-after skill in STEM-related studies and careers, but it has only reached a fraction of students with visual impairments. Therefore, there is a need to explore new methods for teaching and learning. This study aims to understand the potential of using drones to create an authentic learning environment to help students with visual impairments learn programming. Based on a month-long engagement with five students with visual impairments, we present insights on using drones to support programming education for students with visual impairments.

The practice of sound design involves creating and manipulating environmental sounds for music, films, or games. Recently, an increasing number of studies have adopted generative AI to assist in sound design co-creation. Most of these studies focus on the needs of novices, and less on the pragmatic needs of sound design practitioners. In this paper, we aim to understand how generative AI models might support sound designers in their practice. We designed two interactive generative AI models as Creative Support Tools (CSTs) and invited nine professional sound design practitioners to apply the CSTs in their practice. We conducted semi-structured interviews and reflected on the challenges and opportunities of using generative AI in mixed-initiative interfaces for sound design. We provide insights into sound designers’ expectations of generative AI and highlight opportunities to situate generative AI-based tools within the design process. Finally, we discuss design considerations for human-AI interaction researchers working with audio.

People with a fear of being in water rarely engage in water activities and hence miss out on the associated health benefits. Prior research suggested virtual exposure to treat fears. However, when it comes to a fear of being in water, virtual water might not capture water’s immersive qualities, while real water can pose safety risks. We propose extended reality to combine both advantages: We conducted a study (N=12) where participants with a fear of being in water interacted with playful water-inspired virtual reality worlds while floating inside a floatation tank. Our findings, supported quantitatively by heart rate variability and qualitatively by interviews, suggest that playful extended reality could mitigate fear responses in an entertaining way. We also present insights for the design of future systems that aim to help people with a fear of being in water and other phobias by using the best of the virtual and physical worlds.

In collaborative settings where multiple individuals are tasked with completing a shared goal, understanding one’s partner’s emotional state could be crucial for achieving a successful outcome. This is particularly relevant in remote collaboration contexts, where physical distance can impede understanding, empathy, and mutual comprehension between partners. In this paper, we demonstrate representing emotional patterns from physiological data in a shared Virtual Reality (VR) environment, and explore how it impacted communication styles. A user study investigated the potential effects of this emotional representation in fostering empathetic communication during remote collaboration. The study’s findings revealed that although there was minimal variance in the workload associated with observing physiological cues, participants generally preferred monitoring their partner’s attentional state. However, with the assembly task chosen, most participants only directed a minimal proportion of their attention toward the physiological cues displayed by their partner, and were frequently uncertain of how to interpret and use the information obtained. We also discuss limitations of the research and opportunities for future work.

Technology integration in educational settings has led to the development of novel sensor-based tools that enable students to measure and interact with their environment. Although reports from using such tools can be positive, evaluations are often conducted under controlled conditions and short timeframes. There is a need for longitudinal data collected in realistic classroom settings. However, sustained and authentic classroom use requires technology platforms to be seen by teachers as both easy to use and of value. We describe our development of a sensor-based platform to support science teaching that followed a 14-month design process. We share insights from this design and development approach, and report findings from a six-month large-scale evaluation involving 35 schools and 1245 students. We share lessons learnt, including that technology integration is not an educational goal per se and that technology should be a transparent tool to enable students to achieve their learning goals.

Journaling is a widely adopted technique, known to improve mental health and well-being by enabling reflection on past events. Large amounts of text in digital journaling applications could hinder the reflection process due to information overload. Abstractive summarization can solve this problem by generating short summaries to quickly glance at and reminisce. In this paper, we present an investigation of the utility of large language models in the context of autobiographical text summarization. We study two approaches to adapt a self-supervised learning (SSL) model to the domain of autobiographical text. One model employs transfer learning using our new autobiographical text summary dataset to fine-tune the SSL model. The second model leverages existing news datasets for high-quality text summarization mixed with our autobiographical summary dataset. We conducted mixed methods research to analyze the performance of these two models. Through objective evaluation using ROUGE and BART scores, we find that both these approaches perform significantly better than the SSL model fine-tuned with only high-quality news datasets, showing the importance of domain adaptation and autobiographical text summary dataset for this task. Secondly, through a subjective evaluation on a crowd-sourcing platform, we evaluated the summaries generated from these models on various quality criteria such as grammar, non-redundancy, structure, and coherence. We found that on all criteria, these summaries score >4 out of 5, and the two models show comparable results. We deployed a proof-of-concept web-based journaling application to assess the practical real-world implications of incorporating abstractive summarization in a digital journaling context. We found that the participants showed a high consensus that the summaries generated by the system captured the main idea of their journal entry (80% of the 75 participants gave a Likert scale rating of ≥5.0 out of 7.0, with the overall mean rating of 5.56 ± 1.32) while being factually correct, and they found it to be a useful feature of a journaling application. Finally, we conducted human evaluation studies to compare the quality of the summaries generated from a commercial tool ChatGPT and mixed distribution fine-tuned SSL model, and present insights into these systems in the context of autobiographical abstractive text summarization. We have made our model, dataset, and subjective evaluation questionnaire openly available to the research community.

Controllable generation in StyleGANs is usually achieved by training the model using labeled data. For audio textures, however, there is currently a lack of large semantically labeled datasets. Therefore, to control generation, we develop a method for semantic control over an unconditionally trained StyleGAN in the absence of such labeled datasets. In this paper, we propose an example-based framework to determine guidance vectors for audio texture generation based on user-defined semantic attributes. Our approach leverages the semantically disentangled latent space of an unconditionally trained StyleGAN. By using a few synthetic examples to indicate the presence or absence of a semantic attribute, we infer the guidance vectors in the latent space of the StyleGAN to control that attribute during generation. Our results show that our framework can find user-defined and perceptually relevant guidance vectors for controllable generation for audio textures. Furthermore, we demonstrate an application of our framework to other tasks, such as selective semantic attribute transfer.

Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches would certainly benefit from an accurately labeled, real-world, diverse dataset that enables the development of generalizable ML models. We introduce 'VR.net', a dataset comprising 165-hour gameplay videos from 100 real-world games spanning ten diverse genres, evaluated by 500 participants. VR.net accurately assigns 24 motion sickness-related labels for each video frame, such as camera/object movement, depth of field, and motion flow. Building such a dataset is challenging since manual labeling would require an infeasible amount of time. Instead, we implement a tool to automatically and precisely extract ground truth data from 3D engines' rendering pipelines without accessing VR games' source code. We illustrate the utility of VR.net through several applications, such as risk factor detection and sickness level prediction. We believe that the scale, accuracy, and diversity of VR.net can offer unparalleled opportunities for VR motion sickness research and beyond.We also provide access to our data collection tool, enabling researchers to contribute to the expansion of VR.net.

Science literacy is an increasingly important skill in the 21st century. With engagement and motivation as vital precursors to learning science, we believe introducing children to playful interactions with scientific phenomena would improve their motivations and attitudes towards science. To investigate this, we developed a tablet application where children journey in a story-driven game to capture virtual creatures by manipulating the sound measured using the built-in microphone. This game was designed with feedback from 16 children and 10 parents. In this paper, we describe the iterative design process and findings in a multi-day study with 11 more children aged between 8 and 12. Children were motivated by the game, demonstrated a strong association between sound and its behaviour in the physical world, and expressed enthusiasm to learn more in the classroom.

In this paper, we propose a data-driven approach to train a Generative Adversarial Network (GAN) conditioned on "soft-labels" distilled from the penultimate layer of an audio classifier trained on a target set of audio texture classes. We demonstrate that interpolation between such conditions or control vectors provide smooth morphing between the generated audio textures, and show similar or better audio texture morphing capability compared to the state-of-the-art methods. The proposed approach results in a well-organized latent space that generates novel audio outputs while remaining consistent with the semantics of the conditioning parameters. This is a step towards a general data-driven approach to designing generative audio models with customized controls capable of traversing out-of-distribution regions for novel sound synthesis.

Knowledge of programming and computing is becoming increasingly valuable in today's world, and thus it is crucial that students from all backgrounds have the opportunity to learn. As the teaching of computing at high-school becomes more common, there is a growing need for approaches and tools that are effective and engaging for all students. Especially for students from groups that are traditionally underrepresented at university level, positive experiences at high-school can be an important factor for their future academic choices. In this paper we report on a hands-on programming workshop that we ran over multiple sessions for Maori and Pasifika high-school students who are underrepresented in computer science at the tertiary level in New Zealand. In the workshop, participants developed Scratch programs starting from a simple template we provided. In order to control the action in their programs, half of the participants used standard mouse and keyboard inputs, and the other half had access to plug-and-play sensors that provided real-time environmental data. We explore how students' perceptions of self-efficacy and outcome expectancy -- both key constructs driving academic career choices -- changed during the workshop and how these were impacted by the availability of the sensor toolkit. We found that participants enjoyed the workshop and reported improved self-efficacy with or without use of the toolkit, but outcome expectancy improved only for students who used the sensor toolkit.

Novel AI-generated audio samples are evaluated for descriptive qualities such as the smoothness of a morph using crowdsourced human listening tests. However, the methods to design interfaces for such experiments and to effectively articulate the descriptive audio quality under test receive very little attention in the evaluation metrics literature. In this paper, we explore the use of visual metaphors of image-schema to design interfaces to evaluate AI-generated audio. Furthermore, we highlight the importance of framing and contextualizing a descriptive audio quality under measurement using such constructs. Using both pitched sounds and textures, we conduct two sets of experiments to investigate how the quality of responses vary with audio and task complexities. Our results show that, in both cases, by using image-schemas we can improve the quality and consensus of AI-generated audio evaluations. Our findings reinforce the importance of interface design for listening tests and stationary visual constructs to communicate temporal qualities of AI-generated audio samples, especially to naïve listeners on crowdsourced platforms.

Corneal Surface Reflections, or reflections on our eye-surface, have been shown as a valid and more socially acceptable source of information for passive lifelogging applications by prior work. However, automatic analysis of corneal surface reflections from a single RGB camera to support passive lifelogging is not extensively investigated in prior work. To address this, we developed a synthetic and self-supervised learning-based two-stage pipeline of deep learning models to detect objects in these reflections. Our prototype only consists a single RGB camera looking into the eye. We collected data from different users in uncontrolled environments using the prototype and trained our system to detect multiple classes of objects present in a typical office environment. We then evaluated our model in partially-controlled and in-the-wild scenarios. In addition, based on the findings from a follow up user study and prior work, we discuss strengths and weaknesses of our system and using corneal surface reflections for passive lifelogging. Finally, we opensource our source codes and trained checkpoints.

Abstract Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and generator components of RAG for the task of domain adaptation in ODQA. We propose RAG-end2end, an extension to RAG that can adapt to a domain-specific knowledge base by updating all components of the external knowledge base during training. In addition, we introduce an auxiliary training signal to inject more domain-specific knowledge. This auxiliary signal forces RAG-end2end to reconstruct a given sentence by accessing the relevant information from the external knowledge base. Our novel contribution is that, unlike RAG, RAG-end2end does joint training of the retriever and generator for the end QA task and domain adaptation. We evaluate our approach with datasets from three domains: COVID-19, News, and Conversations, and achieve significant performance improvements compared to the original RAG model. Our work has been open-sourced through the HuggingFace Transformers library, attesting to our work’s credibility and technical consistency.

Open source hardware such as RISC-V has been gaining substantial momentum. Recently, they have begun to embrace Google’s Android operating system to leverage its software ecosystem. Despite the encouraging progress, a challenging issue arises: a majority of Android applications are written in native languages and need to be recompiled to target new hardware platforms. Unfortunately, this recompilation process is not scalable because of the explosion of new hardware platforms. To address this issue, we present WasmAndroid, a high-performance cross-platform runtime for native Android applications. With WasmAndroid, developers can compile their source code to WebAssembly, an efficient and portable bytecode format that can be executed everywhere without additional reconfiguration. Developers can also transpile existing application binaries to WebAssembly when source code is not available. WebAssembly’s language model is very different from other common languages. This mismatch leads to many unique implementation challenges. In this article, we provide workable solutions and conduct a thorough system evaluation. We show that WasmAndroid provides acceptable performance to execute native applications in a cross-platform manner.

This paper presents XRtic, a prototyping toolkit enabling real-world cloth deformations to be used in novel ways in eXtended Reality (XR) applications. XRtic was developed based on the insights gathered from semi-structured interviews with XR developers. It consists of custom-made actuators that can be attached to regular clothing, a controller bus system, and a controller interface. Using our toolkit, users can design and integrate different cloth deformation types synchronised with virtual content in a plug-and-play manner. Along with a technical analysis of the actuation behaviour of the XRtic actuators, we present the findings gathered from a user study with eight XR developers, focusing on the usability of the system and creative support. Overall, participants found it an easy-to-use toolkit that supports iterative and rapid prototyping, and enables cloth to be deformed in unique ways in synchronisation with XR applications. Based on the findings, we also report limitations and future work relating to our system.

Emotions are complicated psycho-physiological processes that are related to numerous external and internal changes in the body. They play an essential role in human-human interaction and can be important for human-machine interfaces. Automatically recognizing emotions in conversation could be applied in many application domains like health-care, education, social interactions, entertainment, and more. Facial expressions, speech, and body gestures are primary cues that have been widely used for recognizing emotions in conversation. However, these cues can be ineffective as they cannot reveal underlying emotions when people involuntarily or deliberately conceal their emotions. Researchers have shown that analyzing brain activity and physiological signals can lead to more reliable emotion recognition since they generally cannot be controlled. However, these body responses in emotional situations have been rarely explored in interactive tasks like conversations. This paper explores and discusses the performance and challenges of using brain activity and other physiological signals in recognizing emotions in a face-to-face conversation. We present an experimental setup for stimulating spontaneous emotions using a face-to-face conversation and creating a dataset of the brain and physiological activity. We then describe our analysis strategies for recognizing emotions using Electroencephalography (EEG), Photoplethysmography (PPG), and Galvanic Skin Response (GSR) signals in subject-dependent and subject-independent approaches. Finally, we describe new directions for future research in conversational emotion recognition and the limitations and challenges of our approach.

Block-based programming environments are a popular way to introduce programming as they provide helpful visual cues and remove the complexities of syntax, allowing learners to focus on being creative. However, input to programs in block-based environments is often limited to the keyboard and mouse, meaning programs respond only to the direct actions of the user. Allowing programs to respond to changes in the physical environment may influence the types of programs students create and their motivation and interest towards learning programming. We explore this idea by integrating real-time sensor data into Scratch. With our platform, students simply connect a sensor to their computer via USB and use custom blocks, alongside conventional Scratch code, to read and react to the sensor data in real-time. We evaluated this approach using a field study involving 25 students with limited prior experience in Scratch. We found students were highly motivated by the sensors and expressed a strong desire to use them in future projects. Analysis of the created programs revealed that, compared to using standard input sources, students explored a wider variety of Scratch blocks when controlling sprites using the sensors.

Despite the increasing popularity of VR games, one factor hindering the industry’s rapid growth is motion sickness experienced by the users. Symptoms such as fatigue and nausea severely hamper the user experience. Machine Learning methods could be used to automatically detect motion sickness in VR experiences, but generating the extensive labeled dataset needed is a challenging task. It needs either very time consuming manual labeling by human experts or modification of proprietary VR application source codes for label capturing. To overcome these challenges, we developed a novel data collection tool, VRhook, which can collect data from any VR game without needing access to its source code. This is achieved by dynamic hooking, where we can inject custom code into a game’s run-time memory to record each video frame and its associated transformation matrices. Using this, we can automatically extract various useful labels such as rotation, speed, and acceleration. In addition, VRhook can blend a customized screen overlay on top of game contents to collect self-reported comfort scores. In this paper, we describe the technical development of VRhook, demonstrate its utility with an example, and describe directions for future research.

Emotional Self-Awareness (ESA) plays a vital role in physical and mental well-being. Recent advancements in artificial intelligence technologies have shown promising emotion recognition results, opening new opportunities to build systems to support ESA. However, little research has been done to understand users' perspectives on artificial-intelligence-based emotion recognition systems. We introduce Troi, an automatic emotion recognition mobile app using wearable signals. With Troi, we ran a multi-day user study with 12 users to understand user preference parameters, such as perceived accuracy, confidence, preferred emotion representations, effect of self-awareness of emotions, and real-time use cases. Further, we extend our study to evaluate the machine learning model in-the-wild to understand behaviours in-the-wild. We found that users perceived accuracy of the emotion recognition model is higher than the actual model prediction accuracy; there was no strong preference for one specific emotion representation, and users' self-awareness of emotions improved over time.

This research aimed to investigate how children with autism interacted with rich audio and visual augmented reality (AR) tabletop games. Based on in-depth needs analysis facilitated through autism centers in China, we designed and developed MagicBLOCKS, a series of tabletop AR interactive games for children with autism. We conducted a four-week field study with 15 male children. We found that the interactive dynamics in games were rewarding and played critical roles in motivation and sustained interest. In addition, based on post-hoc interviews and video analysis with expert therapists, we found that MagicBLOCKS provided opportunities for children with autism to engage with each other through player performances and audience interactions with episodes of cooperation and territoriality. We discuss the limitations and the insights offered by this research.

Tickling is a type of sensation that is associated with laughter, smiling, or other similar reactions. Psychology research has shown that tickling and laughter can significantly relieve stress. Although several tickling artifacts have been suggested in prior work, limited knowledge is available if those artifacts could evoke laughter. In this article, we aim at filling this gap by designing and developing a novel foot-tickling mechanism that can evoke laughter. We first developed an actuator that can create tickling sensations along the sole of the foot utilising magnet-driven brushes. Then, we conducted two studies to identify the most ticklish locations of the foot’s sole and stimulation patterns that can evoke laughter. In a follow-up study with a new set of participants, we confirmed that the identified stimuli could evoke laughter. From the participants’ feedback, we derived several applications that such a simulation could be useful. Finally, we embedded our actuators into a flexible insole, demonstrating the potential of a wearable tickling insole.

We present ANISMA, a software and hardware toolkit to prototype on-skin haptic devices that generate skin deformation stimuli like pressure, stretch, and motion using shape-memory alloys (SMAs). Our toolkit embeds expert knowledge that makes SMA spring actuators more accessible to human–computer interaction (HCI) researchers. Using our software tool, users can design different actuator layouts, program their spatio-temporal actuation and preview the resulting deformation behavior to verify a design at an early stage. Our toolkit allows exporting the actuator layout and 3D printing it directly on skin adhesive. To test different actuation sequences on the skin, a user can connect the SMA actuators to our customized driver board and reprogram them using our visual programming interface. We report a technical analysis, verify the perceptibility of essential ANISMA skin deformation devices with 8 participants, and evaluate ANISMA regarding its usability and supported creativity with 12 HCI researchers in a creative design task.

Scientific inquiry involves prediction, observation and explanation (POE) of phenomena and data. Appropriate guidance through these steps is essential for helping students learn and form positive attitudes towards science. Sensor-based education toolkits are becoming a popular way to provide this guidance, but they typically present different interfaces for measurement and learning materials which places a high cognitive demand on learners. To address this challenge, we developed a web application to integrate the scientific inquiry method where students are guided step-by-step, using a scaffolded-learning approach, through slide-based worksheets that provide direct interaction with real-time sensor measurements. We evaluate this approach through a qualitative analysis of data collected from two field studies in classrooms with a total of 42 students. We show that our approach encouraged positivity and further learning in science. Students displayed and expressed interest to conduct science experiments outside of class. We identify design implications for seamless learning, storytelling and integration of POE guided scientific inquiry with sensor-based toolkits.

Emerging research has demonstrated the viability of on-textile actuation mechanisms, however, an easily customizable and versatile on-cloth actuation mechanism is yet to be explored. In this paper, we present ClothTiles along with its rapid fabrication technique that enables actuation of clothes. ClothTiles leverage flexible 3D-printing and Shape-Memory Alloys (SMAs) alongside new parametric actuation designs. We validate the concept of fabric actuation using a base element, and then systematically explore methods of aggregating, scaling, and orienting prospects for extended actuation in garments. A user study demonstrated that our technique enables multiple actuation types applied across a variety of clothes. Users identified both aesthetic and functional applications of ClothTiles. We conclude with a number of insights for the Do-It-Yourself community on how to employ 3D-printing with SMAs to enable actuation on clothes.

Open-source hardware such as RISC-V has been gaining substantial momentum. Recently, they have begun to embrace Google's Android operating system to leverage its software ecosystem. Despite the encouraging progress, a challenging issue arises: a majority of Android applications are written in native languages and need to be recompiled to target new hardware platforms. Unfortunately, this recompilation process is not scalable because of the explosion of new hardware platforms. To address this issue, we present WasmAndroid, a high-performance cross-platform runtime for native programming languages on Android. WasmAndroid only requires developers to compile their source code to WebAssembly, an efficient and portable bytecode format that can be executed everywhere without additional reconfiguration. WasmAndroid can also trans-pile existing application binaries to WebAssembly when source code is not available. WebAssembly's language model is very different from C/C++ and this mismatch leads to many unique implementation challenges. In this paper, we provide workable solutions and conduct a preliminary system evaluation. We show that WasmAndroid provides acceptable performance to execute native applications in a cross-platform manner.

The sense of touch offers interesting possibilities as a robust and ubiquitous communication channel. In this paper, we present OM, a tool that enables users to design subjective vibrotactile expressions associated with contextualised information relevant to them. OM consists of a pair of wrist-worn devices that can reproduce complex vibrotactile symbols and a companion editor smartphone app that allows users to create, customise and store personalised expressions. We studied OM in real-world contexts by allowing 13 participants to explore the functionalities of OM throughout their daily interactions with complete autonomy. We highlight relevant scenarios, design considerations, and future directions towards a tool that can help people unveil an alternative, ubiquitous and private communication system accessible to all.

Self-Tracking stress at work is an important aspect of stress management and is often the first step to improving mental health and personal well-being. We introduce StressShoe, a DIY toolkit for self-tracking stress and just-in-time stress interventions for office workers performing sedentary tasks. Informed by a focus group study and a pilot study, we designed and tested a toolkit that consists of a single, off-the-shelf, shoe-mounted inertial measurement unit (IMU), a machine learning model that estimates acute stress, and a companion mobile app. The mobile app allows for user-defined just-in-time interventions on the estimated stress level. To demonstrate the benefits of our system we evaluated StressShoe with 10 users over 4 weeks. We identified several effective just-in-time personalised intervention users created and found that StressShoe is easy to use and helped them reflect on their daily stressful experiences.

With voice user interfaces (VUIs) becoming ubiquitous and speech synthesis technology maturing, it is possible to synthesise voices to resemble our friends and relatives (which we will collectively call 'kin') and use them on VUIs. However, designing such interfaces and investigating how the familiarity of kin voices affect user perceptions remain under-explored. Our surveys and interviews with 25 users revealed that VUIs using kin voices were perceived as more engaging, persuasive and safer yet eerier than VUIs using common virtual assistant voices. We then developed a technology probe, KinVoice, an Alexa-based VUI that was deployed in three households over two weeks. Users set reminders using KinVoice, which in turn, gave the reminders in synthesised kin voices. This was to explore users' needs, uncover challenges involved and inspire new applications. We discuss design guidelines for integrating familiar kin voices into VUIs, applications that benefit from its usage, and implications for balancing voice realism and usability with security and diversification.

Human-Computer Integration (HInt) is an emerging paradigm in which computational and human systems are closely interwoven. Integrating computers with the human body is not new. however, we believe that with rapid technological advancements, increasing real-world deployments, and growing ethical and societal implications, it is critical to identify an agenda for future research. We present a set of challenges for HInt research, formulated over the course of a five-day workshop consisting of 29 experts who have designed, deployed and studied HInt systems. This agenda aims to guide researchers in a structured way towards a more coordinated and conscientious future of human-computer integration.

We present a wearable forearm augmentation that enables the recreation of natural touch sensation by applying shear-forces onto the skin. In contrast to previous approaches, we arrange light-weight and stretchable 3x3cm plasters in a matrix onto the skin. Individual plasters were embedded with lines of shape-memory alloy (SMA) wires to generate shear-forces. Our design is informed by a series of studies investigating the perceptibility of different sizes, spacings, and attachments of plasters on the forearm. Our matrix arrangement enables the perception of touches, for instance, feeling ones wrist being grabbed or the arm being stroked. Users rated the recreated touch sensations as being fairly similar to a real touch (4.1/5). Even without a visual representation, users were able to correctly distinguish them with an overall accuracy of 94.75%. Finally, we explored two use cases showing how AR and VR could be empowered with experiencing recreated touch sensations on the forearm.

Speech Emotion Recognition (SER) has been a challenging task on which researchers have been working for decades. Recently, Deep Learning (DL) based approaches have been shown to perform well in SER tasks; however, it has been noticed that their superior performance is limited to the distribution of the data used to train the model. In this paper, we present an analysis of using autoencoders to improve the generalisability of DL based SER solutions. We train a sparse autoencoder using a large speech corpus extracted from social media. Later, the trained encoder part of the autoencoder is reused as the input to a long short-term memory (LSTM) network, and the encoder-LSTM modal is re-trained on an aggregation of five commonly used speech emotion corpora. Our evaluation uses an unseen corpus in the training & validation stages to simulate 'in the wild' condition and analyse the generalisability of our solution. A performance comparison is carried out between the encoder based model and a model trained without an encoder. Our results show that the autoencoder based model improves the unweighted accuracy of the unseen corpus by 8%, indicating autoencoder based pre-training can improve the generalisability of DL based SER solutions.

Multimodal emotion recognition from speech is an important area in affective computing. Fusing multiple data modalities and learning representations with limited amounts of labeled data is a challenging task. In this paper, we explore the use of modality-specific "BERT-like" pretrained Self Supervised Learning (SSL) architectures to represent both speech and text modalities for the task of multimodal speech emotion recognition. By conducting experiments on three publicly available datasets (IEMOCAP, CMU-MOSEI, and CMU-MOSI), we show that jointly fine-tuning "BERT-like" SSL architectures achieve state-of-the-art (SOTA) results. We also evaluate two methods of fusing speech and text modalities and show that a simple fusion mechanism can outperform more complex ones when using SSL models that have similar architectural properties to BERT.

We present MAGHair, a novel wearable technique that provides subtle haptic sensation by stimulating the body hair without touching the skin. Our approach builds on previous research in magnetic hair stimulation and magnetic locomotion. We use magnetic cosmetics to augment the body hair, which can then be stimulated by a wearable apparatus that combines electromagnets and permanent magnets. We provide technical insights on the implementation of a fully functional wrist-worn form factor and early adaptations into other form factors. In addition, we provide a workflow for evaluating and characterizing the magnetic cosmetic recipes. Finally, we evaluate MAGHair, which demonstrated that users could detect the sensation of hair movement that they described as gentle and unique.

Sensing interfaces relying on head or facial gestures provide effective solutions for hands-free scenarios. Most of these interfaces utilize sensors attached to the face, as well as into the mouth, being either obtrusive or limited in input bandwidth. In this paper, we propose ChewIt -- a novel intraoral input interface. ChewIt resembles an edible object that allows users to perform various hands-free input operations, both simply and discreetly. Our design is informed by a series of studies investigating the implications of shape, size, locations for comfort, discreetness, maneuverability, and obstructiveness. Additionally, we evaluated potential gestures that users could use to interact with such an intraoral interface.

The correct execution of exercises, such as squats and dead-lifts, is essential to prevent various bodily injuries. Existing solutions either rely on expensive motion tracking or multiple Inertial Measurement Units (IMU) systems require an extensive set-up and individual calibration. This paper introduces a proof of concept, GymSoles, an insole prototype that provides feedback on the Centre of Pressure (CoP) at the feet to assist users with maintaining the correct body posture, while performing squats and dead-lifts. GymSoles was evaluated with 13 users in three conditions: 1) no feedback, 2) vibrotactile feedback, and 3) visual feedback. It has shown that solely providing feedback on the current CoP, results in a significantly improved body posture.

In this paper, we present M-Hair, a novel method for providing tactile feedback by stimulating only the body hair without touching the skin. It works by applying passive magnetic materials to the body hair, which is actuated by external magnetic fields. Our user study suggested that the value of the M-hair mechanism is in inducing affective sensations such as pleasantness, rather than effectively discriminating features such as shape, size, and direction. This work invites future research to use this method in applications that induce emotional responses or affective states, and as a research tool for investigations of this novel sensation.

Currently, smartwatches are equipped with Photoplethysmography (PPG) sensors to measure Heart Rate (HR) and Heart Rate Variability (HRV). However, PPG sensors consume considerably high energy, making it impractical to monitor HR & HRV continuously for an extended period. Utilising low power accelerometers to estimate HR has been broadly discussed in previous decades. Inspired by prior work, we introduce CompRate, an alternative method to measure HR continuously for an extended period in low-intensity physical activities. CompRate model calibrated for individual users only has an average performance of Root Mean Squared Error (RMSE) 1.58 Beats Per Minute (BPM). Further, CompRate used 3.75 times less energy compared to the built-in PPG sensor. We also demonstrate that CompRate model can be extended to predict HRV. We will demonstrate CompRate in several application scenarios: self-awareness of fatigue and just-in-time interruption while driving; enabling teachers to be aware of students’ mental effort during a learning activity; and the broadcasting of the location of live victims in a disaster situation.

Cognitive-affective states during learning or interactions with technologies is dependent on the mental effort of the learner and / or the cognitive load imposed by the system. Despite the growing research on the importance of understanding cognitive-affective state and their relationship to learning, measurement of such states during the learning process in Kindergartners is still unclear. While most assessments of learning and usability evaluations with Kindergartners focus on performance, self-reports and inferring from observable behaviours, they provide limited insights into the cognitive load and emotional state during the learning or interaction that are essential for a holistic picture of learning. Through a study with 18 kindergartners, we explore the feasibility of understanding cognitive-affective states associated with mental effort by triangulating the data obtained from observations, physiological markers, self-reports and performance as they performed tasks of varying mental effort. We present findings on the reliable markers within these sources across tasks. Results reveal that such a triangulation offers deeper insights into the cognitive-affective state of the learner. We believe this work would be a step towards better understanding of the learning process thereby facilitating instruction that is more aligned with the learner's cognitive-affective architecture as well as establishing guidelines for comprehensive usability / evaluation processes based on well-defined associations between child behaviour and child action.

Rhythm is the first musical concept deaf people learn in music classes. However, hearing loss limits the amount of information that allows a deaf person to evaluate his or her performance and stay in sync with other musicians. In this paper, we investigated how a visual and vibrotactile music-sensory-substitution device, MuSS-Bits++, affects rhythm discrimination, reproduction, and expressivity of deaf people. We conducted a controlled study with 11 deaf children and found that most participants felt more confident wearing the device in vibration mode even when it did not objectively improve their accuracy. Furthermore, we studied how MuSS-Bits++ can be used in music classes at deaf schools and what challenges and opportunities arise in such a setting. Based on these studies, we discuss insights and future directions that support the design and development of music-sensory-substitution systems for music making.

Conventionally, keys on a physical keyboard have only two states: "released'' and "pressed''. As such, various techniques, such as hotkeys, are designed to enhance the keyboard expressiveness. Realizing that user inevitably perform touch actions during keystrokes, we propose GestAKey, leveraging location and motion of the touch on individual keycaps to augment the functionalities of existing keystrokes. With a log study, we collected touch data for both normal usage (typing and hotkeys) and while performing touch gestures (location and motion), which are analyzed to assess the viability of augmenting keystrokes with simultaneous gestures. A controlled experiment was conducted to compare GestAKey with existing keyboard interaction techniques, in terms of efficiency and learnability. The results show that GestAKey has comparable performance with hotkey. We further discuss the insights of integrating such touch modality into existing keyboard interaction, and demonstrate several usage scenarios.

Most interfaces of our interactive devices such as phones and laptops are flat and are built as external devices in our environment, disconnected from our bodies. Therefore, we need to carry them with us in our pocket or in a bag and accommodate our bodies to their design by sitting at a desk or holding the device in our hand. We propose Hand Range Interface, an input surface that is always at our fingertips. This body-centric interface is a semi-sphere attached to a user's wrist, with a radius the same as the distance from the wrist to the index finger. We prototyped the concept in virtual reality and conducted a user study with a pointing task. The input surface can be designed as rotating with the wrist or fixed relative to the wrist. We evaluated and compared participants' subjective physical comfort level, pointing speed and pointing accuracy on the interface that was divided into 64 regions. We found that the interface whose orientation was fixed had a much better performance, with 41.2% higher average comfort score, 40.6% shorter average pointing time and 34.5% lower average error. Our results revealed interesting insights on user performance and preference of different regions on the interface. We concluded with a set of guidelines for future designers and developers on how to develop this type of new body-centric input surface.

Construction and sensing within malleable interfaces is usually limited by a number of constraints. Building the interface from diverse combinations of conductive and nonconductive soft materials, such as fabrics or foams combined with various sensors, complicates the manufacturing process and offers limited options in shaping. In this paper we propose "Unique Topologies for Acoustic Propagation" (UTAP), a novel approach for algorithmic design of malleable tangible interfaces. A fundamental feature of our approach is the implementation of algorithmically generated topologically distinct lattices that, attached to piezoelectric (PZT) transducers, allow us to sense and recognize changes in a modulated acoustic signal on deformation and classify it into different interaction states. Our systematic approach to manufacturing malleable interfaces opens possibilities to design shapes that allow implementation in a wide range of potential use cases. We demonstrate the UTAP approach on multiple interfaces assembled using laser cut and 3D printed lattices in conjunction with silicon compound moulding. Finally, we present a technical evaluation of our method based on studies of four distinct interface designs, assessing performance in sensing and localising simple deformations such as pressing on single and multiple spots, as well as different force levels and actions, including bending and twisting.

This paper describes the design of "ArmSleeve", a patient monitoring system to support occupational therapists in their upper limb rehabilitation work with stroke patients. Occupational therapists can provide rehabilitation in clinics, but they have limited insights into how much their patients use their affected arm and hand in daily life, which is critical for effective recovery to occur. Our work addresses this problem through three interrelated studies: (1) interviews with therapists to examine their current rehabilitation practices; (2) the design of the "ArmSleeve Sensor" to monitor a patient's upper limb movement; and (3) the design and evaluation of the "ArmSleeve Dashboard" to visualize this information for therapists. The findings show the importance of collecting objective data to assess exercise and activities outside therapy, but also a lack of contextual information to interpret this data. We discuss considerations for how to address this issue through patient engagement as well as considerations for designing wearable sensor technology that is usable in everyday life.

Learning a musical instrument can be a challenging task for a deaf person due to limited access to sound. Prior work has developed visual and vibrotactile approaches to provide music-to-sound feedback to deaf people. However, these systems are not designed for ad-hoc access to sound, which enables a deaf person to explore sound from various audio sources and receive real-time feedback. In this paper we present the development of a music sensory substitution system that enables ad-hoc access to musical sounds. It provides the technical basis to study deeper research questions about understanding and creating sound.

Accessing printed text in a mobile context is a major challenge for the blind. A preliminary study with blind people reveals numerous difficulties with existing state-of-the-art technologies including problems with alignment, focus, accuracy, mobility and efficiency. In this paper, we present a finger-worn device, FingerReader, that assists blind users with reading printed text on the go. We introduce a novel computer vision algorithm for local-sequential text scanning that enables reading single lines, blocks of text or skimming the text with complementary, multimodal feedback. This system is implemented in a small finger-worn form factor, that enables a more manageable eyes-free operation with trivial setup. We offer findings from three studies performed to determine the usability of the FingerReader.

In this paper we present zSense, which provides greater input expressivity for spatially limited devices such as smart wearables through a shallow depth gesture recognition system using non-focused infrared sensors. To achieve this, we introduce a novel Non-linear Spatial Sampling (NSS) technique that significantly cuts down the number of required infrared sensors and emitters. These can be arranged in many different configurations; for example, number of sensor emitter units can be as minimal as one sensor and two emitters. We implemented different configurations of zSense on smart wearables such as smartwatches, smartglasses and smart rings. These configurations naturally fit into the flat or curved surfaces of such devices, providing a wide scope of zSense enabled application scenarios. Our evaluations reported over 94.8% gesture recognition accuracy across all configurations.

We present an approach to combining digital fabrication and craft, demonstrating a hybrid interaction paradigm where human and machine work in synergy. The FreeD is a hand-held digital milling device, monitored by a computer while preserving the makers freedom to manipulate the work in many creative ways. Relying on a pre-designed 3D model, the computer gets into action only when the milling bit risks the objects integrity, preventing damage by slowing down the spindle speed, while the rest of the time it allows complete gestural freedom. We present the technology and explore several interaction methodologies for carving. In addition, we present a user study that reveals how synergetic cooperation between human and machine preserves the expressiveness of manual practice. This quality of the hybrid territory evolves into design personalization. We conclude on the creative potential of open-ended procedures within this hybrid interactive territory of manual smart tools and devices.

This paper presents StickEar, a system consisting of a network of distributed 'Sticker-like' sound-based sensor nodes to propose a means of enabling sound-based interactions on everyday objects. StickEar encapsulates wireless sensor network technology into a form factor that is intuitive to reuse and redeploy. Each StickEar sensor node consists of a miniature sized microphone and speaker to provide sound-based input/output capabilities. We provide a discussion of interaction design space and hardware design space of StickEar that cuts across domains such as remote sound monitoring, remote triggering of sound, autonomous response to sound events, and controlling of digital devices using sound. We implemented three applications to demonstrate the unique interaction capabilities of StickEar.

This article addresses the broad question of understanding whether and how a combination of tactile and visual information could be used to enhance the experience of music by the hearing impaired. Initially, a background survey was conducted with hearing-impaired people to find out the techniques they used to “listen” to music and how their listening experience might be enhanced. Information obtained from this survey and feedback received from two profoundly deaf musicians were used to guide the initial concept of exploring haptic and visual channels to augment a musical experience. The proposed solution consisted of a vibrating “Haptic Chair” and a computer display of informative visual effects. The Haptic Chair provided sensory input of vibrations via touch by amplifying vibrations produced by music. The visual display transcoded sequences of information about a piece of music into various visual sequences in real time. These visual sequences initially consisted of abstract animations corresponding to specific features of music such as beat, note onset, tonal context, and so forth. In addition, because most people with impaired hearing place emphasis on lip reading and body gestures to help understand speech and other social interactions, their experiences were explored when they were exposed to human gestures corresponding to musical input. Rigorous user studies with hearing-impaired participants suggested that musical representation for the hearing impaired should focus on staying as close to the original as possible and is best accompanied by conveying the physics of the representation via an alternate channel of perception. All the hearing-impaired users preferred either the Haptic Chair alone or the Haptic Chair accompanied by a visual display. These results were further strengthened by the fact that user satisfaction was maintained even after continuous use of the system over a period of 3 weeks. One of the comments received from a profoundly deaf user when the Haptic Chair was no longer available (“I am going to be deaf again”), poignantly expressed the level of impact it had made. The system described in this article has the potential to be a valuable aid in speech therapy, and a user study is being carried out to explore the effectiveness of the Haptic Chair for this purpose. It is also expected that the concepts presented in this paper would be useful in converting other types of environmental sounds into a visual display and/or a tactile input device that might, for example, enable a deaf person to hear a doorbell ring, footsteps approaching from behind, or a person calling him or her, or to make understanding conversations or watching television less stressful. Moreover, the prototype system could be used as an aid in learning to play a musical instrument or to sing in tune. This research work has shown considerable potential in using existing technology to significantly change the way the deaf community experiences music. We believe the findings presented here will add to the knowledge base of researchers in the field of human–computer interaction interested in developing systems for the hearing impaired.

The 'Haptic Chair' [3] delivers vibrotactile stimulation to several parts of the body including the palmar surface of the hand (palm and fingers), and has been shown to have a significant positive effect on the enjoyment of music even by the profoundly deaf. In this paper, we explore the effectiveness of using the Haptic Chair during speech therapy for the deaf. We conducted a 24-week study with 20 profoundly deaf users to validate our initial observations. The improvements in word clarity observed over the duration of this study indicate that the Haptic Chair has the potential to make a significant contribution to speech therapy for the deaf.

Remote monitoring allows us to understand the regular living behaviors of the elderly and alert their loved ones in emergency situations. In this paper, we describe WatchMe, a software and hardware platform that focuses on making ambient monitoring intuitive and seamless. WatchMe system consists of the WatchMe server application and a WatchMe client application implemented on a regular wristwatch. Thus, it requires minimal effort to monitor and is less disruptive to the user. We hope that the WatchMe system will contribute to improving the lives of the elderly by creating a healthy link between them and their loved ones.

SPARSH presents a seamless way of passing data among multiple users and devices. The user touches a data item they wish to copy from a device, conceptually saving it in the user's body. Next, the user touches the other device they want to paste/pass the saved content. SPARSH uses touch-based interactions as indications for what to copy and where to pass it. Technically, the actual transfer of media happens via the information cloud. Accompanying video shows some of the SPARSH scenarios.

SPARSH explores a novel interaction method to seamlessly transfer data among multiple users and devices in a fun and intuitive way. The user touches a data item they wish to copy from a device, conceptually saving in the user's body. Next, the user touches the other device they want to paste/pass the saved content. SPARSH uses touch-based interactions as indications for what to copy and where to pass it. Technically, the actual transfer of media happens via the information cloud.

Music is a multi-dimensional experience informed by much more than hearing alone, and is thus accessible to people of all hearing abilities. In this paper we describe a prototype system designed to enrich the experience of music for the deaf by enhancing sensory input of information via channels other than in-air audio reception by the ear. The system has two main components-a vibrating 'Haptic Chair' and a computer display of informative visual effects that correspond to features of the music. The Haptic Chair provides sensory input of vibrations via touch. This system was developed based on an initial concept guided by information obtained from a background survey conducted with deaf people from multi-ethnic backgrounds and feedback received from two profoundly deaf musicians. A formal user study with 43 deaf participants suggested that the prototype system enhances the musical experience of a deaf person. All of the users preferred either the Haptic Chair alone (54%) or the Haptic Chair with the visual display (46%). The prototype system, especially the Haptic Chair was so enthusiastically received by our subjects that it is possible this system might significantly change the way the deaf community experiences music.

Finding the shortest path from a given source to a given destination is a well known and widely applicable problem. Most of the work done in the area have used static route planning algorithms such as A*, Dijkstra’s, Bellman-Ford algorithm etc. Although these algorithms are said to be optimum, they are not capable of dealing with certain real life scenarios. For example, most of these single objective optimizations fails to find the equally good solutions when there is more than one optimum (shortest distance path, least congested path). We believe that the Genetic Algorithm (GA) based route planning algorithm proposed in this paper has the ability to tackle the above problems. In this paper, the proposed GA based route planning algorithm is successfully tested on the entire Singapore map with more than 10,000 nodes. Performance of the proposed GA is compared with an ant based path planning algorithm. Simulation results demonstrate the effectiveness of the proposed algorithm over ant based algorithm. Moreover, the proposed GA may be used as a basis for developing an intelligent route planning system.