Instrumental music conveys rich affective experiences through acoustic cues, yet instrumental passages often remain inaccessible to Deaf and Hard-of-Hearing (DHH) audiences. Although captioning practices for vocal songs have expanded, instrumental music remains largely uncaptioned, with no established criteria for representing musical content in text. We propose µCap (Music Captions), an automatic instrumental music captioning system that transforms instrumental audio into time-aligned, non-lexical textual renderings enhanced with simple visuals. Drawing on Preliminary surveys with DHH individuals and expert group discussions, we developed a phonetic-like captioning schema grounded in music sound analysis and linguistics. We then implemented µCap using audio feature extraction and a retrieval-augmented generation pipeline to produce expressive, sound-mimetic captions. Two user evaluations with DHH participants (n=20 and n=15) showed that µCap enhanced music appreciation, immersion, and perceived presence of acoustic detail. This work contributes empirical evidence and insights for designing caption-based visual representations that make instrumental music more accessible.

Songwriting has long served as a powerful medium for expressing unconscious emotions and fostering self-awareness in psychotherapy. Due to the auditory-centric nature of traditional approaches, Deaf and Hard-of-Hearing (DHH) individuals have often been excluded from music’s therapeutic benefits. In response, this study presents a music psychotherapy tool co-designed with therapists, integrating conversational agents (CAs) and music generative AI as symbolic and therapeutic media. Through a usage study with 23 DHH individuals, we found that collaborative songwriting with the CA enabled them to experience emotional release, reinterpretation, and deeper self-understanding. In particular, the CA’s strategies—supportive empathy, example response options, and visual-based metaphors—were found to facilitate musical dialogue effectively for DHH individuals. These findings contribute to inclusive AI design by showing the potential of human–AI collaboration to bridge therapeutic and artistic practices.

The rapid advancement of generative AI (GenAI) is expanding access to songwriting, offering a new medium of self-expression for Deaf and Hard-of-Hearing (DHH) individuals. However, emerging technologies that support DHH individuals in expressing themselves through music have largely been evaluated in single-session settings and often fall short in helping users unfamiliar with songwriting convey personal narratives or sustain engagement over time. This paper explores songwriting as an extended, music-based journaling practice that supports sustained emotional reflection over multiple sessions. We introduce SoulNote, a GenAI system enabling DHH to engage in iterative songwriting. Grounded in user-centered design, including a design workshop, a preliminary study, and a multi-session diary study, our findings show that ongoing songwriting with SoulNote facilitated emotional growth across three dimensions: self-insight, emotion regulation, and everyday attitudes toward emotions and self-care. Overall, this work demonstrates how GenAI can support marginalized communities by transforming creative expression into a daily practice of self-discovery and reflection.

Stimulus-evoked gaze dynamics offer a secure and hands-free signal in virtual reality (VR), yet the underlying design space of effective visual stimuli remains poorly understood. This work examines how preattentive processing and binocular rivalry can inform stimulus design for gaze-based identification in VR. We conducted a two-part study: (1) a feasibility assessment of closed-set identification performance with 26 participants and 44,928 gaze samples collected by using a commercial headset (Meta Quest Pro), and (2) a usability study with 16 participants comparing the same interaction in a login context to PIN and out-of-band methods as a potential authentication technique. Our findings confirm the feasibility of personal identification, highlight usability advantages, and reveal participants’ desire for greater transparency to understand individual variations in login results. Together, these results offer conceptual insights into the perceptual mechanisms shaping stimulus-evoked gaze behavior, and outline design implications for future VR authentication workflows.

We explore how humanoid robots can be repurposed as haptic media, extending beyond their conventional role as social, assistive, collaborative agents. To illustrate this approach, we implemented HumanoidTurk, taking a first step toward a humanoid-based haptic system that translates in-game g-force signals into synchronized motion feedback in VR driving. A pilot study involving six participants compared two synthesis methods, leading us to adopt a filter-based approach for smoother and more realistic feedback. A subsequent study with sixteen participants evaluated four conditions: no-feedback, controller, humanoid+controller, and human+controller. Results showed that humanoid feedback enhanced immersion, realism, and enjoyment, while introducing moderate costs in terms of comfort and simulation sickness. Interviews further highlighted the robot’s consistency and predictability in contrast to the adaptability of human feedback. From these findings, we identify fidelity, adaptability, and versatility as emerging themes, positioning humanoids as a distinct haptic modality for immersive VR.

Although students need to self-monitor and manage their learning process for effective metacognition, it can be particularly challenging in solitary e-learning environments that rely on pre-recorded videos. Unlike interactive e-learning or physical classrooms, typical e-learning environments prevent students from interacting with their teachers and peers, thereby hindering metacognitive support. To address this challenge, we introduce GPTalk, a system designed to support students’ learning experiences by facilitating interactions with LLM-based virtual companions. Through interviews with students and teachers, we identified design recommendations and implemented them in GPTalk. A user study involving 32 high-school students demonstrated that, compared to a baseline system, GPTalk fostered richer metacognitive engagement and self-regulated learning processes during video-based study (e.g., more monitoring questions and in-situ reflections), while short-term content understanding accuracy remained comparable across conditions. Overall, our findings suggest that students’ interactions with a virtual teacher and peer can support key aspects of their metacognition and self-regulated e-learning processes.

Affective and prosodic captions convey not only what a speaker says, but also how they say it-louder words may appear thicker, quieter ones thinner; angry in red, calm in blue.These captions can improve access, satisfaction, and engagement for d/Deaf and Hard-of-Hearing (dhh) users.While prior work has explored their design space, it has focused largely on dhh participants in North America, limiting generalizability beyond English and Latin-based scripts.To uncover the role of culture and language, we ran an exploratory study with 49 dhh participants from North America and South Korea using CuCap, a tool that allowed them to personalize which speech features were displayed, and how.While emotion visualization was a universally favored choice, confirming prior findings, prosody preferences varied across cultures, reflecting linguistic and hearing factors.These findings point to the need for flexible captioning systems that account for cultural, linguistic, and individual differences.

Biased news articles can distort readers' perceptions by presenting information in a way that favors or disfavors a particular point of view.Subtly embedded in the text, these biased news articles can shape our views daily without people even realizing it.To address this issue, we propose BIASsist, an LLM-based approach designed to mitigate bias in news articles.Based on existing research, we defned six types of bias and introduced three assistive components-identifcation, explanation, and neutralization-to provide a broader range of bias information and enhance readers' bias-awareness.We conducted a mixed-method study with 36 participants to evaluate the efectiveness of BIASsist.The results show participants' bias awareness signifcantly improved and their interest in identifying bias increased.Participants also tended to engage more actively in critically evaluating articles.Based on these fndings, we discuss its potential to improve media literacy and critical thinking in today's information overload era.

Dance learning through online videos has gained popularity, but it presents challenges in providing comprehensive information and personalized feedback. This paper introduces DanceSculpt, a system that utilizes 3D human reconstruction and tracking technology to enhance the dance learning experience. DanceSculpt consists of a dancer viewer that reconstructs dancers in video into 3D avatars and a dance feedback tool that analyzes and compares the user’s performance with that of the reference dancer. We conducted a comparative study to investigate the effectiveness of DanceSculpt against conventional video-based learning. Participants’ dance performances were evaluated using a motion comparison algorithm that measured the temporal and spatial deviation between the users’ and reference dancers’ movements in terms of pose, trajectory, formation, and timing accuracy. Additionally, user experience was assessed through questionnaires and interviews, focusing on aspects such as effectiveness, usefulness, and satisfaction with the system. The results showed that participants using DanceSculpt achieved significant improvements in dance performance compared to those using conventional methods. Furthermore, the participants rated DanceSculpt highly in terms of effectiveness (avg. 4.27) and usefulness (avg. 4.17) for learning dance. The DanceSculpt system demonstrates the potential of leveraging 3D human reconstruction and tracking technology to provide a more informative and interactive dance learning experience. By offering detailed visual information, multiple viewpoints, and quantitative performance feedback, DanceSculpt addresses the limitations of traditional video-based learning and supports learners in effectively analyzing and improving their dance skills.

Integrating mixed-hearing groups in musical collaboration presents unique challenges and opportunities for their communication and equal contribution. This observational study aims to explore their collaborative work, focusing on the way for equitable music-making. We observed two music-making workshops to identify the potential and dynamics of their musical collaboration. While the first workshop proceeded in a traditional manner of music-making, the second workshop used an assistive tool with multimodality. Our findings highlight the dynamics in musical collaboration that foster engagement and bridge interaction gaps. In turn, sensory inclusion with multimodal music-making promoted role transition in mixed-hearing groups and their equal contributions, leading to the embracing of diverse cultural perspectives. Based on the insights derived from the observations, we propose a design guideline and future research directions for harnessing group dynamics and building equitable musical collaborations for an inclusive environment for mixed-hearing groups.

Deaf and hard of hearing (DHH) people enjoy music and access it using a music-sensory substitution system that delivers sound together with the corresponding visual and tactile feedback. However, it is often challenging for them to comprehend the colorful visuals and strong vibrations that are designed to represent music. We confirmed that it is necessary to conceptualize cross-modal mapping before experiencing music sensory substitution through focus group interviews with 24 DHH people. To improve the music appreciation experience, a cross-modal music conceptualization system was implemented herein, which is a prototype that allows DHH people to explore the visuals and vibrations associated with music to perceive and appreciate. An evaluation with 28 DHH individuals demonstrated the capability of the system to improve subjective music appreciation experience via music-sensory substitution. Eventually, DHH people with negative attitudes toward music became positive in the exploration and customization process with our system.

Gesture data collection in a controlled lab environment often restricts participants to performing gestures in a uniform or biased manner, resulting in gesture data which may not sufficiently reflect gesture variability to build robust gesture recognition models. Crowdsourcing has been widely accepted as an efficient high-sample-size method for collecting more representative and variable data. In this study, we evaluated the effectiveness of crowdsourcing for gesture data collection, specifically for gesture variability. When compared to a controlled lab environment, crowdsourcing resulted in improved recognition performance of 8.98% and increased variability for various gesture features, eg, a 142% variation increase for Quantity of Movement. Integrating a supplemental gesture data collection methodology known as Styling Words increased recognition performance by an additional 2.94%. The study also investigated the efficacy of gesture collection methodologies and gesture memorization paradigms.

Captions help deaf and hard-of-hearing (DHH) individuals visually communicate voice information to better understand video content. In speech, the literal content and paralinguistic cues (e.g., pitch and nuance) work together to create real intention. However, current captions are limited in their capacity to deliver fine nuances because they cannot fully convey these paralinguistic cues. This paper proposes an audio-visualized caption system that automatically visualizes paralinguistic cues into various caption elements (thickness, height, font type and motion). A comparative study with 20 DHH participants demonstrates how our system supports DHH individuals to be better accessible to paralinguistic cues while watching videos. Particularly in the case of formal talks, they could accurately identify the speaker’s nuance more often compared to current captions, without any practice or training. Addressing some issues on legibility and familiarity, the proposed caption system has potentials to enrich DHH individuals’ video watching experience more as hearing people enjoy.

Manual sign systems have been introduced to improve the communication of children with intellectual developmental disabilities (IDD). Due to the lack of learning support tools, teachers face many practical challenges in teaching manual sign to children, such as low attention span and the need for persistent intervention. To address these issues, we collaborated with teachers to develop the Sondam Rhythm Game, a gesture-based rhythm game that assists in teaching manual sign language, and ran a four-week empirical study with five teachers and eight children with IDD. Based on video annotation and post-hoc interviews, our game-based learning approach has the potential to be effective at teaching manual sign to children with IDD. Our approach improved children attention span and motivation while also increasing the number of voluntary gestures made without the need for prompting. Other practical issues and learning challenges were also uncovered to improve teaching paradigms for children with IDD.

Immersion plays a crucial role in video watching, leading viewers to a positive experience, such as increased engagement and decreased fatigue. However, few studies measure immersion while watching videos, and questionnaires are typically used in the measurement of immersion for other applications. These methods may rely on the viewer's memory and cause biased results. Therefore, we propose an objective immersion detection model by leveraging people's gaze behavior while watching videos. In a lab study with 30 participants, an in-depth analysis is carried out on a number of gaze features and machine learning (ML) models to identify the immersion state. Several gaze features are highly indicative of immersion and ML models with these features are able to detect an immersion state of video watchers. Post-hoc interviews demonstrate that our approach is applicable to measure immersion in the middle of watching a video, where some practical issues are discussed as well.

Due to advances in deep learning, gestures have become a more common tool for human-computer interaction. When implementing a large amount of training data, deep learning models show remarkable performance in gesture recognition. Since it is expensive and time consuming to collect gesture data from people, we are often confronted with a practicality issue when managing the quantity and quality of training data. It is a well-known fact that increasing training data variability can help to improve the generalization performance of machine learning models. Thus, we directly intervene in the collection of gesture data to increase human gesture variability by adding some words (called styling words) into the data collection instructions, e.g., giving the instruction "perform gesture #1 faster" as opposed to "perform gesture #1." Through an in-depth analysis of gesture features and video-based gesture recognition, we have confirmed the advantageous use of styling words in gesture training data collection.

Activity recognition is a key component of context-aware computing to support people's physical activity, but conventional approaches often lack in their generalizability and scalability due to problems of diversity in how individuals perform activities, overfitting when building activity models, and collection of a large amount of labeled data from end users. To address these limitations, we propose a semipopulation-based approach that exploits activity models trained from other users; therefore, a new user does not need to provide a large volume of labeled activity data. Instead of relying on any additional information from users like their weight or height, our approach directly measures the fitness of others' models on a small amount of labeled data collected from the new user. With these shared activity models among users, we compose a hybrid model of Bayesian networks and support vector machines to accurately recognize the activity of the new user. On activity data collected from 28 people with a diversity in gender, age, weight, and height, our approach produced an average accuracy of 83.4% (kappa: 0.852), compared with individual and (standard) population models that had accuracies of 77.3% (kappa: 0.79) and 77.7% (kappa: 0.743), respectively. Through an analysis on the performance of our approach and users' demographic information, our approach outperforms others that rely on users' demographic information for recognizing their activities, which may contradict the commonly held belief that physically similar people would have similar activity patterns.

Using the physiological system to perform affect modeling has great potential but also introduces many challenging issues in pervasive and interactive computing. With the advances in low-power mobile sensors, it is now possible to create a good quality of affect models based on physiological responses, which are useful in understanding how people express affect in real-world environments. In this paper, we have investigated an affect modeling technique that analyzes physiological changes and models user affect with data gathered in the field. In particular, we have identified a number of sensor channels and features that are discriminable in recognizing stress with Support Vector Machines. We have empirically investigated the value of creating an affect model by using a subset of informative features for an individual on physiological data collected in real-world environments (i.e. outside the lab), and we provide a discussion of the remaining challenging issues in performing field-based physiological analysis.

Driving aggressively increases the risk of accidents. Assessing a person's driving style is a useful way to guide aggressive drivers toward having safer driving behaviors. A number of studies have investigated driving style, but they often rely on the use of self-reports or simulators, which are not suitable for the real-time, continuous, automated assessment and feedback on the road. In order to understand and model aggressive driving style, we construct an in-vehicle sensing platform that uses a smartphone instead of using heavyweight, expensive systems. Utilizing additional cheap sensors, our sensing platform can collect useful information about vehicle movement, maneuvering and steering wheel movement. We use this data and apply machine learning to build a driver model that evaluates drivers' driving styles based on a number of driving-related features. From a naturalistic data collection from 22 drivers for 3 weeks, we analyzed the characteristics of drivers who have an aggressive driving style. Our model classified those drivers with an accuracy of 90.5% (violation-class) and 81% (questionnaire-class). We describe how, in future work, our model can be used to provide real-time feedback to drivers using only their current smartphone.

With advances in physiological sensors, we are able to understand people's physiological status and recognize stress to provide beneficial services. Despite the great potential in physiological stress recognition, there are some critical issues that need to be addressed such as the sensitivity and variability of physiology to many factors other than stress (e.g., physical activity). To resolve these issues, in this paper, we focus on the understanding of physiological responses to both stressor and physical activity and perform stress recognition, particularly in situations having multiple stimuli: physical activity and stressors. We construct stress models that correspond to individual situations, and we validate our stress modeling in the presence of physical activity. Analysis of our experiments provides an understanding on how physiological responses change with different stressors and how physical activity confounds stress recognition with physiological responses. In both objective and subjective settings, the accuracy of stress recognition drops by more than 14% when physical activity is performed. However, by modularizing stress models with respect to physical activity, we can recognize stress with accuracies of 82% (objective stress) and 87% (subjective stress), achieving more than a 5-10% improvement from approaches that do not take physical activity into account.

It is becoming harder to find an app on one's smart phone due to the increasing number of apps available and installed on smart phones today. We collect sensory data including app use from smart phones, to perform a comprehensive analysis of the context related to mobile app use, and build prediction models that calculate the probability of an app in the current context. Based on these models, we developed a dynamic home screen application that presents icons for the most probable apps on the main screen of the phone and highlights the most probable one. Our models outperformed other strategies, and, in particular, improved prediction accuracy by 8% over Most Frequently Used from 79.8% to 87.8% (for 9 candidate apps). Also, we found that the dynamic home screen improved accessibility to apps on the phone, compared to the conventional static home screen in terms of accuracy, required touch input and app selection time.

Much research in ubiquitous computing assumes that a user's phone will be always on and at-hand, for collecting user context and for communicating with a user. Previous work with the previous generation of mobile phones has shown that such an assumption is false. Here, we investigate whether this assumption about users' proximity to their mobile phones holds for a new generation of mobile phones, smart phones. We conduct a data collection field study of 28 smart phone owners over a period of 4 weeks. We show that in fact this assumption is still false, with the within arm's reach proximity being true close to 50% of the time, similar to the earlier work. However, we also show that smart phone proximity within the same room (arm+room) as the user is true almost 90% of the time. We discuss the reasons for these phone proximities and the implications of this on the development of mobile phone applications, particularly those that collect user and environmental context, and delivering notification to users. We also show that we can accurately predict the proximity at the arm level and arm+room level with 75 and 83% accuracy, respectively, with features simple to collect and model on a mobile phone. Further we show that for several individuals who are almost always within the arm+room level, we can predict this level with over 90% accuracy.

Fingerprint classification reduces the searching time of an automated fingerprint identification system. Since fingerprints have properties of intra-class diversities and inter-class similarities, the ambiguous example causes a difficult problem in the fingerprint classification. In order to addres s the problem, we have analyzed fingerprints' subclasses with multiple decision templates. It clusters the soft outputs of support vector machines (SVMs) into several sub-classes using the self-organizing maps, and estimates a localized template for each sub-class. For an input fingerprint, the proposed method matches the output vector of SVMs to each template and finally categorizes the sample into the class of the most similar template. Experimental results on the FingerCode dataset demonstrate the effectiveness of the subclass-based approach compared with previous methods.

Image enhancement is an important issue in digital image processing. Various approaches have been developed to solve image enhancement problems, but most of them require deep expert knowledge to design appropriate image filters. To automatically design a filter, we propose a novel approach based on the genetic algorithm that optimizes a set of standard filters by determining their types and order. Moreover, the proposed method is able to manage various types of noise factors. We applied the proposed method to local and global image enhancement problems such as impulsive noise reduction, interpolation, and orientation enhancement. In terms of subjective and objective evaluations, the results show the superiority of the proposed method.

Gene selection is an important issue for cancer classification based on gene expression profiles. Filter and wrapper approaches are used widely for gene selection, where the former is hard to measure the relationship between genes and the latter requires lots of computation. We present a novel method, called gene boosting, to select relevant gene subsets by integrating filter and wrapper approaches. It repeatedly selects a set of top-ranked informative genes by a filtering algorithm with respect to a temporal training dataset constructed according to the classification result for the original training dataset. Empirical results on three microarray benchmark datasets have shown that the proposed method is effective and efficient in finding a relevant and concise gene subset. Competitive performance was achieved with fewer genes in a reasonable time. This also led to the identification of some genes selected frequently as useful features.

In recent years, the concept of “autonomous mental development” (AMD) has been applied to the construction of artificial systems such as conversational agents, in order to resolve some of the difficulties involved in the manual definition of their knowledge bases and behavioral patterns. AMD is a new paradigm for developing autonomous machines, which are adaptive and flexible to the environment. Language development, a kind of mental development, is an important aspect of intelligent conversational agents. In this paper, we propose an intelligent conversational agent and its language development mechanism by putting together five promising techniques: Bayesian networks, pattern matching, finite-state machines, templates, and genetic programming (GP). Knowledge acquisition implemented by finite-state machines and templates, and language learning by GP are used for language development. Several illustrations and usability tests show the usefulness of the proposed developmental conversational agent.

As the usage of service robots becomes more sophisticated, direct communication by means of human language is required to increase the efficiency of their performance. In natural speech interaction, however, people often omit some words and rely on background knowledge or the context, resulting in ambiguity. In order to develop smarter service robots, therefore, managing the context of interaction is essential. In this correspondence, we have investigated the mixed-initiative interaction that prompts for missing information and clarifies ambiguous statements based on hierarchically designed Bayesian networks. Simulation with the Kephera II robot and a usability test have demonstrated the usefulness of the proposed method.

The service robot supports people in their daily activities, while the interaction between humans and robots seems to be an important part of its performance. Dialogue may be beneficial to the robot to increase the flexibility and facility of the interaction. Traditional robots have merely dealt with simple queries like commands, but in conversation people often omit some words because of the background knowledge or the context of the conversation. Since environments contain various uncertainties, managing the context of a dialogue or the uncertainties should be necessary to support smarter service robots. In order to establish a natural communication between people and robots, we have been investigating the use of mixed-initiative interaction that prompts for missing concepts and clarifies for spurious concepts. Hierarchically designed Bayesian networks are presented for the mixed-initiative interaction. A simulation and a real robot are constructed for the demonstration of the proposed method, and experiments also show the usefulness.

Various digital characters, which are automatic and intelligent, are attempted with the introduction of artificial intelligence or artificial life. Since a character's behavior is designed by a developer, the style can be static and simple. Even complex patterns designed by a developer cannot satisfy various users and easily make them feel tedious. A game should maintain various and complex character's behaviors, but it is not easy for the developer to design them. In this paper, we adopt genetic algorithm to produce various and excellent behavior-styles for characters especially focusing on Robocode which is one of the promising simulators for artificial intelligence.

Evolvable hardware adjusts oneself to changeable environments by self-organizing the circuit. Due to its high productivity and creativity for designing circuit, it is widely investigated. However, it is very difficult to apply it to a complicated circuit, because the search space increases exponentially as the complexity of hardware. In this paper, we propose a modular approach to evolving complex hardware circuits effectively. A comparative experiment with the conventional evolutionary approach indicates that the proposed method works 50/spl sim/1000 times faster and yields a more optimized hardware.