Songwriting has long served as a powerful medium for expressing unconscious emotions and fostering self-awareness in psychotherapy. Due to the auditory-centric nature of traditional approaches, Deaf and Hard-of-Hearing (DHH) individuals have often been excluded from music’s therapeutic benefits. In response, this study presents a music psychotherapy tool co-designed with therapists, integrating conversational agents (CAs) and music generative AI as symbolic and therapeutic media. Through a usage study with 23 DHH individuals, we found that collaborative songwriting with the CA enabled them to experience emotional release, reinterpretation, and deeper self-understanding. In particular, the CA’s strategies—supportive empathy, example response options, and visual-based metaphors—were found to facilitate musical dialogue effectively for DHH individuals. These findings contribute to inclusive AI design by showing the potential of human–AI collaboration to bridge therapeutic and artistic practices.

The rapid advancement of generative AI (GenAI) is expanding access to songwriting, offering a new medium of self-expression for Deaf and Hard-of-Hearing (DHH) individuals. However, emerging technologies that support DHH individuals in expressing themselves through music have largely been evaluated in single-session settings and often fall short in helping users unfamiliar with songwriting convey personal narratives or sustain engagement over time. This paper explores songwriting as an extended, music-based journaling practice that supports sustained emotional reflection over multiple sessions. We introduce SoulNote, a GenAI system enabling DHH to engage in iterative songwriting. Grounded in user-centered design, including a design workshop, a preliminary study, and a multi-session diary study, our findings show that ongoing songwriting with SoulNote facilitated emotional growth across three dimensions: self-insight, emotion regulation, and everyday attitudes toward emotions and self-care. Overall, this work demonstrates how GenAI can support marginalized communities by transforming creative expression into a daily practice of self-discovery and reflection.

Self-annotation is the gold standard for collecting affective state labels in affective computing. Existing methods typically rely on full annotation, requiring users to continuously label affective states across entire sessions. While this process yields fine-grained data, it is time-consuming, cognitively demanding, and prone to fatigue and errors. To address these issues, we present PREFAB, a low-budget retrospective self-annotation method that targets affective inflection regions rather than full annotation. Grounded in the peak-end rule and ordinal representations of emotion, PREFAB employs a preference learning model to detect relative affective changes, directing annotators to label only selected segments while interpolating the remainder of the stimulus. We further introduce a preview mechanism that provides brief contextual cues to assist annotation. We evaluate PREFAB through a technical performance study and a 25-participant user study. Results show that PREFAB outperforms baselines in modeling affective inflections while mitigating workload (and conditionally mitigating temporal burden). Importantly, PREFAB improves annotator confidence without degrading annotation quality.

Although students need to self-monitor and manage their learning process for effective metacognition, it can be particularly challenging in solitary e-learning environments that rely on pre-recorded videos. Unlike interactive e-learning or physical classrooms, typical e-learning environments prevent students from interacting with their teachers and peers, thereby hindering metacognitive support. To address this challenge, we introduce GPTalk, a system designed to support students’ learning experiences by facilitating interactions with LLM-based virtual companions. Through interviews with students and teachers, we identified design recommendations and implemented them in GPTalk. A user study involving 32 high-school students demonstrated that, compared to a baseline system, GPTalk fostered richer metacognitive engagement and self-regulated learning processes during video-based study (e.g., more monitoring questions and in-situ reflections), while short-term content understanding accuracy remained comparable across conditions. Overall, our findings suggest that students’ interactions with a virtual teacher and peer can support key aspects of their metacognition and self-regulated e-learning processes.

A novel reference search algorithm is proposed in this paper to significantly reduce the reference search time of embedded spin transfer torque magnetic random access memory (STT-MRAM). Unlike conventional methods that sequentially search reference levels with linearly increasing references the proposed Dual Read Reference Search (DRRS) algorithm requires only two array read operations. By analyzing the statistical characteristics of the read data using a customized function the optimal reference level can be quickly determined in a few steps. Consequently the number of read operations required for a reference search is reduced providing a substantial improvement in the reference search time. The DRRS algorithm can be operated on-chip its effectiveness was confirmed through simulations. The optimization speed was improved by 85% compared to the conventional methods. Additionally an Triple Read Reference Search (TRRS) algorithm is proposed to decrease the variation occurring across different cell arrays and to enhance optimization accuracy. STT-MRAM is composed of numerous cell arrays where the cell distributions in each array exhibit different characteristics. The TRRS algorithm enhances optimization accuracy for variations occurring in each array achieving over a 2x increase in accuracy compared to the DRRS algorithm. Furthermore Simultaneous Reference Search for P and AP (SRS) algorithm that significantly reduces the search time by simultaneously optimizing Parallel (P) and Anti-parallel state (AP) reference cells is also proposed. Lastly regarding cell degradation after power-up we enable prompt re-optimization through revolutionary time-saving algorithms (DRRS TRRS and SRS). This allows for rapid re-optimization in the event of errors caused by cell degradation and ensures regular optimization to maintain maximum read margin even before errors occur thereby enhancing reliability.

Many individuals experience everyday stress. Effective stress management in daily life is crucial before this stress accumulates. Music has been extensively studied as a method for reducing stress. In particular, music therapy is widely used to reduce stress and enhance the well-being of various clinical groups. However, traditional music therapy has physical constraints that require patients to visit a therapeutic location. VR music therapy has been studied to address these issues, but most research focuses on receptive music therapy, failing to utilize VR’s interactive potential fully. Additionally, the potential of applying gamification to VR active music therapy to enhance user engagement and encourage long-term use of the therapy application has not been explored. This paper proposes VR active music therapy based on conventional rhythm-following music therapy methods and VR gamified active music therapy by adding game elements based on Self-Determination Theory (SDT). Our between-subject comparative study (n=33) revealed the stress reduction effects of VR receptive music therapy, VR active music therapy, and VR gamified active music therapy. Importantly, participant interviews provided valuable insights into the user experiences with each VR content, confirming the potential for long-term use of VR gamified active music therapy. Moreover, this research delves into the effect of gamification on stress reduction. Through pilot test and experiments, we identify game elements that could potentially increase stress and provide guidelines for applying gamification to mitigate these factors, thereby enhancing the effectiveness of VR gamified active music therapy.

Integrating mixed-hearing groups in musical collaboration presents unique challenges and opportunities for their communication and equal contribution. This observational study aims to explore their collaborative work, focusing on the way for equitable music-making. We observed two music-making workshops to identify the potential and dynamics of their musical collaboration. While the first workshop proceeded in a traditional manner of music-making, the second workshop used an assistive tool with multimodality. Our findings highlight the dynamics in musical collaboration that foster engagement and bridge interaction gaps. In turn, sensory inclusion with multimodal music-making promoted role transition in mixed-hearing groups and their equal contributions, leading to the embracing of diverse cultural perspectives. Based on the insights derived from the observations, we propose a design guideline and future research directions for harnessing group dynamics and building equitable musical collaborations for an inclusive environment for mixed-hearing groups.

Deaf and hard of hearing (DHH) people enjoy music and access it using a music-sensory substitution system that delivers sound together with the corresponding visual and tactile feedback. However, it is often challenging for them to comprehend the colorful visuals and strong vibrations that are designed to represent music. We confirmed that it is necessary to conceptualize cross-modal mapping before experiencing music sensory substitution through focus group interviews with 24 DHH people. To improve the music appreciation experience, a cross-modal music conceptualization system was implemented herein, which is a prototype that allows DHH people to explore the visuals and vibrations associated with music to perceive and appreciate. An evaluation with 28 DHH individuals demonstrated the capability of the system to improve subjective music appreciation experience via music-sensory substitution. Eventually, DHH people with negative attitudes toward music became positive in the exploration and customization process with our system.

Advances in virtual reality (VR) technology have sparked novel avenues of growth in the musical domain. Following the COVID-19 pandemic, the rise of VR technology has led to growing interest in VR music concerts as an alternative to traditional live concerts. These virtual settings can provide immersion like attending real concerts for physically distant audiences and performers, and also can offer new creative possibilities. VR music concert research is still in its infancy, and advances in technologies such as multimodal devices are rapidly expanding the diversity of research, requiring a unified understanding of the field. To identify trends in VR music concert technology, we conducted a PRISMA-based systematic literature review covering the period from 2018 to 2023. After a thorough screening process, a total of 27 papers were selected for review. The studies were classified and analyzed based on the research topic (audience, performer, concert venue), interaction type (user-environment, user-user), and hardware used (head-mounted display, additional hardware). Furthermore, we categorized the evaluation metrics into user experience, usability, and performance. Our review contributes to advancing the understanding of recent developments in VR music concert technology, shedding light on the diversification and potential of this emerging field.

Manual sign systems have been introduced to improve the communication of children with intellectual developmental disabilities (IDD). Due to the lack of learning support tools, teachers face many practical challenges in teaching manual sign to children, such as low attention span and the need for persistent intervention. To address these issues, we collaborated with teachers to develop the Sondam Rhythm Game, a gesture-based rhythm game that assists in teaching manual sign language, and ran a four-week empirical study with five teachers and eight children with IDD. Based on video annotation and post-hoc interviews, our game-based learning approach has the potential to be effective at teaching manual sign to children with IDD. Our approach improved children attention span and motivation while also increasing the number of voluntary gestures made without the need for prompting. Other practical issues and learning challenges were also uncovered to improve teaching paradigms for children with IDD.

Immersion plays a crucial role in video watching, leading viewers to a positive experience, such as increased engagement and decreased fatigue. However, few studies measure immersion while watching videos, and questionnaires are typically used in the measurement of immersion for other applications. These methods may rely on the viewer's memory and cause biased results. Therefore, we propose an objective immersion detection model by leveraging people's gaze behavior while watching videos. In a lab study with 30 participants, an in-depth analysis is carried out on a number of gaze features and machine learning (ML) models to identify the immersion state. Several gaze features are highly indicative of immersion and ML models with these features are able to detect an immersion state of video watchers. Post-hoc interviews demonstrate that our approach is applicable to measure immersion in the middle of watching a video, where some practical issues are discussed as well.

Telepresence avatars enable users in different environments to interact with each other. In order to increase the effectiveness of these interactions, however, the movements of avatars must be adjusted accordingly to account for differences between user environments. For instance, if a user moves from one point to another in one environment, the avatar’s locomotion speed must be adjusted to move to the corresponding target point in another environment at the same time. Several locomotion styles can be used to achieve this speed change. This paper investigates how different avatar locomotion styles (speed, stride, and glide), body visibility levels (full body and head-to-knee), and views (front views and side views) influence human perceptions of the naturalness of motion, similarity to the user’s locomotion, and the degree of preserving the user’s intention. Our results indicate that 1) speed and stride styles are perceived as more natural than the glide style, while the glide style is more intention-preserving than the others, 2) a greater locomotion speed of the avatar is perceived as more natural, similar, and intention-preserving than slower motion, 3) the perception of naturalness has the greatest impact on people’s preferences for locomotion styles, and that 4) head-to-knee body visibility may enhance the perception of naturalness for the glide style.