This paper presents a sign language conversation system based on the See-Through Face Display to address the challenge of maintaining eye contact in remote sign language interactions. A camera positioned behind a transparent display allows users to look at the face of their conversation partner while appearing to maintain direct eye contact. Unlike conventional methods that rely on software-based gaze correction or large-scale half-mirror setups, this design reduces visual distortions and simplifies installation. We implemented and evaluated a videoconferencing system that integrates See-Through Face Display, comparing it to traditional videoconferencing methods. We explore its potential applications for Deaf and Hard of Hearing (DHH), including multi-party sign language conversations, corpus collection, remote interpretation, and AI-driven sign language avatars. Collaboration with DHH communities will be key to refining the system for real-world use and ensuring its practical deployment.

Real-time captions presented through AR glasses hold great promise for enhancing accessibility for Deaf and Hard of Hearing (DHH) individuals. However, conventional approaches often overlook the diversity of user preferences and perceptual characteristics. In this study, we explore how spatial personalization—adjustments in display method, position, and depth—can improve the user experience of AR captions.We conducted two experiments. Experiment 1 focused on 2D position personalization, revealing that user-controlled positioning enhanced visibility and comfort, though comprehension gains were modest. Experiment 2 introduced a novel concept, Simulated Depth Personalization, which emulates different viewing distances despite a fixed AR focal plane (4 m). This revealed clear individual differences: participants varied in whether their preferences aligned more with the presentation method (e.g., monocular vs. binocular) or viewing distance.Our findings emphasize that effective AR captioning is not one-size-fits-all but must flexibly adapt to individual spatial preferences. We argue for incorporating spatial personalization as a core design principle for accessible AR systems.

Auditory directionality remains challenging for d/Deaf and hard of hearing (DHH) people. We evaluated a shoulder-mounted tactile system that vibrates bilaterally to signal sound direction and urgency in multi-person meetings. Eight DHH participants watched subtitled VR meetings with and without the device, pressing a trigger when their name or an alarm appeared and selecting the speaker afterward. The device increased reaction count for name calls and shortened reaction time for both name and alarm alerts (all p < 0.05), while speaker identification accuracy stayed high and unchanged. Post-experiment surveys confirmed strong demand for directional cues (p < 0.01) and alarm sound alerts (p < 0.05). Reflective Thematic Analysis of open-ended responses highlighted (i) the need to customize target sounds, (ii) meeting-focused use cases, and (iii) requests for refined vibration patterns and lighter design. These findings demonstrate the system’s value for timely notifications and emphasize directional feedback and customizability as critical to practical adoption.

Abstract While AI technology is becoming increasingly prevalent in our daily lives, the comprehension of machine learning (ML) among non-experts remains limited. Interactive machine learning (IML) has the potential to serve as a tool for end users, but many existing IML systems are designed for users with a certain level of expertise. Consequently, it remains unclear whether IML experiences can enhance the comprehension of ordinary users. In this study, we conducted a public event using an IML system to assess whether participants could gain technical comprehension through hands-on IML experiences. We implemented an interactive sound classification system featuring visualization of internal feature representation and invited visitors at a science museum to freely interact with it. By analyzing user behavior and questionnaire responses, we discuss the potential and limitations of IML systems as a tool for promoting technical comprehension among non-experts.

When playing musical instruments, deaf and hard-of-hearing (DHH) people typically sense their music from the vibrations transmitted by the instruments or the movements of their bodies while performing. Sensory substitution devices now exist that convert sounds into light and vibrations to support DHH people’s musical activities. However, these devices require specialized hardware, and the marketing profiles assume that standard musical instruments are available. Hence, a significant gap remains between DHH people and their musical performance enjoyment. To address this issue, this study identifies end users’ preferred gestures when using smartphones to emulate the musical experience based on the instrument selected. This gesture elicitation study applies 10 instrument types. Herein, we present the results and a new taxonomy of musical instrument gestures. The findings will support the design of gesture-based instrument interfaces to enable DHH people to more directly enjoy their musical performances.

Deaf and hard of hearing (DHH) people have limited access to auditory input, so they mainly receive visual information during online meetings. In recent years, the usability of a system that visualizes the ongoing topic in a conference has been confirmed, but it has not been verified in a remote conference that includes DHH people. One possible reason is that visual dispersion occurs when there are multiple sources of visual information. In this study, we introduce “Word Cloud for Meeting,” a system that generates a separate word cloud for each participant and displays it in the background of each participant’s video to visualize who is saying what. We conducted an experiment with seven DHH participants and obtained positive qualitative feedback on the ease of recognizing topic changes. However, when the topic changed in a sequence, it was found to be distracting. Additionally, we discuss the design implications for visualizing topics for DHH people in online meetings.

Real-time captioning is a useful technique for deaf and hard-of-hearing (DHH) people to talk to hearing people. With the improvement in device performance and the accuracy of automatic speech recognition (ASR), real-time captioning is becoming an important tool for helping DHH people in their daily lives. To realize higher-quality communication and overcome the limitations of mobile and augmented-reality devices, real-time captioning that can be used comfortably while maintaining nonverbal communication and preventing incorrect recognition is required. Therefore, we propose a real-time captioning system that uses a transparent display. In this system, the captions are presented on both sides of the display to address the problem of incorrect ASR results, and the highly transparent display makes it possible to see both the body language and the captions.

In the midst of the coronavirus disease 2019 pandemic, online meetings are rapidly increasing. Deaf or hard of hearing (DHH) people participating in an online meeting often face difficulties in capturing the affective states of other speakers. Recent studies have shown the effectiveness of emoji-based representation of spoken text to capture such affective states. Nevertheless, in voice-only online meetings, it is still not clear how emoji-based spoken texts can assist DHH people to understand the feelings of speakers without perceiving their facial expressions. We therefore conducted a preliminary experiment to understand the effect of emoji-based text representation during voice-only online meetings by leveraging an emoji-based captioning system. Our preliminary results demonstrate the necessity of designing an advanced system to help DHH people understanding the voice-only online meetings more meaningfully.