Tactile exploration is essential for blind and low vision (BLV) individuals to understand objects and spaces. Yet little is known about how camera-based devices can support hand-centric exploration: tactilely examining exhibits while inquiring about and processing information. We investigate a finger-worn ring camera that captures images from the palm side while allowing tactile exploration, comparing it with hand-centered smartphones. We conducted a Wizard-of-Oz study with 11 BLV participants in a science museum. Results showed that the ring camera supported effective bimanual strategies: exploring with both hands, lifting the camera-worn hand while keeping the other as an anchor during inquiry, and resuming bimanual touch for information processing. In contrast, smartphones led to effortful, fragmented exploration. Building on these findings, we developed an interactive system and evaluated its reliability and practicality with 6 BLV participants. We contribute insights and design implications for wearable camera systems that augment tactile exploration in real-world settings.

Blind people face persistent challenges in learning body movements such as exercise, dance, and rehabilitation routines. Verbal instructions are widely used but often ambiguous, while tactile graphics or 3D models can illustrate static postures but not transitions. Humanoid robots can present dynamic motions, suggesting potential to convey trajectories and simultaneous limb actions. We conducted an exploratory study with 11 blind participants comparing humanoid robot demonstrations with audio instructions. Quantitative evaluation assessed reproduction accuracy, learning time, and usability ratings, while qualitative interviews captured perceived benefits and challenges. Results show that simple movements could be conveyed through both modalities, but robots were particularly effective for complex transitions and concurrent limb coordination that audio could not easily express. These findings highlight design opportunities for integrating multimodal instruction to support movement learning for blind people.

Visual impairments create barriers to learning physical activities, since conventional training methods rely on visual demonstrations or often inadequate verbal descriptions. This research explores 3D-printed human body models to enhance movement comprehension for blind individuals. Through a participatory design approach in collaboration with a blind designer, we developed detailed 3D models representing various body movements and incorporated tactile reference elements to enhance spatial understanding. We conducted two user studies with 10 blind participants across different activities: static yoga poses and sequential calisthenic movements. The results demonstrated that 3D models significantly improved understanding speed, reduced questions for clarification, and enhanced movement accuracy compared to conventional teaching methods. Participants consistently rated 3D models higher for ease of understanding, effectiveness, and motivation.

Tactile picture books, which incorporate transparent embossed tactile graphics directly onto illustrations, allow visually impaired individuals to enjoy picture books.However, the complexity of converting illustrations into tactile graphics while preserving the original context limits the availability of such books.This study explores the potential of Generative AI (GenAI) by evaluating tactile graphics produced through a synergistic process involving GenAI and the expertise of tactile graphic designers, comparing them with graphics created using conventional methods.We conducted a comparative study with two totally blind participants, each with over 40 years of experience supervising tactile picture books, evaluating seven tactile graphics created by both approaches.Results revealed that some GenAI-generated graphics were rated as highly as conventionally produced graphics in terms of tactile interpretability and informative effectiveness.However, challenges were also identified in some GenAI outputs, such as extremely thin lines or unclear overlapping elements.Our findings suggest that while GenAI-assisted tactile graphics have the potential to generate contextually understandable illustrations through touch, a humanin-the-loop approach remains essential to address the unique design requirements of tactile perception.

This paper presents a sign language conversation system based on the See-Through Face Display to address the challenge of maintaining eye contact in remote sign language interactions. A camera positioned behind a transparent display allows users to look at the face of their conversation partner while appearing to maintain direct eye contact. Unlike conventional methods that rely on software-based gaze correction or large-scale half-mirror setups, this design reduces visual distortions and simplifies installation. We implemented and evaluated a videoconferencing system that integrates See-Through Face Display, comparing it to traditional videoconferencing methods. We explore its potential applications for Deaf and Hard of Hearing (DHH), including multi-party sign language conversations, corpus collection, remote interpretation, and AI-driven sign language avatars. Collaboration with DHH communities will be key to refining the system for real-world use and ensuring its practical deployment.

The future of the metaverse will feature the sociality of SNS and the openness of the WWW, however, the problem of asset data being ripped and leaked to third parties remains persistent. In this demo, we propose Mirrored Billboards as a simple yet effective approach to mitigate this issue. Mirrored Billboards adopt object-level remote rendering, allowing asset owners to stream viewpoint-dependent images to an unspecified audience in the metaverse, so that the streamed content behaves as if it were an authentic 3D object. As a side effect, Mirrored Billboards also make it possible to introduce new types of assets—such as recent realistic gaussian avatar formats—into the metaverse in the form of images, regardless of whether the metaverse application itself natively supports those formats. Through this demo, we demonstrate that Mirrored Billboards can be realized at practical cost and contribute to the extensibility of the metaverse, for example by enabling safer models of asset sales and rentals in metaverse environments.

This research represents the initial stages of our efforts to establish a digital library run by visually impaired individuals. We use the Action Research (AR) methodology, which involves researchers and community members collaborating to solve real-world problems. Our study focuses on documenting the process of creating a non-profit organization (NPO), building a supportive community, and gathering insights through interviews with key stakeholders. Specifically, the lead author established an NPO registered under Article 37, Paragraph 3 of the Japanese Copyright Act and partnered with the National Diet Library to upload accessible e-books. The NPO expanded its network through local business events, university collaborations, and social entrepreneur communities. Interviews underscored a decline in Braille library volunteers and severe delays in accessibility resources for minority languages like Mongolian. These findings will shape our future efforts to improve accessibility and self-directed learning for visually impaired individuals.

Visually impaired individuals often face challenges related to physical inactivity, stemming from limited accessibility to sports activities. While assistive technologies and adaptive sports have made progress, high-intensity contact sports remain largely inaccessible. This paper presents an approach to improve the accessibility of kickboxing for visually impaired individuals. We prioritize simple, immediate solutions that maximize the use of residual vision, diverging from conventional assistive technologies that rely on auditory or tactile feedback. Employing a participatory design methodology, we involved visually impaired individuals throughout the development process of specialized kickboxing equipment. This approach enabled us to address user-specific needs and challenges directly. Our research contributes to expanding sports participation opportunities for the visually impaired and fostering a more inclusive sports environment. We detail the development process of our adaptive kickboxing equipment and discuss its effectiveness in enhancing the sports experience for visually impaired individuals.

Tactile interfaces such as embossment facilitate information transfer through touch in Human-Computer Interaction (HCI). Traditional embossing methods, while enabling the creation of intricate patterns, face limitations due to mold reliance and material thickness restrictions, hindering bespoke embossment creation. In this study, we propose High-Intensity Focused Ultrasound (HIFU) as an alternative technique to produce tailored embossed designs on acrylic without the need for traditional molds. We uncover specific HIFU parameters, such as amplitude, irradiation time, and distance that directly impact essential qualities of embossment including embossment height, transparency, and line generation. Additionally, the capability of embossing without the use of molds expands the applications for quick prototyping and customization of embossed designs within HCI. Furthermore, we introduce a user interface developed to streamline the design and application of customizable tactile graphics using HIFU, aimed at non-expert users. Preliminary user studies reveal positive feedback on the interface’s intuitiveness and the quality of the HIFU embossment. Our study indicates that HIFU embossment presents a viable approach for creating embossed features in interactive systems, with the potential to offer methods for personal customization in the design of tactile materials.

When playing musical instruments, deaf and hard-of-hearing (DHH) people typically sense their music from the vibrations transmitted by the instruments or the movements of their bodies while performing. Sensory substitution devices now exist that convert sounds into light and vibrations to support DHH people’s musical activities. However, these devices require specialized hardware, and the marketing profiles assume that standard musical instruments are available. Hence, a significant gap remains between DHH people and their musical performance enjoyment. To address this issue, this study identifies end users’ preferred gestures when using smartphones to emulate the musical experience based on the instrument selected. This gesture elicitation study applies 10 instrument types. Herein, we present the results and a new taxonomy of musical instrument gestures. The findings will support the design of gesture-based instrument interfaces to enable DHH people to more directly enjoy their musical performances.

It is important for photographers to have the best possible lighting configuration at the time of shooting; otherwise, they need post-processing on images, which may cause artifacts and deterioration. Thus, photographers often struggle to find the best possible lighting configuration by manipulating lighting devices, including light sources and modifiers, in a trial-and-error manner. In this paper, we propose a novel computational framework to support photographers. This framework assumes that every lighting device is programmable; that is, its adjustable parameters (e.g., orientation, intensity, and color temperature) can be set using a program. Using our framework, photographers do not need to learn how the parameter values affect the resulting lighting, and even do not need to determine the strategy of the trial-and-error process; instead, photographers need only concentrate on evaluating which lighting configuration is more desirable among options suggested by the system. The framework is enabled by our novel photographer-in-the-loop Bayesian optimization, which is sample-efficient (i.e., the number of required evaluation steps is small) and which can also be guided by providing a rough painting of the desired lighting configuration if any. We demonstrate how the framework works in both simulated virtual environments and a physical environment, suggesting that it could find pleasing lighting configurations quickly in around 10 iterations. Our user study suggests that the framework enables the photographer to concentrate on the look of captured images rather than the parameters, compared with the traditional manual lighting workflow.

Deaf and hard of hearing (DHH) people have limited access to auditory input, so they mainly receive visual information during online meetings. In recent years, the usability of a system that visualizes the ongoing topic in a conference has been confirmed, but it has not been verified in a remote conference that includes DHH people. One possible reason is that visual dispersion occurs when there are multiple sources of visual information. In this study, we introduce “Word Cloud for Meeting,” a system that generates a separate word cloud for each participant and displays it in the background of each participant’s video to visualize who is saying what. We conducted an experiment with seven DHH participants and obtained positive qualitative feedback on the ease of recognizing topic changes. However, when the topic changed in a sequence, it was found to be distracting. Additionally, we discuss the design implications for visualizing topics for DHH people in online meetings.

Real-time captioning is a useful technique for deaf and hard-of-hearing (DHH) people to talk to hearing people. With the improvement in device performance and the accuracy of automatic speech recognition (ASR), real-time captioning is becoming an important tool for helping DHH people in their daily lives. To realize higher-quality communication and overcome the limitations of mobile and augmented-reality devices, real-time captioning that can be used comfortably while maintaining nonverbal communication and preventing incorrect recognition is required. Therefore, we propose a real-time captioning system that uses a transparent display. In this system, the captions are presented on both sides of the display to address the problem of incorrect ASR results, and the highly transparent display makes it possible to see both the body language and the captions.

In the midst of the coronavirus disease 2019 pandemic, online meetings are rapidly increasing. Deaf or hard of hearing (DHH) people participating in an online meeting often face difficulties in capturing the affective states of other speakers. Recent studies have shown the effectiveness of emoji-based representation of spoken text to capture such affective states. Nevertheless, in voice-only online meetings, it is still not clear how emoji-based spoken texts can assist DHH people to understand the feelings of speakers without perceiving their facial expressions. We therefore conducted a preliminary experiment to understand the effect of emoji-based text representation during voice-only online meetings by leveraging an emoji-based captioning system. Our preliminary results demonstrate the necessity of designing an advanced system to help DHH people understanding the voice-only online meetings more meaningfully.

Live streaming is becoming an increasingly popular medium for social interaction on mobile devices. However, in most cases live streaming experience is still confined to a single-viewpoint, monoscopic video stream, which, in active scenarios such as outdoor or public event broadcasting, makes it hard or even impossible for viewers to understand a remote situation. This study investigates how live stream viewing can be improved through introduction of user spatial data into video stream composition and interaction, by developing a system that allows watching multiple spatially oriented video streams in virtual spaces simultaneously, and interaction with them through voice and spatially distributed reaction buttons. According to results of a preliminary study, users show a statistically significant interest in using spatial live stream viewing and interaction features. Believing that this solution presents a novel approach to social live video streaming, the rest of the article discusses the results and possible extensions of developed system.

Deep neural networks (DNNs) applications are now increasingly pervasive and powerful. However, fashion designers are lagging behind in leveraging this increasingly common technology. DNNs are not yet a standard part of fashion de sign practice, either clothes patterns or prototyping tools. In this paper, we present DeepWear, a method using deep convolutional generative adversarial networks for clothes design. The DNNs learn the feature of specific brand clothes and generate images then patterns instructed from the images are made, and an author creates clothes based on that. We evaluated this system by evaluating the credibility of the actual sold clothes on market with our clothes. As the result, we found it is possible to make clothes look like actual products from the generated images. Our findings have implications for collaborative design between machine and human intelligence.

This paper presents EXController, a new controller-mounted finger posture recognition device specially designed for VR handheld controllers. We seek to provide additional input through real-time vision sensing by attaching a near infrared (NIR) camera onto the controller. We designed and implemented an exploratory prototype with a HTC Vive controller. The NIR camera is modified from a traditional webcam and applied with a data-driven Convolutional Neural Network (CNN) classifier. We designed 12 different finger gestures and trained the CNN classifier with a dataset from 20 subjects, achieving an average accuracy of 86.17% across - subjects, and, approximately more than 92% on three of the finger postures, and more than 89% on the top-4 accuracy postures. We also developed a Unity demo that shows matched finger animations, running at approximately 27 fps in real-time.

We propose a novel method of spatial audio rendering using ultrasound. An ultrasonic phased array generates one or more focal points in air, and they act as point sources of audible sound when the ultrasound waves are modulated. Our sound-point loudspeaker has two major advantages over conventional ultrasound-based sound-beam (superdirectional) loudspeakers. The higher audience selectivity means that our sound-point loudspeaker can deliver sound to the ears of the target person, whereas a sound-beam loudspeaker delivers sound to not only the target person but also other persons standing in the same direction. The other advantage is lower exposure to ultrasound; while an audible sound beam travels along an ultrasonic beam in a soundbeam loudspeaker, audible sound can be heard along the direction perpendicular to the ultrasonic beam in our soundpoint loudspeaker. This paper reports the principles of our sound-point loudspeaker, prototype construction, evaluation, and applications.

We propose an immersive telepresence system for puppetry that transmits a human performer's body and facial movements into a puppet with audiovisual feedback to the performer. The cameras carried in place of puppet's eyes stream live video to the HMD worn by the performer, so that performers can see the images from the puppet's eyes with their own eyes and have a visual understanding of the puppet's ambience. In conventional methods to manipulate a puppet (a hand-puppet, a string-puppet, and a rod-puppet), there is a need to practice manipulating puppets, and there is difficulty carrying out interactions with the audience. Moreover, puppeteers must be positioned exactly where the puppet is. The proposed system addresses these issues by enabling a human performer to manipulate the puppet remotely using his or her body and facial movements. We conducted several user studies with both beginners and professional puppeteers. The results show that, unlike the conventional method, the proposed system facilitates the manipulation of puppets especially for beginners. Moreover, this system allows performers to enjoy puppetry and fascinate audiences.

We propose a system for transmitting a human performer's body and facial movements to a puppet with audiovisual feedback to the performer. The system consists of a head-mounted display (HMD) that shows the performer the video recording of the puppet's view, a microphone for voice capture, and photoreflectors for detecting the mouth movements of the human performer. In conventional puppetry, there is also the need for practice in the manipulation of the puppets to achieve good performance. The proposed telepresence system addresses these issues by enabling the human performer to manipulate the puppet through their own body and facial movements. The proposed system is expected to contribute to the development of new applications of puppetry and expand the interactivity of puppetry and the scope of entertainment.

We introduce Bubble Cloud ---a technology that uses a bubble cluster for displaying images onto a screen. Bubble Cloud projects images onto bubbles without confining a fog and can float in the air by changing an operating condition of the bubble cluster such as altering the size of the bubble. Moreover, the user can modify the display because its shape is composed of clusters. A floating bubble cluster can be controlled and adjusted by hand in arbitrary directions by electrifying the bubble cluster.

We propose Human Coded Orchestra, a new approach to enable a group of individuals to sing in harmony by using computed directional speakers. The possibilities of musical performance by an untrained group have been explored in the fields of science and art. However, previous work has rarely proceeded beyond simple rhythm-based music and failed to achieve musical complexity. Human Coded Orchestra employs a number of directional speakers, each set at a different pitch, enabling them to deliver different pitches to each participant to sing to, according to their positions. Experiments demonstrated that participants succeeded in singing in harmony extemporaneously, and they reported that they enjoyed both the experience of singing and the feeling that they were able to participate in an activity with others. Our system does not require preparation on the part of singers, which opens up the possibility of practical application in the area of interactive performance.

We present a new method of rendering aerial haptic images that uses femtosecond-laser light fields and ultrasonic acoustic fields. In conventional research, a single physical quantity has been used to render aerial haptic images. In contrast, our method combines multiple fields (light and acoustic fields) at the same time. While these fields have no direct interference, combining them provides benefits such as multi-resolution haptic images and a synergistic effect on haptic perception. We conducted user studies with laser haptics and ultrasonic haptics separately and tested their superposition. The results showed that the acoustic field affects the tactile perception of the laser haptics. We explored augmented reality/virtual reality (AR/VR) applications such as providing haptic feedback of the combination of these two methods. We believe that the results of this study contribute to the exploration of laser haptic displays and expand the expression of aerial haptic displays based on other principles.

We present a novel manipulation method that subconsciously changes the walking direction of users via visual processing on a head mounted display (HMD). Unlike existing navigation systems that require users to recognize information and then follow directions as two separate, conscious processes, the proposed method guides users without them needing to pay attention to the information provided by the navigation system and also allows them to be graphically manipulated by controllers. In the proposed system, users perceive the real world by means of stereo images provided by a stereo camera and the HMD. Specifically, while walking, the navigation system provides users with real-time feedback by processing the images they have just perceived and giving them visual stimuli. This study examined two image-processing methods for manipulation of human's walking direction: moving stripe pattern and changing focal region. Experimental results indicate that the changing focal region method most effectively leads walkers as it changes their walking path by approximately 200 mm/m on average.

We present a method of rendering aerial and volumetric graphics using femtosecond lasers. A high-intensity laser excites physical matter to emit light at an arbitrary three-dimensional position. Popular applications can thus be explored, especially because plasma induced by a femtosecond laser is less harmful than that generated by a nanosecond laser. There are two methods of rendering graphics with a femtosecond laser in air: producing holograms using spatial light modulation technology and scanning of a laser beam by a galvano mirror. The holograms and workspace of the system proposed here occupy a volume of up to 1 cm 3 ; however, this size is scalable depending on the optical devices and their setup. This article provides details of the principles, system setup, and experimental evaluation, and discusses the scalability, design space, and applications of this system. We tested two laser sources: an adjustable (30--100fs) laser that projects up to 1,000 pulses/s at an energy of up to 7mJ/pulse and a 269fs laser that projects up to 200,000 pulses/s at an energy of up to 50μJ/pulse. We confirmed that the spatiotemporal resolution of volumetric displays implemented using these laser sources is 4,000 and 200,000 dots/s, respectively. Although we focus on laser-induced plasma in air, the discussion presented here is also applicable to other rendering principles such as fluorescence and microbubbles in solid or liquid materials.

We propose a novel graphics system based on the expansion of 3D acoustic-manipulation technology. In conventional research on acoustic levitation, small objects are trapped in the acoustic beams of standing waves. We expand this method by changing the distribution of the acoustic-potential field (APF). Using this technique, we can generate the graphics using levitated small objects. Our approach makes available many expressions, such as the expression by materials and non-digital appearance. These kinds of expressions are used in many applications, and we aim to combine them with digital controllability. In the current system, multiple particles are levitated together at 4.25-mm intervals. The spatial resolution of the position is 0.5 mm. Particles move at up to 72 cm/s. The allowable density of the material can be up to 7 g/cm 3 . For this study, we use three options of APF: 2D grid, high-speed movement, and combination with motion capture. These are used to realize floating screen or mid-air raster graphics, mid-air vector graphics, and interaction with levitated objects. This paper reports the details of the acoustic-potential field generator on the design, control, performance evaluation, and exploration of the application space. To discuss the various noncontact manipulation technologies in a unified manner, we introduce a concept called "computational potential field" (CPF).