The quality of piano performance depends on nuanced timing, articulation, and dynamic control, but practice feedback is often summary-based and hard to act on. We introduce Profy, a weakly supervised system that learns from take-level labels derived from aggregated listener ratings (expert-labeled vs. amateur-labeled) to produce time-aligned highlights for review during piano practice. We collected synchronized 1 kHz key-motion and audio from 73 pianists and used 1,083 valid takes for modeling and evaluation. The model outputs clip-level predictions together with evidence scores on a shared resampled model time base for visualization. On 20 amateur clips from short technique studies annotated by 21 expert pianists, the displayed highlight score aligns with passages that expert pianists marked for review despite training without localized labels (Pearson r=0.61, ROC-AUC 0.75). Rather than summarizing a take with a single global score, Profy helps learners decide where to inspect next by supporting scrubbing, looping, and focused replay of time-localized passages associated with expert-amateur differences.

Silent and whispered speech offer promise for always-available voice interaction with AI, yet existing methods struggle to balance vocabulary size, wearability, silence, and noise robustness. We present NasoVoce, a nose-bridge–mounted interface that integrates a microphone and a vibration sensor. Positioned at the nasal pads of smart glasses, it unobtrusively captures both acoustic and vibration signals. The nasal bridge, close to the mouth, allows access to bone- and skin-conducted speech and enables reliable capture of low-volume utterances such as whispered speech. While the microphone captures high-quality audio, it is highly sensitive to environmental noise. Conversely, the vibration sensor is robust to noise but yields lower signal quality. By fusing these complementary inputs, NasoVoce generates high-quality speech robust against interference. Evaluation with Whisper Large-v2, PESQ, STOI, and MUSHRA ratings confirms improved recognition and quality. NasoVoce demonstrates the feasibility of a practical interface for always-available, continuous, and discreet AI voice conversations.

This paper investigates how visual texture presentation influences tactile perception when interacting with electrostatic cloth displays. We propose a visuo-haptic system that allows users to pinch and rub virtual fabrics while feeling realistic frictional sensations modulated by electrostatic actuation. Through a user study, we examined the cross-modal effects between visual roughness and perceived tactile friction. The results demonstrate that visually rough textures amplify the perceived frictional force, even under identical electrostatic stimuli. These findings contribute to the understanding of multimodal texture perception and provide design insights for haptic feedback in virtual material interfaces.

In today's globalized world, there are increasing opportunities for individuals to communicate using a common non-native language (lingua franca). Non-native speakers often have opportunities to listen to foreign languages, but may not comprehend them as fully as native speakers do. To aid real-time comprehension, live transcription of subtitles is frequently used in everyday life (e.g., during Zoom conversations, watching YouTube videos, or on social networking sites). However, simultaneously reading subtitles while listening can increase cognitive load. In this study, we propose Dynamik, a system that reduces cognitive load during reading by decreasing the size of less important words and enlarging important ones, thereby enhancing sentence contrast. Our results indicate that Dynamik can reduce certain aspects of cognitive load, specifically, participants' perceived performance and effort among individuals with low proficiency in English, as well as enhance the users' sense of comprehension, especially among people with low English ability. We further discuss our methods' applicability to other languages and potential improvements and further research directions.

Visual and auditory guidance in smart glasses suffers from distraction and noise sensitivity. To address these limitations, haptic feedback has been explored, and nose-pad vibrotactile navigation offers a promising approach, yet the effect of stimulus intervals is unexplored. This study compared cueing intervals of 0.3 s (high frequency) and 1.0 s (low frequency) in a direction-finding task. Results showed that the high-frequency condition reduced task completion time but exhibited overshooting tendencies and induced user discomfort. Although both conditions showed comparable workload and usability, participants reported trade-offs between responsiveness and comfort, suggesting context-adaptive intervals.

Haptic displays play an important role in enhancing the sense of presence in VR and telepresence. Displaying the tactile properties of fabrics has potential in the fashion industry, but there are difficulties in dynamically displaying different types of tactile sensations while maintaining their flexible properties. The vibrotactile stimulation of fabrics is an important element in the tactile properties of fabrics, as it greatly affects the way a garment feels when rubbed against the skin. To dynamically change the vibrotactile stimuli, many studies have used mechanical actuators. However, when combined with fabric, the soft properties of the fabric are compromised by the stiffness of the actuator. In addition, because the vibration generated by such actuators is applied to a single point, it is not possible to provide a uniform tactile sensation over the entire surface of the fabric, resulting in an uneven tactile sensation. In this study, we propose a Pinching Tactile Display: a conductive cloth that changes the tactile sensation by controlling electrostatic adsorption. By controlling the voltage and frequency applied to the conductive cloth, different tactile sensations can be dynamically generated. This makes it possible to create a tactile device in which tactile sensations are applied to the entire fabric while maintaining the thin and soft characteristics of the fabric. As a result, users could experiment with tactile sensations by picking up and rubbing the fabric in the same way they normally touch it. This mechanism has the potential for dynamic tactile transformation of soft materials.

Silent speech recognition is a promising technology that decodes human speech without requiring audio signals, enabling private human-computer interactions. In this paper, we propose Watch Your Mouth, a novel method that leverages depth sensing to enable accurate silent speech recognition. By leveraging depth information, our method provides unique resilience against environmental factors such as variations in lighting and device orientations, while further addressing privacy concerns by eliminating the need for sensitive RGB data. We started by building a deep-learning model that locates lips using depth data. We then designed a deep learning pipeline to efficiently learn from point clouds and translate lip movements into commands and sentences. We evaluated our technique and found it effective across diverse sensor locations: On-Head, On-Wrist, and In-Environment. Watch Your Mouth outperformed the state-of-the-art RGB-based method, demonstrating its potential as an accurate and reliable input technique.

Juiciness is visual pizzazz used to improve player experience and engagement in games. Most research has focused on juicy particle effects. However, text effects are also commonly used in games, albeit not always juiced up. One type is onomatopoeia, a well-defined element of human language that has been translated to visual media, such as comic books and games. Another is semantic text, often used to provide performance feedback in games. In this work, we explored the relationship between juiciness and text effects, aiming to replicate juicy user experiences with text-based juice and combining particle and text juice. We show in a multi-phase within-subjects experiment that users rate juicy text effects similarly to particles effects, with comparable performance, and more reliable feedback. We also hint at potential improvement in user experience when both are combined, and how text stimuli may be perceived differently than other visual ones. We contribute empirical findings on the juicy-text connection in the context of visual effects for interactive media.

Recognizing whispered speech and converting it to normal speech creates many possibilities for speech interaction. Because the sound pressure of whispered speech is significantly lower than that of normal speech, it can be used as a semi-silent speech interaction in public places without being audible to others. Converting whispers to normal speech also improves the speech quality for people with speech or hearing impairments. However, conventional speech conversion techniques do not provide sufficient conversion quality or require speaker-dependent datasets consisting of pairs of whispered and normal speech utterances. To address these problems, we propose WESPER, a zero-shot, real-time whisper-to-normal speech conversion mechanism based on self-supervised learning. WESPER consists of a speech-to-unit (STU) encoder, which generates hidden speech units common to both whispered and normal speech, and a unit-to-speech (UTS) decoder, which reconstructs speech from the encoded speech units. Unlike the existing methods, this conversion is user-independent and does not require a paired dataset for whispered and normal speech. The UTS decoder can reconstruct speech in any target speaker’s voice from speech units, and it requires only an unlabeled target speaker’s speech data. We confirmed that the quality of the speech converted from a whisper was improved while preserving its natural prosody. Additionally, we confirmed the effectiveness of the proposed approach to perform speech reconstruction for people with speech or hearing disabilities.

Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.

The tactile sensation of textiles is critical in determining the comfort of clothing. For remote use, such as online shopping, users cannot physically touch the textile of clothes, making it difficult to evaluate its tactile sensation. Tactile sensing and actuation devices are required to transmit the tactile sensation of textiles. The sensing device needs to recognize different garments, even with hand-held sensors. In addition, the existing actuation device can only present a limited number of known patterns and cannot transmit unknown tactile sensations of textiles. To address these issues, we propose Telextiles, an interface that can remotely transmit tactile sensations of textiles by creating a latent space that reflects the proximity of textiles through contrastive self-supervised learning. We confirm that textiles with similar tactile features are located close to each other in the latent space through a two-dimensional plot. We then compress the latent features for known textile samples into the 1D distance and apply the 16 textile samples to the rollers in the order of the distance. The roller is rotated to select the textile with the closest feature if an unknown textile is detected.

This paper introduces Retinal Homing Display, which presents focus-free stereoscopic images via retinal projection, thus eliminating the need for the user to wear additional equipment. Traditional 3D displays, typically classified as either naked-eye stereoscopic or wearable, present inherent challenges: the former involves a compromise between resolution and accurate depth perception, while the latter imposes an additional burden on the user. Our proposed display employs optical and mechanical mechanisms to converge projector light at the user’s pupil center, simultaneously tracking eye movements. This lets the user perceive focus-free, high-resolution stereoscopic images without wearable equipment. We implemented a proof-of-concept system utilizing a robotic arm and a Dihedral Corner Reflector Array (DCRA), subsequently evaluating image quality and its eyebox. Finally, we discuss the limitations of the current prototype and outline potential directions for future research.

Speech is inappropriate in many situations, limiting when voice control can be used. Most unvoiced speech text entry systems can not be used while on-the-go due to movement artifacts. Using a dental retainer with capacitive touch sensors, SilentSpeller tracks tongue movement, enabling users to type by spelling words without voicing. SilentSpeller achieves an average 97% character accuracy in offline isolated word testing on a 1164-word dictionary. Walking has little effect on accuracy; average offline character accuracy was roughly equivalent on 107 phrases entered while walking (97.5%) or seated (96.5%). To demonstrate extensibility, the system was tested on 100 unseen words, leading to an average 94% accuracy. Live text entry speeds for seven participants averaged 37 words per minute at 87% accuracy. Comparing silent spelling to current practice suggests that SilentSpeller may be a viable alternative for silent mobile text entry.

Voice interactive devices often use keyword spotting for device activation. However, this approach suffers from misrecognition of keywords and can respond to keywords not intended for calling the device (e.g., ”You can ask Alexa about it.”), causing accidental device activations. We propose a method that leverages prosodic features to differentiate calling/not-calling voices (F1 score: 0.869), allowing devices to respond only when called upon to avoid misactivation. As a proof of concept, we built a prototype smart speaker called Aware that allows users to control the device activation by speaking the keyword in specific prosody patterns. These patterns are chosen to represent people’s natural calling/not-calling voices, which are uncovered in a study to collect such voices and investigate their prosodic difference. A user study comparing Aware with Amazon Echo shows Aware can activate more correctly (F1 score 0.93 vs. 0.56) and is easy to learn and use.

When beginners learn to speak a non-native language, it is difficult for them to judge for themselves whether they are speaking well. Therefore, computer-assisted pronunciation training systems are used to detect learner mispronunciations. These systems typically compare the user’s speech with that of a native speaker in units of rhythm, phonemes, or words, and calculate the differences. However, they require extensive speech data with detailed annotations of non-native and native speakers, which are usually difficult to collect. To overcome this problem, we propose a new language learning support system that detects mispronunciations by beginners based on a small amount of unannotated native speaker speech data. The proposed system uses deep learning–based speech processing to display the pronunciation score of the learner’s speech and the difference/distance between the learner’s and the model’s pronunciation in an intuitively visual manner. Learners can gradually improve their pronunciation by eliminating differences and shortening the distance from the model until they become sufficiently proficient. We also built an application to help nonnative speakers learn English and confirmed that it can improve users’ speech intelligibility.

This article presents SSR7000, a corpus of synchronized ultrasound tongue and lip images designed for end-to-end silent speech recognition (SSR). Although neural end-to-end models are successfully updating the state-of-the-art technology in the field of automatic speech recognition, SSR research based on ultrasound tongue imaging has still not evolved past cascaded DNN-HMM models due to the absence of a large dataset. In this study, we constructed a large dataset, namely SSR7000, to exploit the performance of the end-to-end models. The SSR7000 dataset contains ultrasound tongue and lip images of 7484 utterances by a single speaker. It contains more utterances per person than any other SSR corpus based on ultrasound imaging. We also describe preprocessing techniques to tackle data variances that are inevitable when collecting a large dataset and present benchmark results using an end-to-end model. The SSR7000 corpus is publicly available under the CC BY-NC 4.0 license.

We propose a system displaying audience eye gaze and nod reactions for enhancing synchronous remote communication. Recently, we have had increasing opportunities to speak to others remotely. In contrast to offline situations, however, speakers often have difficulty observing audience reactions at once in remote communication, which makes them feel more anxious and less confident in their speeches. Recent studies have proposed methods of presenting various audience reactions to speakers. Since these methods require additional devices to measure audience reactions, they are not appropriate for practical situations. Moreover, these methods do not present overall audience reactions. In contrast, we design and develop CalmResponses, a browser-based system which measures audience eye gaze and nod reactions only with a built-in webcam and collectively presents them to speakers. The results of our two user studies indicated that the number of fillers in speaker’s speech decreases when audiences’ eye gaze is presented, and their self-rating score increases when audiences’ nodding is presented. Moreover, comments from audiences suggested benefits of CalmResponses for them in terms of co-presence and privacy concerns.

Interactions based on automatic speech recognition (ASR) have become widely used, with speech input being increasingly utilized to create documents. However, as there is no easy way to distinguish between commands being issued and text required to be input in speech, misrecognitions are difficult to identify and correct, meaning that documents need to be manually edited and corrected. The input of symbols and commands is also challenging because these may be misrecognized as text letters. To address these problems, this study proposes a speech interaction method called DualVoice, by which commands can be input in a whispered voice and letters in a normal voice. The proposed method does not require any specialized hardware other than a regular microphone, enabling a complete hands-free interaction. The method can be used in a wide range of situations where speech recognition is already available, ranging from text input to mobile/wearable computing. Two neural networks were designed in this study, one for discriminating normal speech from whispered speech, and the second for recognizing whisper speech. A prototype of a text input system was then developed to show how normal and whispered voice can be used in speech text input. Other potential applications using DualVoice are also discussed.

While presenting a photorealistic appearance plays a major role in immersion in Augmented Virtuality environment, displaying that of real objects remains a challenge. Recent developments in photogrammetry have facilitated the incorporation of real objects into virtual space. However, reproducing complex appearances, such as subsurface scattering and transparency, still requires a dedicated environment for measurement and possesses a trade-off between rendering quality and frame rate.

This paper illustrates design work carried out to develop an interactive theater performance. HCI has started to address the challenges of designing interactive performances, as both audience and performers’ experiences are considered and a variety of professional expertise involved. Nevertheless, research has overlooked how such design unfolds in practice, and what role artists play in exploring both the creative opportunities and the challenges associated with interweaving digital technologies. A two-day workshop was conducted to tailor the use of the ChameleonMask, a telepresence technology, within a performance. The analysis highlights the artists’ work to make the mask work while framing, exploring and conceptualizing its use. The discussion outlines the artists’ skills and design expertise, and how they redefine the role of HCI in performance-led research.

Gastroenterological endoscopic surgery needs complex surgical skills such as a sensation of body movement and manipulation of the endoscope that is hard to be explained verbally. Prior research reported that endoscopic surgery is one of the most challenging surgery to teach. Thus, surgeons need long-term practice to master the skills. To support learning such skills, we developed a surgical scene replay system. First, we surveyed 12 surgeons to reveal the reason for the difficulty and elicited three requirements for our system: (1) provide multiple videos that include an endoscope, a fluoroscopy, and hand manipulation for the endoscope to observe the surgery from multiple aspects; (2) visualize an experts' gaze position to understand experts' intention of hand manipulation, and (3) enlarge the size of the gazed video to inform learners where they should pay attention to. Our user study with the system indicates that participants could understand the experts' intentions and tacit knowledge easier than the existing video materials.

Conversational agents are widely used in many situations, especially for speech tutoring. However, their contents and functions are often pre-defined and not customizable for people without technical backgrounds, thus significantly limiting their flexibility and usability. Besides, conventional agents often cannot provide feedback in the middle of training sessions because they lack technical approaches to evaluate users' speech dynamically. We propose JustSpeak: automated and interactive speech tutoring agents with various configurable feedback mechanisms, using any speech recordings with its transcription text as the template for speech training. In JustSpeak, we developed an automated procedure to generate customized tutoring agents from user-inputted templates. Moreover, we created a set of methods to dynamically synchronize speech recognizers' behavior with the agent's tutoring progress, making it possible to detect various speech mistakes dynamically such as being stuck, mispronunciation, and rhythm deviations. Furthermore, we identified the design primitives in JustSpeak to create different novel feedback mechanisms, such as adaptive playback, follow-on training, and passive adaptation. They can be combined to create customized tutoring agents, which we demonstrate with an example for language learning. We believe JustSpeak can create more personalized speech learning opportunities by enabling tutoring agents that are customizable, always available, and easy-to-use.

In this paper, we proposed the rotation based browsing method for video learning in personal training. SuppleView, which is flexible in respect of the user's physical position while viewing a video, enables coordinate translation free viewing between an observer and an actor. Previous work on video learning have not enough explored the limitation on the observation angle, although its angle effects for observer's comprehension and caused only in video learning not in observation with the actual trainer. The method solve this basic limitation by inferring the 3D pose of frames in a video. Based on those poses, we create an virtual agent with 3D model as an actor of movements, that is same with the movement in an original 2D video. The system transition for the two actors depends on the physical rotation of the user's head so that the angle of view for observing the actor also changes. Hence, the content rendering in proposed viewer could be provided to trainees as in kind-full form for their observation in the point of an observation angle of view. We report the method overview and our prototyping to show the proof of concept.

With the increased use of smart speakers, silent speech interaction (SSI) is attracting attention. Unfortunately, traditional silent speech interaction methods require the addition of obtrusive sensors and devices around the user's face, making wearability and portability a challenge. Considering that most uses for smart speakers do not require many words, we suggest a more casual approach, TieLent, which can easily be worn between the neck and the chest. TieLent's RGB camera is set away from the user's face, presenting less interference with the user. Although TieLent's camera is not able to capture the whole mouth, when combined with our image-to-speech neural network model, it is able to generate the recognizable speech of 15 commands with an average accuracy of 94%.

Learning to speak in foreign languages is hard. Speech shadowing has been rising as a proven way to practice speaking, which asks a learner to listen and repeat a native speech template as simultaneously as possible. However, shadowing can be hard to do because learners can frequently fail to follow the speech and unintentionally interrupt a practice session. Worse, as a technical way to evaluate shadowing performance in real-time has not been established, no automated solutions are available to help. In this paper, we propose a technical framework with context-dependent speech recognition to evaluate shadowing in real-time. We propose a shadowing tutor system called WithYou, which can automatically adjust the playback and the difficulty of a speech template when learners fail, so shadowing becomes smooth and tailored. Results from a user study show that WithYou provides greater speech improvements (14%) than the conventional method (2.7%) with a lower cognitive load.

One of the most difficult things in practicing musical instruments is improving timbre. Unlike pitch and rhythm, timbre is a high-dimensional and sensuous concept, and learners cannot evaluate their timbre by themselves. To efficiently improve their timbre control, learners generally need a teacher to provide feedback about timbre. However, hiring teachers is often expensive and sometimes difficult. Our goal is to develop a low-cost learning system that substitutes the teacher. We found that a variational autoencoder (VAE), which is an unsupervised neural network model, provides a 2-dimensional user-friendly mapping of timbre. Our system, SonoSpace, maps the learner's timbre into a 2D latent space extracted from an advanced player's performance. Seeing this 2D latent space, the learner can visually grasp the relative distance between their timbre and that of the advanced player. Although our system was evaluated mainly with an alto saxophone, SonoSpace could also be applied to other instruments, such as trumpets, flutes, and drums.

In tennis training, beginner players can fail to return the ball when the ball moves faster than what they can react to. In this paper, we propose a new training process of mediated-timescale learning (MTL) to manipulate the incoming ball’s motion. The ball first moves in slow motion, allowing more time for players to react and develop skills. The ball then moves in faster motion, challenging players with improved skills. To evaluate MTL, we implemented it in a virtual reality (VR)-oriented tennis training system. We piloted the MTL implementations (N = 12) to study players’ physical enjoyment. We then conducted an efficacy study (N = 8) to evaluate MTL’s training effects on player’s real-world performance. We found that in comparison to real-world training, five participants improved more in hitting the sweet spot after training with MTL.

The availability of digital devices operated by voice is expanding rapidly. However, the applications of voice interfaces are still restricted. For example, speaking in public places becomes an annoyance to the surrounding people, and secret information should not be uttered. Environmental noise may reduce the accuracy of speech recognition. To address these limitations, a system to detect a user's unvoiced utterance is proposed. From internal information observed by an ultrasonic imaging sensor attached to the underside of the jaw, our proposed system recognizes the utterance contents without the user's uttering voice. Our proposed deep neural network model is used to obtain acoustic features from a sequence of ultrasound images. We confirmed that audio signals generated by our system can control the existing smart speakers. We also observed that a user can adjust their oral movement to learn and improve the accuracy of their voice recognition.

Resistive force (e.g., due to object elasticity) and impact (e.g., due to recoil) are common effects in our daily life. However, resistive force continuously changes due to users' movements while impact instantly occurs when an event triggers it. These feedback are still not realistically provided by current VR haptic methods. In this paper, a wearable device, ElasticVR, which consists of an elastic band, servo motors and mechanical brakes, is proposed to provide the continuously-changing resistive force and instantly-occurring impact upon the user's hand to enhance VR realism. By changing two physical properties, length and extension distance, of the elastic band, ElasticVR provides multilevel resistive force with no delay and impact with little delay, respectively, for realistic and versatile VR applications. A force perception study was performed to observe users' force distinguishability of the resistive force and impact, and the prototype was built based on its results. A VR experience study further proves that the resistive force and impact from ElasticVR both outperform those from current approaches in realism. Applications using ElasticVR are also demonstrated.

Development of 3D joints inference technology from 2D RGB video enables us to create 3D annotations for each frame of a video including movements. We here proposed SuppleView which is supple about the user's physically posing while viewing a video and enables the coordinate translation free viewing. By inferring the 3D joints of frames in a video, we create the 3D model as an actor of movements which are same with the actor's one in an original 2D video. The system transitions these two actor depends on the physical rotation of the user's head. The angle of view for observing the actor also changes. Hence rendering contents of this viewer can be ease for user to move one's body to be same with a target one. From the results of the prototype system, we considered our proof of concept had worked properly in the actual scene of movement imitation. We assumed that SuppleView has a potential to ease movement imitations by decreasing physically limitations on viewing motion with traditional displays.

There has been a long history in electrical muscle stimulation (EMS), which has been used for medical and interaction purposes. Human-computer interaction (HCI) researchers are now working on various applications, including virtual reality (VR), notification, and learning. For the electric signals applied to the human body, various types of waveforms have been considered and tested. In typical applications, pulses with short duration are applied, however, many perspectives are required to be considered. In addition to the duration and polarity of the pulse/waves, the wave shapes can also be an essential factor to consider. A problem of conventional EMS toolkits and systems are that they have a limitation to the variety of signals that it can produce. For example, some may be limited to monophonic pulses. Furthermore, they are usually limited to rectangular pulses and a limited range of frequencies, and other waveforms cannot be produced. These kinds of limitations make us challenging to consider variations of EMS signals in HCI research and applications. The purpose of “wavEMS” is to encourage testing of a variety of waveforms for EMS, which can be manipulated through audio output. We believe that this can help improve HCI applications, and to open up new application areas.

With the growing popularity of drones, we start to see more wearable drone concepts. For the user to carry around a drone, it must be lightweight as well as have a small portable form factor. However, these constraints affect the battery capacity and therefore decrease the flight time of the vehicle. With Aerial Tethered Companion, a bigger battery is installed in the user's backpack allowing to extend significantly the flight time. Moreover, without an on-board battery, the quadcopter can carry more payload. Such system can be used in various scenarios for example in sports augmentation where the user would see itself through the drone's camera. Furthermore, Aerial Tethered Companion can be applied in telepresence where an external user would be able to see and navigate around the local user.

One of the possible causes of the phantom limb pain (PLP), which have not been clarified medically, is said that a brain is unable to adapt to the loss of limbs. Even though a brain sends a signal to move limbs, the feedback on the signal is not given because the limbs to which a brain sent the signal are no longer in existence. Recently, some kinds of treatment are introduced, which is making patients feel that their phantom limbs are in existence and they are able to move the phantom limbs actually by their own will, but each of them has some difficulty. In this research, we propose the method of PLP treatment using an optical see-through Head Mounted Display (HMD) that can adapt to the telescoping of phantom limbs and make it easy for patients to communicate with people around them.

Bouldering is an urban form of rock climbing that requires precise and complex movement. Similarly to other sports, the simplest way to learn bouldering skill is to mimic professional's motion. However, ordinary beginner boulders cannot learn to coaches, so that they learn by themselves or tutorial videos. Even if they managed, bouldering has a communication difficulty between a trainee and a trainer, that is, climbers cannot mimic the trainer's movement in parallel. Accordingly, we considered a video feedback system would be useful for beginners and suggested InterPoser: a novel visualization system for intermediate motion between a beginner climber and a more experienced. InterPoser receives two videos of different subjects climbing the sample problem and generates an intermediate movement. In addition, this motion is transferred into realistic images of the climber. The proposed system is expected to support beginner to acquire more detailed observation and understanding of the motion.

Contributions of recent deep-neural-network (DNN) based techniques have been playing a significant role in human-computer interaction (HCI) and user interface (UI) domains. One of the commonly used DNNs is human pose estimation. This kind of technique is widely used for motion capturing of humans, and to generate or modify virtual avatars. However, in order to gain accuracy and to use such systems, large and precise datasets are required for the machine learning (ML) procedure. This can be especially difficult for extreme/wild motions such as acrobatic movements or motions in specific sports, which are difficult to estimate in typically provided training models. In addition, training may take a long duration, and will require a high-grade GPU for sufficient speed. To address these issues, we propose a method to improve the pose estimation accuracy for extreme/wild motions by using pre-trained models, i.e., without performing the training procedure by yourselves. We assume our method to encourage usage of these DNN techniques for users in application areas that are out of the ML field, and to help users without high-end computers to apply them for personal and end use cases.

Providing an experience that includes high-speed objects, such as tennis balls, with a virtual reality (VR) training environment might provide efficient training for trainers but is challenging to achieve. Because of the drawing performance of the display, high-speed objects are perceived as poor visual information more than in reality, such as images in a stroboscope. The faster the object, the more noticeable it becomes, and the harder it is to perceive it correctly. Therefore, if the training is performed at the actual speed, the perception becomes more difficult than real space training due to the low reproduction accuracy. To solve this problem, we propose the computational time-space that controls high-speed objects in VR space, based on the user’s body movement. The method facilitates the perception of fast-moving objects by synchronizing the time of the ball with the movement of the body.

A head-mounted display (HMD), which is common in virtual reality (VR) systems, normally hides the user's face. This feature prohibits to realize a face-to-face communication, in which two or more users share the same virtual space, or show a participant's face on a surrogate-robot's face when the user remotely connects to the robot through an HMD for tele-immersion. Considering that face-to-face communication is one of the fundamental requirements of real-time communications, and is widely realized and used by many nonVR telecommunication systems, an HMD's face hiding feature is considered to be a serious problem and limits the possibility of VR. To address this issue, we propose the notion of "Face-through HMD" and present a face-capturing HMD configuration called "Behind-the-Mask" with infrared (IR) cut filters and side cameras that can be attached to existing HMDs. As an IR cut filter only reflects infrared light and transmits visible light, it is transparent to the user's eye but reflects the user's face with infrared lights. By merging a prescanned 3D face model of the user with the face image obtained from our HMD, the 3D face model of the user with eyes and mouth movement can be reconstructed. We consider that our proposed HMD can be used in many VR applications.

We propose a system, called ExtVision, to augment visual experiences by generating and projecting context-images onto the periphery of the television or computer screen. A peripheral projection of the context-image is one of the most effective techniques to enhance visual experiences. However, the projection is not commonly used at present, because of the difficulty in preparing the context-image. In this paper, we propose a deep neural network-based method to generate context-images for peripheral projection. A user study was performed to investigate the manner in which the proposed system augments traditional visual experiences. In addition, we present applications and future prospects of the developed system.

This paper introduces a telepresence system called OmniGaze, which displays a remote user's presence using a spherical display on an omnidirectional camera. In a telepresence system, gaze has a variety of important functions such as presenting the direction of attention and making simple eye gestures. An omnidirectional camera is an effective tool for gaze communication because it provides a wider range of view for remote users. OmniGaze is a telepresence system which leverages an omnidirectional camera. We designed a modular spherical LED matrix display that covers an omnidirectional camera and projects a spherical animation to visually convey remote user's gaze information. Telecommunication by gaze interaction differs from face-to-face communication in various aspects, so we conducted a user study to investigate practical applications of our system. This paper presents feedback and findings pertaining to the proposed gaze animation technique, advantages and disadvantages of asymmetric communication, and design implications. We also discuss applying our system to communication of humans and automated robots.

This paper introduces Augmented Jump, a backpack multirotor system for jumping ability augmentation. Augmented Jump hovers and supports users' weight by a constant upward power of thrust. Users can jump higher and stay in the air for a longer time than usual with Augmented Jump. We designed and developed our first proof-of-concept prototype that can be controlled as an octocopter and support user's weight by 50kg at maximum. In our experiments, it is found that the system enabled the user to perform jumping in simulated 75% reduced gravity. From user study, the results showed that our system was effective for extending the height and the duration of jumping.

We present a wearable telepresence system, JackIn Neck, that can be worn on the neck and supports joint activities with a local user, a conversation partner who meet the local user, and a remote user of this system. While previous works demonstrated clear advantages of the neck for the location of wearable devices, a telepresence system tailored for such use had not been developed. JackIn Neck realizes this form factor by combining a camera with a fisheye lens, speakers, and microphones. Because our device is easy to put on and take off, as well as being comfortable to wear, we see the potential of our system to be adopted in the wild, such as sightseeing and event participation remotely.

Voice conversion (VC) algorithms modify the speech of a particular speaker to resemble that of another speaker. Many existing virtual reality (VR) and augmented reality (AR) systems make it possible to change the appearance of users, and if VC is added, then users can also change their voice. State-of-the-art VC methods employ recurrent neural networks (RNNs), including long short-term memory (LSTM) networks, for generating converted speech. However, it is difficult for RNNs to perform parallel computations because the computations at each timestep depend on the results of a previous timestep, which prevents them from operating in real-time. In contrast, we propose a novel VC approach based on a dilated convolutional neural network (Dilated CNN), which is a deep neural network model that allows for parallel computation. We adapted the Dilated CNN model to perform convolutions in both the forward and reverse directions to ensure the learning is successful. In addition, to ensure the model can be parallelized during both the training and inference phases, we developed a model architecture that predicts all output values from the value of the input speech, and does not rely on predicted values for the next input. The results demonstrate that the proposed VC approach has a faster conversion rate relative to that of state-of-the-art methods, while improving speech quality a little and maintaining speaker similarity.

Underwater immersive virtual reality (VR) environments can reproduce unique VR experiences such as swimming in the sea with beautiful coral reefs and a cage surrounded by sharks. Underwater VR poses new technical challenges to creating user interactions because water and the surround-screen make existing methods for realizing user interaction irrelevant. In this research, we present a potential water flow measurement technology aimed at accessing human-computer interaction in underwater VR. Flow measurement can be realized by using tracer particles that are scattered in fluids. However, existing tracer particles are not suitable for underwater immersive VR because the particles stop users from viewing the content on the screen. Therefore, we propose transparent tracer particles that become invisible in water and polarization-based technologies that enable cameras to track the movement of particles. This technology enables virtual objects in VR to react with the actual movement of water and haptic feedback by creating water flow in the swimming pool. These additions would enhance the illusion of immersion in underwater VR.

Immersion is an important factor in video experiences. Therefore, various methods and video viewing systems have been proposed so far. Although head-mounted displays (HMDs) are home-friendly and more available among these devices, they can provide an immersive video experience owing to their wide field-of-view (FoV) and separation of users from the outside environment. They are often used for panoramic and stereoscopic VR videos, but the demand for viewing standard plane videos has increased in recent years. However, the theater mode, which restricts the FoV, is basically used for viewing plane videos. Thus, the advantages of HMDs are not fully utilized. Therefore, we explored an effective method for viewing plane videos by an HMD, in combination with view augmentation by LED implants to the HMD. We used deep neural network (DNN) to generate images for peripheral vision and wide FoV customization.

Researchers have attempted to increase the realism of virtual reality (VR) applications in many ways. Combinations of the visual, auditory and haptic feedback have successfully simulated experiences in VR, however, multimedia contents may also stimulate emotions. In this paper, we especially paid attention to negative emotions that may be perceived in such experiences (e.g., fear). We hypothesized that volunteering, visual, mechanical, and electrical feedback may induce negative emotional feedback to users. In-Pulse is a novel system and approach to explore the potential of bringing this emotional feedback to users. We designed a head-mounted display (HMD) combined with mechanical and electrical muscle stimulation (EMS) actuators. A user study was performed to explore the effect of our approaches with combinations with VR contents. The results suggest that mechanical actuators and EMS can improve the experience of virtual experiences.

The human body has unique electrical characteristics. These characteristics have been investigated in various studies in human-computer interaction (HCI) and related research fields. Such studies include applications for using the body as a conductive lead for transmission or electric field sensing and activating human muscles or organs. However, electricity is not completely safe for the human body; therefore, to avoid harming users, careful consideration is essential when developing such devices. The knowledge required for such consideration is spread throughout a large number research fields, and it can be difficult for researchers in the HCI field to comprehend all of them. The purpose of this article is to support researchers in developing systems that apply electricity to the human body and to serve as a basis for further research. This article reviews previous research pertaining to HCI in which users come into contact with electricity. In addition, considerations of how and where this type of research can be expanded, along with guidelines grounded in other fields for designing systems safely and addressing ethical concerns, are presented. An understanding of the field and of the related safety issues will enhance the understanding of limitations and potential and can clarify the design space.

Although doing sports and games together is effective to create good social connections, a large skill gap among players reduces the enjoyment in such activities. In this research, we technologically enhance sports and gaming activities in the real world. The gaming environment is aimed at mitigating the differences in ability without reducing the enjoyment. For the initial approach of the research, we implemented a setup for enhancing table games by using a liquid crystals display. The system shows different images to each player using a polarization-based technique. As the application, we implemented an augmented PONG game. The game has a function to support players depending on their scores in the match. This setup can apply to a projector for a wider gaming field for enhancing game activities requiring body movements such as Table Tennis and Air Hockey.

We present JackIn Airsoft, a system for generating maps for multi-player sport activities and sharing their first-person views (FPVs). In first-person shooting (FPS) games, maps are generated for the visualization of the player's location; these maps are used for strategic play and discussions. FPS games are a virtual experience of shooting or of military activities; the displaying of maps on a screen is a technique designed for game playing. We address the challenge of adapting this effective map visualization technique to real-world strategic sports. In this paper, we introduce our prototype map generator for airsoft sports that is based on the usage of wearable cameras and simultaneous localization and mapping (SLAM). The system enables users to switch between the FPV and the generated map in order to share a player's experience and a strategic overview for team sports. We applied ORB-SLAM2 to multiple recorded FPV videos to discuss the situations and conditions appropriate for our system.

Sharing one's own immersive experience over the Internet is one of the ultimate goals of telepresence technology. In this paper, we present JackIn Head, a visual telepresence system featuring an omnidirectional wearable camera with image motion stabilization. Spherical omnidirectional video footage taken around the head of a local user is stabilized and then broadcast to others, allowing remote users to explore the immersive visual environment independently of the local user's head direction. We describe the system design of JackIn Head and report the evaluation results of real-time image stabilization and alleviation of cybersickness. Then, through an exploratory observation study, we investigate how individuals can remotely interact, communicate with, and assist each other with our system. We report our observation and analysis of inter-personal communication, demonstrating the effectiveness of our system in augmenting remote collaboration.

Our research explores how humans can understand and develop viewing behaviors with mutual paralleled first person view sharing in which a person can see others' first person video perspectives as well as their own perspective in realtime. We developed a paralleled first person view sharing system which consists of multiple video see-through head mounted displays and an embedded eye tracking system. With this system, four persons can see four shared first person videos of each other. We then conducted workshop based research with two activities, drawing pictures and playing a simple chasing game with our view sharing system. Our results show that 1) people can complement each other's memory and decisions and 2) people can develop their viewing behaviors to understand their own physical embodiment and spatial relationship with others in complex situations. Our findings about patterns of viewing behavior and design implications will contribute to building design experience in paralleled view sharing applications.

In mediated communications, the social presence is said to be enhanced not only by eye contact and body gestures but also by remote haptics. To realize remote haptics, we present a mask telepresence system worn by a surrogate that displays the face of a remote user through a livestream. This system allows someone to shake the hand of a remote user via the surrogate. We conducted experiments where the surrogate contacted the participants to determine with whom the participants thought they shook hands: the surrogate or the remote user. We hy- pothesized that the relationship between the participant and surrogate or remote user is affected by the haptic sensations. Therefore, we prepared four conditions where the remote user was an acquaintance or a stranger and the surrogate was an acquaintance or a stranger. The results showed that, when the surrogate and the remote user were acquaintances, the participants felt like they were shaking hands with the remote user.

We present a method of rendering aerial and volumetric graphics using femtosecond lasers. A high-intensity laser excites physical matter to emit light at an arbitrary three-dimensional position. Popular applications can thus be explored, especially because plasma induced by a femtosecond laser is less harmful than that generated by a nanosecond laser. There are two methods of rendering graphics with a femtosecond laser in air: producing holograms using spatial light modulation technology and scanning of a laser beam by a galvano mirror. The holograms and workspace of the system proposed here occupy a volume of up to 1 cm 3 ; however, this size is scalable depending on the optical devices and their setup. This article provides details of the principles, system setup, and experimental evaluation, and discusses the scalability, design space, and applications of this system. We tested two laser sources: an adjustable (30--100fs) laser that projects up to 1,000 pulses/s at an energy of up to 7mJ/pulse and a 269fs laser that projects up to 200,000 pulses/s at an energy of up to 50μJ/pulse. We confirmed that the spatiotemporal resolution of volumetric displays implemented using these laser sources is 4,000 and 200,000 dots/s, respectively. Although we focus on laser-induced plasma in air, the discussion presented here is also applicable to other rendering principles such as fluorescence and microbubbles in solid or liquid materials.

Fine motor skills like finger/hand manipulations are essential for playing musical instruments and these skills require a great amount of time and effort to acquire. Researchers have been introducing haptic feedback systems in order to facilitate the process of learning motor skills but little research has expanded the possibility of applying to the field of musical instruments. Hence, we developed a system called "MoveMe" that provides three-dimensional haptic support for playing a musical instrument. The system guides a user's hands as if someone else was holding their hands to help a beginner play a musical instrument. With the system, an expert can pre-record his/her movements so that a beginner can play it back later as necessary. Alternatively, the system connects an expert and a beginner via two haptic robots and the expert can, in real time, guide and correct the beginner's movement. In addition to those functionalities, we introduce a new proficiency metric provided by force feedback. A master can evaluate how much a beginner has improved using both audio feedback as well as this new force-based metric. Through the experiments that we conducted, we found that our system is effective in terms of playing a song at a correct speed and rhythm.

"Shepherd Pass" is a method of tuning sport abilities that focuses on passing a ball. The method can be used to overcome skill gaps between the players for design novel sport games. Sports have been changed by technological innovations. Professional players are improving their skills using updated tools, clothing, and training methods. These innovations can help those who enjoy leisure sports as a way to promote communication or better health. Augmented sports is terms of designing novel sports with augmented fields, tools, or players using information technology. Our research involve the use of information technology to tune the sports abilities of expert and non-expert players to fill gaps in their skills and enable a freer sports design. We developed a self-actuated ball that flies via a ball-shaped quadcopter. The ball can change its speed and trajectory based on a players skill. In this paper, we explain the design concept and implementation of this system, and discuss the user experience and future research directions.

Sharing an immersive experience enhances situational awareness, enabling effective collaboration between persons in different places. The development of a head-worn camera enables us to capture the first-person view and share it as the personal experience. However, the limited viewing angle of such cameras prevents a remote viewer from obtaining a complete surrounding situation. In this paper, we propose a system called LiveSphere, which presents the entire surrounding visual environment using a head-worn system with multiple cameras. The image-stabilizing algorithm compensates image motion caused by the wearer's head movement. This enables the remote viewer to look around the environment independently from the wearer's head direction. We developed a prototype to examine how sharing the surrounding visual environment improves collaboration between persons in different places.

Remote collaboration to share abilities over the Internet is one of the ultimate goals of telepresence technology. In this paper, we present JackIn Head, a visual telepresence system with an omnidirectional wearable camera with image motion stabilization. Spherical omnidirectional video footage taken around the head of a local user is stabilized and then broadcast to others, allowing remote users to explore the scene independently of the local user's head direction. We describe the system design of JackIn Head and report the evaluation results of the system's motion decoupling. Then, through an exploratory observation study, we investigate how individuals can remotely interact, communicate with, and assist each other with our system. We report our observation and analysis of inter-personal communication, demonstrating the effectiveness of our system in augmenting remote collaboration.

This editorial introduction describes the aims and scope of the ACM Transactions on Interactive Intelligent Systems special issue on Activity Recognition for Interaction. It explains why activity recognition is becoming crucial as part of the cycle of interaction between users and computing systems, and it shows how the five articles selected for this special issue reflect this theme.

While most traditional user interfaces are intended to pursue "convenience" by eliminating user operations and by typically automating tasks, some new categories of HCI, such as health support, may require explicit human participation and effort to achieve long-term benefits. In these areas, interfaces that require interactions that promote users to perform explicit activities, rather than interfaces that solely perform tasks on behalf of users, are becoming increasingly important. This trend can be a further challenge of interaction design, and we refer it as "inconvenient interactions". In this paper, we discuss why carefully designed inconveniences can enrich our lives, and provide preliminary but concrete examples. We also propose our guidelines for the design of these inconvenient interactions.

In most cases, speeches or presentations at an international event are required to be given in a common language (e.g. English). However, for people who are not proficient in that common language, delivering presentations fluently is very difficult. Simultaneous translation seems to be a solution, but besides its high cost, simultaneous translation undermines the nature of the presentation by substituting the real voice of the lecturer as well as his/her emotions. In this paper, we propose "SmartVoice", a presentation support system, which aims to overcome language barriers. By tracking the lip motion of the lecturer, SmartVoice controls the playback of the narration, which is a sound data prepared in advance or created automatically using a voice synthesizer. SmartVoice also controls the intonation of the sound based on the position and shape of the lecturer's mouth. As the lecturer can talk at his/her own pace with the voice automatically following, it appears as if he/she talks in his/her own voice. In our user evaluation, we confirmed that audiences find it difficult to distinguish between the narration generated by SmartVoice and that by a real voice. We also discuss the possibility of applying SmartVoice to fields other than multi-language presentation support, such as Automated Dialogue Replacement and language study.

Referring to documents is common when making things, but there is a difficulty caused by the gap between a written description and the actual context of making. For example, when cooking following a recipe, people may lose their current position in the recipe, misunderstand the required amount of ingredients because of complicated measuring units, or skip steps by mistake. We address these problems by selecting cooking as our domain. Our proposed cooking support system, MimiCook, embodies a recipe in a real kitchen counter and directly navigates a user. The system consists of a computer, a depth camera, a projector, and a scaling device. It displays step-by-step instructions directly onto the utensils and ingredients, and controls the guidance display in accordance with the user's situations. The integrated scaling device also helps users to avoid mistakes with measuring units. Results of our user study shows participants found it easier to cook with the system and even subjects who had never cooked the assigned recipe did not make any mistakes.

We propose a novel graphics system based on the expansion of 3D acoustic-manipulation technology. In conventional research on acoustic levitation, small objects are trapped in the acoustic beams of standing waves. We expand this method by changing the distribution of the acoustic-potential field (APF). Using this technique, we can generate the graphics using levitated small objects. Our approach makes available many expressions, such as the expression by materials and non-digital appearance. These kinds of expressions are used in many applications, and we aim to combine them with digital controllability. In the current system, multiple particles are levitated together at 4.25-mm intervals. The spatial resolution of the position is 0.5 mm. Particles move at up to 72 cm/s. The allowable density of the material can be up to 7 g/cm 3 . For this study, we use three options of APF: 2D grid, high-speed movement, and combination with motion capture. These are used to realize floating screen or mid-air raster graphics, mid-air vector graphics, and interaction with levitated objects. This paper reports the details of the acoustic-potential field generator on the design, control, performance evaluation, and exploration of the application space. To discuss the various noncontact manipulation technologies in a unified manner, we introduce a concept called "computational potential field" (CPF).

Traditionally, the field of Human Computer Interaction (HCI) was primarily concerned with designing and investigating interfaces between humans and machines. However, with recent technological advances the concept of “enhancing”, “augmenting” or even “redesigning” humans themselves is becoming not only interesting and intriguing but also very feasible and serious topic of scientific research and development. “Augmented Human” is term that I use today to refer to this overall research direction. Although the term “augmentation” has long been used in HCI and AR communities since Douglas Engelbert's landmark research on augmenting intelligence, I think the possibility of human augmentation is not limited to intellectual abilities and can be expound to physical abilities. I believe Augmented Human introduces a fundamental paradigm shift in HCI: from human-computer-interaction to human-computer-integration. In this talk, I will discuss rich possibilities and distinct challenges in enhancing human abilities with technology. I will introduce recent projects conducted by our research group including design and applications if wearable eye sensing for augmenting our perception and memory abilities, design of flying cameras as our external eyes, a home appliance that can increase your happiness, an organic physical wall/window that dynamically mediates the environment, and a human hand control system based on functional electrical stimulation.

We present Mirage, a non body contact technique to infer the amount and type of body motion, gesture, and activity. This approach involves passive measurement of static electric field of the environment flowing through sense electrode. This sensing method leverages electric field distortion by the presence of an intruder (e.g. human body). Mirage sensor has simple analog circuitry, supports ultra-low power operation, and requires no instrumentation to the user. We illustrate the utility of Mirage with a demonstration of real-time interactive applications including activity monitoring and games which allow the users to interact with a computer using body motion.

Mirage proposes an effective non body contact technique to infer the amount and type of body motion, gesture, and activity. This approach involves passive measurement of static electric field of the environment flowing through sense electrode. This sensing method leverages electric field distortion by the presence of an intruder (e.g. human body). Mirage sensor has simple analog circuitry and supports ultra-low power operation. It requires no instrumentation to the user, and can be configured as environmental, mobile, and peripheral-attached sensor. We report on a series of experiments with 10 participants showing robust activity and gesture recognition, as well as promising results for robust location classification and multiple user differentiation. To further illustrate the utility of our approach, we demonstrate real-time interactive applications including activity monitoring, and two games which allow the users to interact with a computer using body motion and gestures.

This paper introduces a new mechanism to induce a virtual force based on human illusory sensations. An asymmetric signal is applied to a tactile actuator consisting of an electromagnetic coil, a metal weight, and a spring, such that the user feels that the device is being pulled (or pushed) in a particular direction, although it is not supported by any mechanical connection to other objects or the ground. The proposed tactile device is smaller (35.0 mm x 5.0 mm x 7.5 mm) and lighter (5.2 g) than any previous force-feedback devices, which have to be connected to the ground with mechanical links. This small form factor allows the device to be implemented in several novel interactive applications, such as a pedestrian navigation system that includes a finger-mounted tactile device or an (untethered) input device that features virtual force. Our experimental results indicate that this illusory sensation actually exists and the proposed device can switch the virtual force direction within a short period. We combined this new technology with visible light transmission via a digital micromirror device (DMD) projector and developed a position guiding input device with force perception.

We propose a telepresence system with a real human face-shaped screen. This system tracks the remote user's face and extracts the head motion and the face image. The face-shaped screen moves along three degree-of-freedom (DOF) by reflecting the user's head gestures. As the face-shaped screen is molded based on the 3D-shape scan data of the user, the projected image is accurate even when it is seen from different angles. We expect this system can accurately convey the user's nonverbal communication, in particular the user's gaze direction in 3D space that is not correctly transmitted by using a 2D screen (which is known as "the Mona Lisa effect"). To evaluate how this system can contribute to the communication, we conducted three experiments. The first one examines the blind angle of a face-shaped screen and a flat screen, and compares the ease with which users can distinguish facial expressions. The second one evaluates how the direction in which the remote user's face points can be correctly transmitted. The third experiment evaluates how the gaze direction can be correctly transmitted. We found that the recognizable angles of the face-shaped screen were larger, and that the recognition of the head directions was better than on a flat 2D screen. More importantly, we found that the face-shaped screen accurately conveyed the gaze direction, resolving the problem of the Mona Lisa effect.

In this paper we present Squama, a programmable physical window or wall that can independently control the visibility of its elemental small square tiles. This is an example of programmable physical architecture, our vision for future architectures where the physical features of architectural elements and facades can be dynamically changed and reprogrammed according to people's needs. When Squama is used as a wall, it dynamically controls the transparency through its surface, and simultaneously satisfies the needs for openness and privacy. It can also control the amount of sunlight and create shadows, called programmable shadows, in order to afford indoor comfort without completely blocking the outer view. In this paper, we discuss how in future, architectural space can become dynamically changeable and introduce the Squama system as an initial instance for exemplifying this concept.

In this video we present Squama, a programmable physical window or wall that can independently control the visibility of its elemental small square tiles. This is an example of programmable physical architecture, our vision for future architectures where the physical features of architectural elements and facades can be dynamically changed and reprogrammed according to people's needs. When Squama is used as a wall, it dynamically controls the transparency through its surface, and simultaneously satisfies the needs for openness and privacy. It can also control the amount of sunlight and create shadows, called programmable shadows, in order to afford indoor comfort without completely blocking the outer view. In this video, we show how in future, architectural space can become dynamically changeable and introduce the Squama system as an initial instance for exemplifying this concept.

If a device can control human hands, the device can be useful for HCI and tangible application's output. To aid the controlling of finger movement, we present PossessedHand, a device with a forearm belt that can inform when and which fingers should be moved. PossessedHand controls the user's fingers by applying electrical stimulus to the muscles around the forearm. Each muscle is stimulated via 28 electrode pads. Muscles at different depths in the forearm can be selected for simulation by varying the stimulation level. PossessedHand can automatically calibrate the system for individuals. The automatic calibration system estimates relations between each electrode pad, stimulation level and muscle movement. Experiments show that PossessedHand can control the motion of 16 joints in the hand. Further, we also discuss an application based on this device to aid in playing a musical instrument.

William James, the noted psychologist and philosopher, believed that smiling has a positive effect on our mind. James' view, which was confirmed by several psychological studies, was that we become happier when we laugh. In this paper, we propose a new digital appliance that encourages the act of smiling in our daily lives. This system is designed for people who may not always realize when they are in low spirits and/or have difficulty with smiling. In addition, we believe that this system will foster casual conversation and prompt communications with other people. Our appliance, called the HappinessCounter, combines visual smile recognition, user feedback, and network communication. We conducted two trials of the HappinessCounter system, the first with a single occupant and the second with a couple living together. The system had positive effects on user's mood and prompted communication among family members, thereby increasing their positive mood as well.

Existing wearable devices like an eyeglass-type head-mounted display(HMD) are very bulky and cannot be worn everyday. On the other hand, the earphone is popular wearable device, bacause it is small. However, the earphone cannot be used as an interaction device because of the lack of input and visual feedback components. In this paper, we propose BrainyHand - an enhanced earphone device for use in interaction systems. BrainyHand consists of a color camera and a laser projector. The camera recognizes the user's hand gestures for the input. For the visual feedback, the images projected on the user's hand or other nearby objects or surfaces using the projector. Since laser microprojector is becoming small, we expect this device configuration would eventwally a become as small as today earphones. We introduce several interaction methods based on hand gesture recognitions and object detections.

Recent commoditization of mobile digital devices and net-working brought us to use them as a very large-scale sensing platform. We call this possibility "Sensonmoy", which is an integration of collective intelligence (also known as "folk-sonomy") and pervasive sensing. As many users own mobile devices with sensing facilities, a collection of sensing data from these devices becomes quite important, and integration of them can be used in a very different manner. Such feature could be a new way to create intelligent systems and inter-faces. In this talk, I am going to discuss a possibility of con-necting a large number of simple devices to produce intelligent interactions. As a realistic example of them, I will introduce a city-scale indoor and outdoor positioning system that we have developed, and how its database can be evolved by using the idea of Sensonomy. I would also like to discuss computer-augmented memory and lifelong computing based on our platform.

A pressure sensing based single-handed interaction model is presented in this paper. Unlike traditional desktop GUI model, mobile UI model has not been established yet. For example, Apple iPhone proposed "Pinch" operation, which use two fingers to zoom-in and zoom-out objects. However, in a today's hand-held situation, manipulation methods using two fingers are not always good solution because they require two hands in order to hold the device itself in most cases. We propose a single-handed UI scheme "GraspZoom": multi-state input model using pressure sensing. Force Sensitive Resistor (FSR) attached on backside of a mobile phone was employed in order to evaluate effectiveness of pressure based control model. We also describe example applications which enable intuitive and continuous zooming and scrolling. By using tiny thumb gesture input along with this pressure sensing method, bi-directional operations (e.g., zoom-in and -out) are also achieved.

A virtually connected medium called tranStick is described that functions both as a "virtual wire" and as a "memory card" containing a shared space. A user can connect two networked devices by simply placing one of a pair of tranSticks with the same identifier into each device. The tranSticks provide feedback indicating that the devices are connected; the connection to be closed or changed in the same way it would be if the devices were connected by a physical cable. A user can also access to a shared space on a network as if the space were in the tranStick. Since tranSticks contain long secret keys, the process of finding another tranStick with the same identifier can be encrypted. The tranStick approach differs from other approaches in that it provides feedback from the connection as well as serving as a medium for establishing a connection, and it enables disconnection and switchover to be done intuitively because the operations are reversible.

As small digital cameras become more popular, opportunities to take photos are rapidly increasing. Photo sharing is a great way to maintain and revitalize relationships between families and friends, and is a major motivator for content sharing. While photo sharing has been well studied, little work exists on sharing multiple photo sets contained in spontaneously connected handheld devices.This paper provides an algorithm to extract photos, based on common memories collected in an ad hoc group. It automatically searches for and presents photos that could become the starting point of a conversation. We found that our mechanism has more uses than simply organizing photos in chronological order.This paper describes our prototype system realized using the above algorithm. We also implemented a synchronized shutters mechanism, that provides a new photo sharing experience. Through subjective tests, we found that our method promotes conversation, even though the users did not know each other beforehand.

Flat3D is a shared 3D virtual world system that can be grown by user's creative activities and communication over the network. In conventional shared virtual 3D world systems, users sharing the virtual 3D world communicated with each other only by text-based chatting through avatars, so the communication capabilities were quite limited. In contrast, the Flat3D system allows users to create objects freely in a wade 3D world by simple manipulations. These creative activities become communication trigger and scene elements directly, so the virtual 3D world would be grown by user's creation and communication. We describe the FlatSD system and its capabilities.

An ID recognition system is described that uses optical beacons and a high-speed image sensor. The ID sensor captures a scene like an ordinary camera and recognizes the ID of a beacon emitted over a long distance. The ID recognition system has three features. The system is robust to changes in the optical environment, e.g. complete darkness, spotlights, and sunlight. It can recognize up to 255 multiple optical beacons simultaneously. Furthermore, it can recognize beacons even over a long distance, e.g. 40 m indoors and 20 m outdoors. Implementation and evaluation of this ID recognition system showed that a mobile augmented reality system can be achieved by combining this ID recognition system with a PDA and a wireless network.

Although graphical user interfaces started as imitations of the physical world, many interaction techniques have since been invented that are not available in the real world. This paper focuses on one of these "previewing", and how a sensory enhanced input device called "PreSense Keypad" can provide a preview for users before they actually execute the commands. Preview important in the real world because it is often not possible to undo an action. This previewable feature helps users to see what will occur next. It is also helpful when the command assignment of the keypad dynamically changes, such as for universal commanders. We present several interaction techniques based on this input device, including menu and map browsing systems and a text input system. We also discuss finger gesture recognition for the PreSense Keypad.

This paper introduces a new sensor architecture for making interactive surfaces that are sensitive to human hand and finger gestures. This sensor recognizes multiple hand positions and shapes and calculates the distance between the hand and the surface by using capacitive sensing and a mesh-shaped antenna. In contrast to camera-based gesture recognition systems, all sensing elements can be integrated within the surface, and this method does not suffer from lighting and occlusion problems. This paper describes the sensor architecture, as well as two working prototype systems: a table-size system and a tablet-size system. It also describes several interaction techniques that would be difficult to perform without using this architecture

This paper investigates the sense of touch as a channel for communicating with miniature handheld devices. We embedded a PDA with a TouchEngineTM --- a thin, miniature lower-power tactile actuator that we have designed specifically to use in mobile interfaces (Figure 1). Unlike previous tactile actuators, the TouchEngine is a universal tactile display that can produce a wide variety of tactile feelings from simple clicks to complex vibrotactile patterns. Using the TouchEngine, we began exploring the design space of interactive tactile feedback for handheld computers. Here, we investigated only a subset of this space: using touch as the ambient, background channel of interaction. We proposed a general approach to design such tactile interfaces and described several implemented prototypes. Finally, our user studies demonstrated 22% faster task completion when we enhanced handheld tilting interfaces with tactile feedback.

The DataTiles system integrates the benefits of two major interaction paradigms: graphical and physical user interfaces. Tagged transparent tiles are used as modular construction units. These tiles are augmented by dynamic graphical information when they are placed on a sensor-enhanced flat panel display. They can be used independently or can be combined into more complex configurations, similar to the way language can express complex concepts through a sequence of simple words. In this paper, we discuss our design principles for mixing physical and graphical interface techniques, and describe the system architecture and example applications of the DataTiles system.

The ToolStone is a cordless, multiple degree-of-freedom (MDOF) input device that senses physical manipulation of itself, such as rotating, flipping, or tilting.As an input device for the non-dominant hand when a bimanual interface is used, the ToolStone provides several interaction techniques including a toolpalette selector, and MDOF interactors such as zooming, 3D rotation, and virtual camera control.In this paper, we discuss the design principles of input devices that effectively use a human's physical manipulation skills, and describe the system architecture and applications of the Tool-Stone input device.

This paper describes our design and implementation of a computer augmented environment that allows users to smoothly interchange digital information among their portable computers, table and wall displays, and other physical objects. Supported by a camera-based object recognition system, users can easily integrate their portable computers with the pre-installed ones in the environment. Users can use displays projected on tables and walls as a spatially continuous extension of their portable computers. Using an interaction technique called hyperdragging, users can transfer information from one computer to another, by only knowing the physical relationship between them. We also provide a mechanism for attaching digital data to physical objects, such as a videotape or a document folder, to link physical and digital spaces.

This paper describes the concept of Time-Machine Computing (TMC), a time-centric approach to organizing information on computers. A system based on Time-Machine Computing allows a user to visit the past and the future states of computers. When a user needs to refer to a document that he/she was working on at some other time, he/she can travel in the time dimension and the system restores the computer state at that time. Since the user's activities on the system are automatically archived, the user's daily workspace is seamlessly integrated into the information archive. The combination of spatial information management of the desktop metaphor and time traveling allows a user to organize and archive information without being bothered by folder hierarchies or the file classification problems that are common in today's desktop environments. TMC also provides a mechanism for linking multiple applications and external information sources by exchanging time information. This paper describes the key features of TMC, a time-machine desktop environment called “TimeScape,” and several time-oriented application integration examples.

In this paper, we propose a multiple-device approach for supporting informal meetings using a digital whiteboard.Traditional digital whiteboard systems often suffer from a limited capability to enter text and the handling of existing data.The large display surface of the whiteboard also makes traditional GUI design ineffective.Our proposed approach provides a hand-held computer for each participant which serves as a tool palette and data entry palette for the whiteboard.Just as an oil painter effectively uses a palette in his/her hand, this hand-held device offers an easy way to create a new text/stroke object, to select existing data from a network, to select pen attributes, and to control thewhiteboard application.This paper also reports our experience with the digital whiteboard systems using a proposed multi-device architecture.

Dragging is one of the most useful and popular techniques in direct manipulation graphical user interfaces.However, dragging has inherent restrictions caused by pixel resolution of a display.Although in some situations the restriction could be negligible, certain kinds of applications, e.g., real world applications where the range of adjustable parameters vastly exceed the screen resolution, require sub-pixel-pitch dragging.We propose a sub-pixel-pitch dragging tool, popup vernier, plus a methodology to transfer smoothly into 'vernier mode' during dragging.A popup vernier consists of locally zoomed grids and vernier scales displayed around them.Verniers provide intuitive manipulation and feedback of fine grain dragging, in that pixel-pitch movements of the grids represent sub-pixel-pitch movements of a dragged object, and the vernier scales show the object's position at a sub-pixel accuracy.The effectiveness of our technique is verified with a proposed evaluation measure that captures the smoothness of transition from standard mode to vernier mode, based on the Fitts' law.

No abstract available.

Article Pick-and-drop: a direct manipulation technique for multiple computer environments Share on Author: Jun Rekimoto Sony Computer Science Laboratory Inc., 3-14-13 Higashigotanda, Shinagawa-ku, Tokyo 141 Japan Sony Computer Science Laboratory Inc., 3-14-13 Higashigotanda, Shinagawa-ku, Tokyo 141 JapanView Profile Authors Info & Claims UIST '97: Proceedings of the 10th annual ACM symposium on User interface software and technologyOctober 1997 Pages 31–39https://doi.org/10.1145/263407.263505Online:01 October 1997Publication History 380citation3,347DownloadsMetricsTotal Citations380Total Downloads3,347Last 12 Months128Last 6 weeks24 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

No abstract available.

This TechNote introduces a novel interaction technique for small screen devices such as palmtop computers or hand-held electric devices, including pagers and cellular phones. Our proposed method uses the tilt of the device itself as input. Using both tilt and buttons, it is possible to build several interaction techniques ranging from menus and scroll bars, to more complicated examples such as a map browsing system and a 3D object viewer. During operation, only one hand is required to both hold and control the device. This feature is especially useful for field workers.

Article Free Access Share on The world through the computer: computer augmented interaction with real world environments Authors: Jun Rekimoto Sony Computer Science Laboratory Inc., Takanawa Muse Building, 3-14-13, Higashi-gotanda, Shinagawa-ku, Tokyo 141 Japan Sony Computer Science Laboratory Inc., Takanawa Muse Building, 3-14-13, Higashi-gotanda, Shinagawa-ku, Tokyo 141 JapanView Profile , Katashi Nagao Sony Computer Science Laboratory Inc., Takanawa Muse Building, 3-14-13, Higashi-gotanda, Shinagawa-ku, Tokyo 141 Japan Sony Computer Science Laboratory Inc., Takanawa Muse Building, 3-14-13, Higashi-gotanda, Shinagawa-ku, Tokyo 141 JapanView Profile Authors Info & Claims UIST '95: Proceedings of the 8th annual ACM symposium on User interface and software technologyDecember 1995 Pages 29–36https://doi.org/10.1145/215585.215639Published:01 December 1995Publication History 258citation3,948DownloadsMetricsTotal Citations258Total Downloads3,948Last 12 Months334Last 6 weeks44 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Proposed here are object-oriented models for handling multimedia data: an object composition model and a playback model.The introduction of "temporal glue", an extension of T E X's glue, and a mechanism for constructing composite multimedia data hierarchically in the models, makes it quite easy to edit and reuse composite multimedia data.This paper explains the models and experimental results obtained for a set of C++ class libraries based on them.