This paper illustrates design work carried out to develop an interactive theater performance. HCI has started to address the challenges of designing interactive performances, as both audience and performers’ experiences are considered and a variety of professional expertise involved. Nevertheless, research has overlooked how such design unfolds in practice, and what role artists play in exploring both the creative opportunities and the challenges associated with interweaving digital technologies. A two-day workshop was conducted to tailor the use of the ChameleonMask, a telepresence technology, within a performance. The analysis highlights the artists’ work to make the mask work while framing, exploring and conceptualizing its use. The discussion outlines the artists’ skills and design expertise, and how they redefine the role of HCI in performance-led research.

Conversational agents are widely used in many situations, especially for speech tutoring. However, their contents and functions are often pre-defined and not customizable for people without technical backgrounds, thus significantly limiting their flexibility and usability. Besides, conventional agents often cannot provide feedback in the middle of training sessions because they lack technical approaches to evaluate users' speech dynamically. We propose JustSpeak: automated and interactive speech tutoring agents with various configurable feedback mechanisms, using any speech recordings with its transcription text as the template for speech training. In JustSpeak, we developed an automated procedure to generate customized tutoring agents from user-inputted templates. Moreover, we created a set of methods to dynamically synchronize speech recognizers' behavior with the agent's tutoring progress, making it possible to detect various speech mistakes dynamically such as being stuck, mispronunciation, and rhythm deviations. Furthermore, we identified the design primitives in JustSpeak to create different novel feedback mechanisms, such as adaptive playback, follow-on training, and passive adaptation. They can be combined to create customized tutoring agents, which we demonstrate with an example for language learning. We believe JustSpeak can create more personalized speech learning opportunities by enabling tutoring agents that are customizable, always available, and easy-to-use.

Learning to speak in foreign languages is hard. Speech shadowing has been rising as a proven way to practice speaking, which asks a learner to listen and repeat a native speech template as simultaneously as possible. However, shadowing can be hard to do because learners can frequently fail to follow the speech and unintentionally interrupt a practice session. Worse, as a technical way to evaluate shadowing performance in real-time has not been established, no automated solutions are available to help. In this paper, we propose a technical framework with context-dependent speech recognition to evaluate shadowing in real-time. We propose a shadowing tutor system called WithYou, which can automatically adjust the playback and the difficulty of a speech template when learners fail, so shadowing becomes smooth and tailored. Results from a user study show that WithYou provides greater speech improvements (14%) than the conventional method (2.7%) with a lower cognitive load.

One of the possible causes of the phantom limb pain (PLP), which have not been clarified medically, is said that a brain is unable to adapt to the loss of limbs. Even though a brain sends a signal to move limbs, the feedback on the signal is not given because the limbs to which a brain sent the signal are no longer in existence. Recently, some kinds of treatment are introduced, which is making patients feel that their phantom limbs are in existence and they are able to move the phantom limbs actually by their own will, but each of them has some difficulty. In this research, we propose the method of PLP treatment using an optical see-through Head Mounted Display (HMD) that can adapt to the telescoping of phantom limbs and make it easy for patients to communicate with people around them.

Bouldering is an urban form of rock climbing that requires precise and complex movement. Similarly to other sports, the simplest way to learn bouldering skill is to mimic professional's motion. However, ordinary beginner boulders cannot learn to coaches, so that they learn by themselves or tutorial videos. Even if they managed, bouldering has a communication difficulty between a trainee and a trainer, that is, climbers cannot mimic the trainer's movement in parallel. Accordingly, we considered a video feedback system would be useful for beginners and suggested InterPoser: a novel visualization system for intermediate motion between a beginner climber and a more experienced. InterPoser receives two videos of different subjects climbing the sample problem and generates an intermediate movement. In addition, this motion is transferred into realistic images of the climber. The proposed system is expected to support beginner to acquire more detailed observation and understanding of the motion.

This paper introduces a telepresence system called OmniGaze, which displays a remote user's presence using a spherical display on an omnidirectional camera. In a telepresence system, gaze has a variety of important functions such as presenting the direction of attention and making simple eye gestures. An omnidirectional camera is an effective tool for gaze communication because it provides a wider range of view for remote users. OmniGaze is a telepresence system which leverages an omnidirectional camera. We designed a modular spherical LED matrix display that covers an omnidirectional camera and projects a spherical animation to visually convey remote user's gaze information. Telecommunication by gaze interaction differs from face-to-face communication in various aspects, so we conducted a user study to investigate practical applications of our system. This paper presents feedback and findings pertaining to the proposed gaze animation technique, advantages and disadvantages of asymmetric communication, and design implications. We also discuss applying our system to communication of humans and automated robots.

This paper introduces Augmented Jump, a backpack multirotor system for jumping ability augmentation. Augmented Jump hovers and supports users' weight by a constant upward power of thrust. Users can jump higher and stay in the air for a longer time than usual with Augmented Jump. We designed and developed our first proof-of-concept prototype that can be controlled as an octocopter and support user's weight by 50kg at maximum. In our experiments, it is found that the system enabled the user to perform jumping in simulated 75% reduced gravity. From user study, the results showed that our system was effective for extending the height and the duration of jumping.

Underwater immersive virtual reality (VR) environments can reproduce unique VR experiences such as swimming in the sea with beautiful coral reefs and a cage surrounded by sharks. Underwater VR poses new technical challenges to creating user interactions because water and the surround-screen make existing methods for realizing user interaction irrelevant. In this research, we present a potential water flow measurement technology aimed at accessing human-computer interaction in underwater VR. Flow measurement can be realized by using tracer particles that are scattered in fluids. However, existing tracer particles are not suitable for underwater immersive VR because the particles stop users from viewing the content on the screen. Therefore, we propose transparent tracer particles that become invisible in water and polarization-based technologies that enable cameras to track the movement of particles. This technology enables virtual objects in VR to react with the actual movement of water and haptic feedback by creating water flow in the swimming pool. These additions would enhance the illusion of immersion in underwater VR.

Researchers have attempted to increase the realism of virtual reality (VR) applications in many ways. Combinations of the visual, auditory and haptic feedback have successfully simulated experiences in VR, however, multimedia contents may also stimulate emotions. In this paper, we especially paid attention to negative emotions that may be perceived in such experiences (e.g., fear). We hypothesized that volunteering, visual, mechanical, and electrical feedback may induce negative emotional feedback to users. In-Pulse is a novel system and approach to explore the potential of bringing this emotional feedback to users. We designed a head-mounted display (HMD) combined with mechanical and electrical muscle stimulation (EMS) actuators. A user study was performed to explore the effect of our approaches with combinations with VR contents. The results suggest that mechanical actuators and EMS can improve the experience of virtual experiences.

The human body has unique electrical characteristics. These characteristics have been investigated in various studies in human-computer interaction (HCI) and related research fields. Such studies include applications for using the body as a conductive lead for transmission or electric field sensing and activating human muscles or organs. However, electricity is not completely safe for the human body; therefore, to avoid harming users, careful consideration is essential when developing such devices. The knowledge required for such consideration is spread throughout a large number research fields, and it can be difficult for researchers in the HCI field to comprehend all of them. The purpose of this article is to support researchers in developing systems that apply electricity to the human body and to serve as a basis for further research. This article reviews previous research pertaining to HCI in which users come into contact with electricity. In addition, considerations of how and where this type of research can be expanded, along with guidelines grounded in other fields for designing systems safely and addressing ethical concerns, are presented. An understanding of the field and of the related safety issues will enhance the understanding of limitations and potential and can clarify the design space.

We present JackIn Airsoft, a system for generating maps for multi-player sport activities and sharing their first-person views (FPVs). In first-person shooting (FPS) games, maps are generated for the visualization of the player's location; these maps are used for strategic play and discussions. FPS games are a virtual experience of shooting or of military activities; the displaying of maps on a screen is a technique designed for game playing. We address the challenge of adapting this effective map visualization technique to real-world strategic sports. In this paper, we introduce our prototype map generator for airsoft sports that is based on the usage of wearable cameras and simultaneous localization and mapping (SLAM). The system enables users to switch between the FPV and the generated map in order to share a player's experience and a strategic overview for team sports. We applied ORB-SLAM2 to multiple recorded FPV videos to discuss the situations and conditions appropriate for our system.

If a device can control human hands, the device can be useful for HCI and tangible application's output. To aid the controlling of finger movement, we present PossessedHand, a device with a forearm belt that can inform when and which fingers should be moved. PossessedHand controls the user's fingers by applying electrical stimulus to the muscles around the forearm. Each muscle is stimulated via 28 electrode pads. Muscles at different depths in the forearm can be selected for simulation by varying the stimulation level. PossessedHand can automatically calibrate the system for individuals. The automatic calibration system estimates relations between each electrode pad, stimulation level and muscle movement. Experiments show that PossessedHand can control the motion of 16 joints in the hand. Further, we also discuss an application based on this device to aid in playing a musical instrument.

In wireless sensor networks (WSN), collaboration is a way to improve the quality of data communication between sensor nodes with restricted resources in terms of memory, processing and energy storage. For receive collaboration, various array processing schemes such as receive beamforming and collaborative channel equalization (CCE) can be used for aggregating data received by each node in the network. The key challenge is the limitation on the number of nodes which can collaborate because of the increased computational load and memory demand when the multiple signals are aggregated. This problem arises when sensor nodes in a CDMA based WSN collaborate, although the low power property of CDMA technique makes it suitable for WSN applications. Here receive collaboration is investigated in CDMA networks using CCE as the collaboration algorithm. We present two novel distributed signal aggregation algorithms: partial and hierarchical aggregation, which distribute computational load and memory demands on collaborative nodes. The positive impacts of receive collaboration on the signal quality and reliability are confirmed experimentally in a WSN scenario using software radios. Then the requirements of collaborative reception using CCE combined with the novel aggregation methods in terms of computational and memory load, as well as energy consumption are evaluated. The results indicate that the distributed signal aggregation algorithms, especially hierarchical aggregation, have computational and memory requirements less than that of centralized CCE, providing greater flexibility and scalability which enables collaboration in WSNs on a larger scale than previously possible.

Existing wearable devices like an eyeglass-type head-mounted display(HMD) are very bulky and cannot be worn everyday. On the other hand, the earphone is popular wearable device, bacause it is small. However, the earphone cannot be used as an interaction device because of the lack of input and visual feedback components. In this paper, we propose BrainyHand - an enhanced earphone device for use in interaction systems. BrainyHand consists of a color camera and a laser projector. The camera recognizes the user's hand gestures for the input. For the visual feedback, the images projected on the user's hand or other nearby objects or surfaces using the projector. Since laser microprojector is becoming small, we expect this device configuration would eventwally a become as small as today earphones. We introduce several interaction methods based on hand gesture recognitions and object detections.

A pressure sensing based single-handed interaction model is presented in this paper. Unlike traditional desktop GUI model, mobile UI model has not been established yet. For example, Apple iPhone proposed "Pinch" operation, which use two fingers to zoom-in and zoom-out objects. However, in a today's hand-held situation, manipulation methods using two fingers are not always good solution because they require two hands in order to hold the device itself in most cases. We propose a single-handed UI scheme "GraspZoom": multi-state input model using pressure sensing. Force Sensitive Resistor (FSR) attached on backside of a mobile phone was employed in order to evaluate effectiveness of pressure based control model. We also describe example applications which enable intuitive and continuous zooming and scrolling. By using tiny thumb gesture input along with this pressure sensing method, bi-directional operations (e.g., zoom-in and -out) are also achieved.

This paper describes an object tracking scheme employs sensor fusion approach which is composed of visual information and location information estimated from Wi-Fi signals. Location information is calculated by a set of received signal strength values of beacon packets from Wi-Fi access points (APs) around the targets. Different from the conventional approaches which use another kind of sensors, our approach can cover wider areas both indoor and outdoor with lower cost because of characteristics of Wi-Fi signals. Particle filter is applied to combine these two different kinds of sensory input to track the target continuously. Wi-Fi observation model is involved in a conventional visual particle filtering scheme in order to evaluate importance weights of each particle. By using multiple modality, robust tracking performance is achieved even if reliability of one sensory input declines. In this paper, we present experimental results applied to outdoor surveillance camera environment.

In this paper, we have developed a robust object-based watermarking algorithm using the scale-invariant feature transform (SIFT) features in conjunction with a new data embedding method based on discrete cosine transform (DCT). The message is embedded in DCT spaces of randomly generated blocks in the selected object region. To recognize the object region after being distorted, its SIFT features are registered in advance. In the detection scheme, we firstly detect the object region by using feature matching. The transformation parameters are then calculated, and the message can be detected. Experimental results demonstrated that our proposed algorithm is very robust to geometrical distortions such as JPEG compression, scaling, rotation, shearing, aspect ratio change, image filtering, and so on.

Object tracking with multiple cameras is a fundamental problem in wide-area surveillance application, but it has difficulties to achieve accurate and stable performance because of disjoint shot areas or initial object identification problems. We propose a novel object tracking method which jointly uses estimated location information of the target derived from a set of Wi-Fi signal strength values with video images from cameras. Apart from sensor fusion techniques proposed in the past which use another kind of sensors (eg., global positioning system (GPS), pressure sensors on the floors to detect foot steps, laser-range scanners, etc), our approach can cover wide-areas both indoor and outdoor in low cost because of propagation characteristics of Wi-Fi signals. Particle filtering is applied to achieve tracking the target from video images, with this Wi-Fi location estimation. This paper describes system architectures and the experimental results of the pedestrian tracking technique with Wi-Fi location estimation.