In mixed reality environments, virtual objects can obscure real-world obstacles, creating a risk of collision when users walk through them. Although users may choose to detour around virtual objects, this behavior also carries risks, such as colliding with obstacles or pedestrians along the detour route. To reduce collision risks, it is essential to understand the factors that determine whether users walk through or detour, as well as the walking paths associated with each behavior. In this research, we investigated users’ walking behavior toward both a static virtual obstacle and a virtual obstacle that disappeared as the user approached. Our findings suggest that individual characteristics and the width of the virtual obstacle influence the decision to walk through or detour. Furthermore, while most users initially chose paths that detoured around the virtual obstacle, once the obstacle began to disappear, they switched their walking paths toward the space where it had been.

Smart rings can serve as a wearable platform for off-device control of nearby devices. However, their thin bands lead to a small touch surface on the ring, limiting touch input expressivity and accuracy. To address this, we investigate the effectiveness of ring shapes that use beveled surfaces in addition to the flat outer surface as input surfaces. The distinct angles of the three surfaces limit touches to the intended surfaces, improving accuracy. A first study showed that with band widths of 6 mm or less, flat rings achieved low accuracy (63.8%) in distinguishing between left and right edge touches, whereas beveled and rounded rings achieved high accuracy (90.0%, 93.3%). In a second study of nine touch gestures on 6-mm rings, beveled rings outperformed the flat rings, achieving 92.9% (sighted) and 91.8% (eyes-free) with FPR \(\le 1\%\). Through these investigations, we identify beveled surfaces as a promising method toward expressive, precise touch input.

Expanding an object’s region in motor space (selection area) using Voronoi tessellation can facilitate the selection of small targets in gaze-based interfaces. While visual feedforward, such as visualizing the selection area, has been shown to improve selection performance in hand-based interfaces, its influence on gaze-based interfaces remains unclear. To address this, we conducted two user studies to examine how visual feedforward affects fixation behavior and selection performance when the centers of a target and its selection area differ. The results reveal that visualizing either the selection area or its center effectively guides users’ gaze toward the center of the selection area, thereby improving selection accuracy. Furthermore, displaying a gaze cursor reduces errors by enabling users to adjust their fixation, even though it does not directly guide gaze toward the center of the selection area.

Figure 1: FlexEar-Tips overview and application examples.A: FlexEar-Tips, an ear tip system for hearables that allows shape adjustment through pressure control, B: user wearing the device, C: haptics features during gameplay, D: music listening experience enhancement and notification features, E: an image of ear tip status transitions.

Earphone-type wearable devices, also known as “hearables,” will have many functions in the future. Some of those functions will require authentication of the wearer for access to the user's privacy information or settlement of payments. In this study, we propose a new personal authentication system for hearables called EarLock. EarLock authenticates the wearer by acquiring and analyzing ear canal and auricle shape information using sound leakage from the device. The system can be implemented using a speaker and external microphone that are highly compatible with hearables. We implemented three prototype devices and investigated EarLock's authentication performance under various practical scenarios, including walking conditions, noisy environments, and situations with object interference. Experimental results showed that the in-ear, open-ear, and bone-conduction devices achieved balanced accuracy (BAC) scores of 87.2–93.7%, 83.4–94.7%, and 85.9–90.0%.

This study shows a method for estimating users’ emotions in Virtual Reality (VR) spaces through the collection of eye behaviors. In our method, we use eye-related information available from the Head Mounted Display (HMD), including the direction vector of the gaze, coordination of the pupil, pupil diameter, and the eyelid opening width, to estimate whether the user is having fun or feeling others emotions. Using the LightGBM algorithm, the estimation accuracy resulted in an AUC of 0.84 and an accuracy of 0.78.

The goal of this study is to quantitatively clarify the experience of forgetting to turn off the microphone and camera in videoconferences and to develop a system to prevent users from forgetting to turn off the camera and microphone in our following research, in which we plan to use users’ unconscious behavior during videoconferences. First, we conducted a questionnaire survey on 60 participants to clarify their experiences of forgetting to turn off the microphone and camera. Next, we conducted a simulated videoconference study with eight participants to analyze gaze and head direction as unconscious behaviors.

This paper presents a system that authenticates users solely based on the action of grasping a door handle.The system authenticates users by utilizing the differences in frequency characteristics between a hand and a door handle when grasped.The frequency characteristics of a hand vary from person to person depending on anatomical differences such as skeletal structure and muscle composition.This system is beneficial because it is not vulnerable to shoulder surfing and allows authentication solely by grasping the door handle.Data were collected from 25 users, and the system achieved an F1 score of 93.0% in the 25-class classification.In the One-vs-Rest classification using unknown users as test data, the average equal error rate was 4.76%.

Thumb-to-finger gestures, such as 12 phalange taps, provide diverse input.This gesture enables eyes-free, rich, and one-handed smartwatch controls.Previous research on recognizing thumb-to-finger gestures has typically relied on additional sensors, making it difficult to achieve recognition using only a commercial off-the-shelf (COTS) smartwatch.In this paper, we developed a thumb-to-finger gesture recognition system that only uses the accelerometers built into a COTS smartwatch.Our recognition system achieved 80.1% accuracy for 17 gestures.Additionally, we developed optimal gesture sets for different numbers of gestures based on the recognition performance of the accelerometer.Consequently, we achieved 94.7% accuracy for 4 gestures in leave-one-participant-out crossvalidation and 90.2% accuracy for 11 gestures in participant-specific leave-one-session-out cross-validation.

We introduce EarHover, an innovative system that enables mid-air gesture input for hearables. Mid-air gesture input, which eliminates the need to touch the device and thus helps to keep hands and the device clean. However, existing mid-air gesture input methods for hearables have been limited to adding cameras or infrared sensors. By focusing on the sound leakage phenomenon unique to hearables, we have realized mid-air gesture recognition using a speaker and an external microphone that are highly compatible with hearables. The signal leaked to the outside of the device due to sound leakage can be measured by an external microphone, which detects the differences in reflection characteristics caused by the hand’s speed and shape during mid-air gestures. Among 27 types of gestures, we determined the seven suitable gestures for EarHover in terms of signal discrimination and user acceptability. We then evaluated the gesture detection and classification performance of two prototype devices (in-ear type/open-ear type) for real-world application scenarios.

The Steering Law is a robust model to predict the movement time (MT) for steering through a constrained path, and the most representative example in human-computer interaction (HCI) is navigating cascaded menus. In typical implementations of cascaded menus, however, users can deviate from the path for a short time; we call this error-accepting delay, or Tdelay. Yamanaka modified the Steering Law to predict MT under several Tdelay conditions, and our goal is to investigate the reproducibility of his model with more various Tdelay values. In addition, HCI researchers have recently formed a consensus that the goodness of models should be judged by the prediction accuracy for future (untested) task conditions. Thus, for the sake of completeness, we conducted two analyses: a shuffle-split cross-validation and leave-one-Tdelay-out cross-validation. The results showed that, regardless of the all-data and cross-validation analyses, Yamanaka’s modified model outperformed the baseline Steering Law, which strengthened his original experimental report.

Studies on gaze-based interactions have utilized natural eye-related information to detect user intent. Most use a machine learning-based approach to minimize the cost of choosing appropriate eye-related information. While those studies demonstrated the effectiveness of an intent detection system, understanding which eye-related information is useful for interactions is important. In this paper, we reanalyze how eye-related information affected the detection performance of a previous study to develop better intent detection systems in the future. Specifically, we analyzed two aspects of dimensionality reduction and adaptation to different tasks. The results showed that saccade and fixation are not always useful, and the direction of gaze movement could potentially cause overfitting.

In this paper, we first position the current dwell selection among gaze-based interactions and its advantages against head-gaze selection, which is the mainstream interface for HMDs. Next, we show how dwell selection and head-gaze selection are used in an actual interaction situation. By comparing these two selection methods, we describe the potential of dwell selection as an essential AR/VR interaction.

Among various methods to learn a second language (L2), such as listening and shadowing, Extensive Viewing involves learning L2 by watching many videos. However, it is difficult for many L2 learners to smoothly and effortlessly comprehend video contents made for native speakers at the original speed. Therefore, we developed a language learning assistance system that automatically adjusts the playback speed according to the learner's comprehension. Our system judges that learners understand the contents if they laugh at the punchlines of comedy dramas, and vice versa. Experimental results show that this system supports learners with relatively low L2 ability (under 700 in TOEIC Score in the experimental condition) to understand video contents. Our system can widen learners' possible options of native speakers' videos as Extensive Viewing material.

We investigated the relationship between dwell-time and the model human processor (MHP). First, we devised an equation that can represent the time taken for recognizing an image based on MHP. Then, we evaluated whether the equation can represent the time and wheter the time estimated by the equation matches the user’s preferred dwell-time. The experiment consisted of two tasks: image selection with a button (button-task) and image selection with a dwell (dwell-task). From the results of the button-task, we found that the equation derived by MHP can estimate the time; the time taken for button selection was 662 ms on average, and the time estimated by the equation was 660 ms on average. Also, we showed that the estimated time represented the user’s preferred dwell-time; all participants in the experiment answered that 500 ms and 600 ms were their preferred dwell-times.

We developed a hat-shaped touch interface for virtual reality viewpoint control. The hat is made of conductive fabric and thus is lightweight. The user can touch, drag, and push the surface, enabling three-dimensional viewpoint control.

We propose a hand gesture interaction method using a low-resolution infrared image sensor on an inner wrist. We attach the sensor to the strap of a wrist-worn device, on the palmar side, and apply machine-learning techniques to recognize the gestures made by the opposite hand. As the sensor is placed on the inner wrist, the user can naturally control its direction to reduce privacy invasion. Our method can recognize four types of hand gestures: static hand poses, dynamic hand gestures, finger motion, and the relative hand position. We developed a prototype that does not invade surrounding people's privacy using an 8 x 8 low-resolution infrared image sensor. Then we conducted experiments to validate our prototype, and our results imply that the low-resolution sensor has sufficient capabilities for recognizing a rich array of hand gestures. In this paper, we introduce an implementation of a mapping application that can be controlled by our specified hand gestures, including gestures that use both hands.

We demonstrate a gaze-based command activation technique that is robust against unintentional command activations using a series of dwelling on a target and performing a specific gesture (dwell-thengesture manipulation). The gesture adopted is a simple two-level stroke, which consists of a sequence of two orthogonal strokes. To achieve robustness against unintentional command activations, we designed and fine-tuned a gesture detection system based on how users move their gaze, as revealed through three experiments. Although our technique seems to simply combine well-known dwelland gesture-based manipulations, implying a low rate of success, our technique is actually the first technique that consists of a short time dwelling for target selection and a simple gesture for command activation. In addition, our technique will be the first technique adopting a marking menu, which is a traditional menu for command activation used in mouseor pen-based interactions to gaze-based interactions.

Urushi (Japanese lacquer) is a natural resin paint with electrical insulating capability. We focused on Japanese patterns on an urushi-ware that is a product coated urushi. To make urushi-wares interactive without losing elegance and beauty, we transformed Japanese patterns into near-field communication (NFC) antenna patterns. We developed three types of prototype antennas and confirmed their functionality. In addition, we developed an IC key tag and an interactive lunch box as example applications of interactive urushi-wares.

We present AudioTouch, a minimally invasive approach for sensing micro-gestures using active bio-acoustic sensing. It only requires attaching two piezo-electric elements, acting as a surface mounted speaker and microphone, on the back of the hand. It does not require any instrumentation on the palm or fingers; therefore, it does not encumber interactions with physical objects. The signal is rich enough to detect small differences in micro-gestures with standard machine-learning classifiers. This approach also allows for the discrimination of different levels of touch-force, further expanding the interaction vocabulary. We conducted four experiments to evaluate the performances of AudioTouch: a user study for measuring the gesture recognition accuracy, a follow-up study investigating the ability to discriminate different levels of touch-force, an experiment assessing the cross-session robustness, and, a systematic evaluation assessing the effect of sensor placement on the back of the hand.

We investigated the performance of 3D pointing using direct touch in a planar surface condition (PC) and a spherical surface condition (SC). In addition, we examined the performance in terms of Fitts’ law. Although the results showed that the performance in SC was slightly worse than PC, SC was higher conformed to Fitts’ law than PC without the conditions involving head rotation (PC’s and SC’s R2 is 0.945 and 0.971, respectively).

In this paper, we introduce a curved QWERTY keyboard, bent spherically in front of the user, to facilitate 3D word-gesture text entry in immersive virtual environments. Using the curved keyboard, the number of candidate words in the 3D word-gesture text entry is reduced compared with that using a planar keyboard. In the pilot study, the text entry performance of the first author was 21.0 WPM (SD = 5.06), with a total error rate of 26.0% (SD = 15.2).

We present a dwell time reduction technique for gaze-based target acquisition. We adopt Fitts' Law to achieve the dwell time reduction. Our technique uses both the eye movement time for target acquisition estimated using Fitts' Law (Te) and the actual eye movement time (Ta) for target acquisition; a target is acquired when the difference between Te and Ta is small. First, we investigated the relation between the eye movement for target acquisition and Fitts' Law; the result indicated a correlation of 0.90 after error correction. Then we designed and implemented our technique. Finally, we conducted a user study to investigate the performance of our technique; an average dwell time of 86.7 ms was achieved, with a 10.0% Midas-touch rate.

Urushi (Japanese lacquer) is a natural resin paint with electrical insulating capability. By using it as a base material and coating material for electronic circuits, it is possible to construct a circuit with an elegant appearance and feel. It is also possible to build a multilayered electronic circuit by using urushi as insulation layers. In this study, we investigate techniques to construct touch interfaces using a multilayered electronic circuit (urushi circuit). At first, we fabricated an urushi circuit with a touch electrode consisting of two layers. To improve its appearance, we fabricated urushi circuits in which all touch electrodes are arranged on the top layer and all wires are hidden in the bottom layer. Moreover, as an extension of the touch interface, we built a grid of touch electrodes that realizes two-dimensional touch sensing.

We present a jaw, face, or head movement (face-related movement) recognition system called CanalSense. It recognizes face-related movements using barometers embedded in earphones. We find that face-related movements change air pressure inside the ear canals, which shows characteristic changes depending on the type and degree of the movement. We also find that such characteristic changes can be used to recognize face-related movements. We conduct an experiment to measure the accuracy of recognition. As a result, random forest shows per-user recognition accuracies of 87.6% for eleven face-related movements and 87.5% for four OpenMouth levels.

We present B2B-Swipe, a single-finger swipe gesture for a rectangular smartwatch that starts at a bezel and ends at a bezel to enrich input vocabulary. There are 16 possible B2B-Swipes because a rectangular smartwatch has four bezels. Moreover, B2B-Swipe can be implemented with a single-touch screen with no additional hardware. Our study shows that B2B-Swipe can co-exist with Bezel Swipe and Flick, with an error rate of 3.7% under the sighted condition and 8.0% under the eyes-free condition. Furthermore, B2B-Swipe is potentially accurate (i.e., the error rates were 0% and 0.6% under the sighted and eyes-free conditions) if the system uses only B2B-Swipes for touch gestures.

Capacitive Blocks is a block-shaped tangible user interface that connects the physical space to the virtual space, allowing users to build 3D models in the virtual space by handling tangible blocks. The system counts the number of stacked blocks by measuring the capacitance with a capacitance meter; the system recognizes the structure of stacked blocks arranged in a grid using the capacitance meter and generates 3D models. In this study, we improved the stability of Capacitive Blocks, presented by Yoshida et al. at UIST '15, by revising the block design: each block of the new system has a built-in capacitor. To evaluate the improvement, we conducted an experiment. The result showed a counting accuracy of 96.5%.

In designing interactive products, it is important for designers to test the product's usability by manufacturing its shape and interface iteratively through rapid prototyping. The goal of our research is to provide the designers with an additional touch sensing method for rapid prototyping interactive products with flat, curved, or flexible surface. In this paper, we present RootCap, a capacitive touch sensing method that can detect a touch on a multi-electrode input surface while maintaining the characteristics of a single-line connection. The key concept behind realizing this goal is the imposition of unique capacitance on each electrode (including the capacitor connected to the touch electrode) branching from the single-line connection. Moreover, we developed a technique for creating a capacitor by printing silver nanoparticle ink on both sides of a sheet of paper, supporting designers in the creation of a multi-electrode input surface, on which each electrode has a unique capacitance.

We present a lightweight technique with which creators can prototype force-sensitive objects by attaching a pair of piezoelectric elements: one a vibration speaker and one a contact microphone. The key idea behind our technique is that touch force, in addition to the way the object is touched, can also be observed as different resonant frequency spectra. We also show that recognition of a touch and estimation of the touch force can be implemented by using the combination of support vector classification (SVC) and support vector regression (SVR). An experiment with an additional pressure sensor revealed that our technique might perform well in estimating touch force. We also show a tool for machine learning based on our technique that uses an animated guide, allowing creators to give the system both the training data and the labels for training machine learning needed for dealing with continuous-valued output such as SVR.

We describe a tool to analyze unlinked documents by visualizing networks extracted using textual analysis. Our focus is on developing an interactive visual analytics tool that enables a user to interactively observe data to detect features of the documents that may be known or unknown in advance. We have tested our tool using two data sets, one consisting of 1000 documents and the other of 360 documents. The resulting visualization showed that our tool provides a simple yet powerful method to identify trends and to find facts in the documents quickly due to its interactivity.

We present a single-handed and eyes-free Japanese kana text input system on touch screens of mobile devices. We first conducted preliminary experiments to investigate the accuracy with which subjects could single-handedly point to and flick without using their eyes. We found from the results that users can point at a screen that was divided into 2 x 2 with 100% accuracy and that users can flick at a 2 x 2 grid without using their eyes with 96.1% accuracy using our algorithm for flick recognition. The system used kana letter input based on two-stroke input with three keys to enable accurate eyes-free typing. First, users flick for consonant input, and then similarly flick for vowel input. We conducted a long-term user study to measure basic text entry speed and error rate performance under eyes-free conditions, and readability of transcribed phrases. As a result, the mean text entry speed was 51.2 characters per minute (cpm) in the 10th session of the user study and the mean error rate was 0.6% of all characters. The mean text entry speed was 33.9 cpm in the 11th session, which was conducted under totally eyes-free conditions and the mean error rate was 4.8% of all characters. We not only measured cpm and error rate, but also measured error rate of reading, which we devised as a novel metric to measure how accurately users can read transcribed phrases. The mean error rate of reading in the 11th session was 5.7% of all phrases.

In this paper, we present a novel acoustic touch sensing technique called Touch & Activate. It recognizes a rich context of touches including grasp on existing objects by attaching only a vibration speaker and a piezo-electric microphone paired as a sensor. It provides easy hardware configuration for prototyping interactive objects that have touch input capability. We conducted a controlled experiment to measure the accuracy and trade-off between the accuracy and number of training rounds for our technique. From its results, per-user recognition accuracies with five touch gestures for a plastic toy as a simple example and six hand postures for the posture recognition as a complex example were 99.6% and 86.3%, respectively. Walk up user recognition accuracies for the two applications were 97.8% and 71.2%, respectively. Since the results of our experiment showed a promising accuracy for the recognition of touch gestures and hand postures, Touch & Activate should be feasible for prototype interactive objects that have touch input capability.

We present an interactive tool for the exploratory analysis of a long-term video from a stationary camera. The tool consists of three key methods: spatial change visualization, temporal change visualization, and similarity-based video retrieval. The first two methods provide the summarization of the long-term video that lets the user know where and when changes frequently occurred during a certain period, enabling the user to find an event of interest from the long-term video. In addition, with the third method, the user can search the video for a similar event, enabling the user to count events of interest and to observe distributions of such events. These methods are uniformly implemented using frame differences in 1-bit depth, making the implementation of these methods simple but efficient.

We present several innovative interaction techniques for 3D target acquisition and indication. These techniques make use of the shape of our cylindrical multi-touch interface, which we are developing as an interface for a volumetric display. Our interaction techniques are used to control a spherical cursor in 3D space. Notably, our techniques use a two-handed operation in which the position and size of the area touched is mapped directly to the position and size of the spherical cursor. This mapping can instantly determine the 4DOF of the spherical cursor, i.e., its 3D coordinates and its radius, and thus allows rapid 3D target acquisition and indication.

We describe interaction techniques for browsing 3D media using our cylindrical multi-touch interface (CMTI). CMTI uses a cylinder wall as its controlling surface. Because the control area of the interface is in cylindrical polar coordinates, the use of the depth along the surface enables the user to interact in 3D space even though the controlling surface is still a 2D surface. Moreover, the nature of the multi-touch interface enables the user to manipulate objects and the viewpoint with both bare hands. The user can easily examine complex 3D media using the interaction techniques described with the power of CMTI.

To support understanding a specific GUI functionality in a program, we propose a visualization technique that presents the correspondence between the screens before and after the operation, as well as the traces of the source code executed by the operation. The presented traces include the executed lines and the method calls, which are fundamental information for understanding the functionality. They are represented as highlights that are superimposed on the entire source code.

This paper presents new interaction techniques that use a laser pointer to directly manipulate applications displayed on a large screen. The techniques are based on goal crossing, and the key is that the goals of crossing are the four peripheral screen areas, which are extremely large. This makes it very easy for users to execute commands, and the crossing-based interaction enables users to execute fast and continuous commands.

Spatial parser generators automatically generate a parser of visual languages by providing grammars. The grammar is specified using text. The grammar would be easier to understand if we used a figure to input the grammar. We describe an approach in This work that graphically defines grammar. Direct manipulation is used to define grammar in our approach, which helps the user understand the meaning. We implement the GIGA system, which enables the user to define the grammar by drawing figures. GIGA displays each element of a rule visually. The user can define the grammar easily and can understand the grammar visually. Moreover, GIGA outputs the defined visual system to a file written in CMC text form. The user can execute the system by inputting the file into Eviss.

To help programmers debug visual programs, we propose an animated execution method with supporting functions to be used with the animated execution. The animated execution animates state transitions as the program is executed. To make it easy for the programmer to interpret the animation, it is displayed in a manner similar to the visual program. The functions are used to narrow the possible locations of bugs and to test fixed components quickly. We have implemented animated execution and the functions on 3D-PP, our three-dimensional visual programming system.

We propose 'Trace View', a static visualization method for monitoring and debugging the dynamic behavior of programs written in dataflow visual programming languages. Trace View presents a hierarchical structure of the dataflow between nodes that is created over the execution time of the program. The view also serves as an interface that allows the programmer to select a data stream link when data must be examined during debugging. Moreover since visualization grows in size according to the life time of the program, we have developed techniques to scale the view using a multi-focus focus+context view.