Effective force feedback is critical for user immersion in VR. However, current solutions have limitations; ungrounded devices using propellers or air-jets often suffer from slow response times and bulky hardware, body-worn devices tend to hinder hand movement, and wrist-worn force-feedback devices usually restrict free wrist movement. To address these challenges, we present ConCon, a wrist-worn 3-DoF force-feedback device utilizing motors with electromagnetic clutches. ConCon’s three actuation units apply force to wrist along the radial/ulnar-deviation, flexion/extension, and proximal/distal directions. Clutches can control force transmission continuously, discretely, or impulsively by suddenly releasing a loaded-state. They also enable unimpeded free movement by minimizing mechanical resistance. We first evaluated ConCon’s technical performance, including force output, wrist manipulation range, wrist impedance while free movement, and clutch response time. Subsequently, a user study (N=12) across six VR scenarios (Slingshot, Door, Fishing, Handfan, Pistol, Spray) showed ConCon provided significantly higher fun, immersion, and realism than vibrotactile feedback.

Virtual Reality (VR) offers portable and flexible workspaces. However, enabling efficient and comfortable interactions without external input devices remains challenging. We propose leveraging redirected input to enable comfortable and touch-like interaction for quick and intuitive control. Our design study revealed that while touch interaction performs well with direct input, its performance degrades significantly under input redirection. In contrast, using pinch improves redirected input by providing self-haptic feedback and reducing input dimensionality, thereby compensating for spatial discrepancies. Based on these findings, we introduce Redirected Pinch, a bare-hand interaction technique that combines input redirection with pinch confirmation. It creates a virtual plane at waist height, remapping hand movements on the plane to a vertical window, with pinch gestures used for confirmation. A user study demonstrated that Redirected Pinch achieves a strong balance of accuracy, efficiency, comfort, and sense of agency across fundamental interactions.

While many modern gaming environments provide haptic feedback, laptop keyboard gaming remains largely without rich tactile interaction, despite a rapidly growing audience. In this paper, we propose the HapPalm interface, a novel laptop interface concept that delivers rich spatio-temporal vibrotactile feedback through the palmrest area, allowing players to feel game events with their palms. Our prototype uses dual 4×6 linear resonant actuator arrays. To render various game events with the HapPalm interface, our first study aims to create a haptic pattern dataset. Iterative design workshops identified 11 haptic pattern templates, of which our second study validated that they convincingly convey diverse game events. Our final study embedded these patterns into a custom game, showing that spatial haptics significantly improved fun, immersion, realism, and presence compared to non-spatial or no-haptic conditions. HapPalm interface demonstrates that palmrest-based haptics can enrich keyboard-only laptop gaming, providing an expressive and immersive tactile channel for future laptop interfaces.

A hard surface feels soft or elastic when carefully controlled vibrations are provided to a pressing finger. This compliance illusion method may be employed to augment graphical objects on smartphone screens with compliant tactile properties. However, product engineers encounter practical questions when attempting to use the compliance illusion method for their products. Should the waveform be sinusoidal? How much time delay is acceptable? Should the vibration axis be perpendicular to the surface? A series of experiments was conducted to address these questions. The first experiment revealed that the waveform could not be distinguished in the context of the compliance illusion method when applied to typical smartphones. The second experiment demonstrated that time delays greater than 25 ms and the use of the three vibration axes could be distinguished in the same context. The third experiment explored how the perceptual qualities of the compliance illusion change with variations in time delay and the axis of vibration. Time delay had significant effects on softness, smoothness, elasticity, and unpleasantness, while the vibration axis had significant effects on softness, smoothness, and elasticity.

Interacting with desktop GUIs often requires users to perform concurrent operations, such as scrolling through a document while selecting text or switching applications during file transfer. Although wheel mice support these tasks with a scroll wheel, touchpads lack this feature. In this paper, we introduce ThumbSwipe, an input technique for touchpads that enables concurrent operations by leveraging the thumb’s dexterity. We conducted two experiments to evaluate its performance for continuous and discrete input tasks. The first experiment demonstrated that ThumbSwipe achieves input throughputs comparable to those of cursor operations using the index or middle finger on a touchpad. The second experiment confirmed ThumbSwipe’s efficiency for discrete input tasks. Additionally, a third experiment assessed ThumbSwipe’s advantages in typical unimanual concurrent operation scenarios, showing improvements in touchpad user experience regarding task completion time, task load, and subjective usability ratings.

Multi-touch mice have been proposed to integrate the rich input vocabulary of multi-touch gestures into traditional computer mice. However, incorporating multi-finger gestures into a mouse introduces a design conflict: Mice are optimized for low friction to support smooth cursor navigation, whereas touch gestures require high friction for stability. In this paper, we present AutoPark—the concept of a multi-touch mouse that dynamically adjusts its friction based on user intent. AutoPark mouse operates in a low-friction mode for cursor movement and shifts to a high-friction state for multi-touch interactions. Our first user study measured the shear forces during touch gestures, confirming that conventional friction levels of mousepads are insufficient to maintain gesture input stability on a mouse. We developed an AutoPark prototype and conducted controlled experiments, comparing it with both conventional low-friction and gesture-favored high-friction setups. The results demonstrate that AutoPark enhances gesture stability while maintaining comparable pointing performance in our study tasks, supporting a richer gesture input vocabulary within mouse-based GUI interactions.

The paradigm of bare-hand interaction has become increasingly prevalent in Augmented Reality (AR) and Virtual Reality (VR) environments, propelled by advancements in hand tracking technology. However, a significant challenge arises in delivering haptic feedback to users’ hands, due to the necessity for the hands to remain bare. In response to this challenge, recent research has proposed an indirect solution of providing haptic feedback to the forearm. In this work, we present QuadStretcher, a skin stretch display featuring four independently controlled stretching units surrounding the forearm. While achieving rich haptic expression, our device also eliminates the need for a grounding base on the forearm by using a pair of counteracting tactors, thereby reducing bulkiness. To assess the effectiveness of QuadStretcher in facilitating immersive bare-hand experiences, we conducted a comparative user evaluation (n = 20) with a baseline solution, Squeezer. The results confirmed that QuadStretcher outperformed Squeezer in terms of expressing force direction and heightening the sense of realism, particularly in 3-DoF VR interactions such as pulling a rubber band, hooking a fishing rod, and swinging a tennis racket. We further discuss the design insights gained from qualitative user interviews, presenting key takeaways for future forearm-haptic systems aimed at advancing AR/VR bare-hand experiences.

We introduce Pro-Tact, a novel eyes-free pointing technique for interacting with out-of-view (OoV) VR menus. This technique combines rapid rough pointing using proprioception with fine-grain adjustments through tactile exploration, enabling menu interaction without visual attention. Our user study demonstrated that Pro-Tact allows users to select menu items accurately (95% accuracy for 54 items) in an eyes-free manner, with reduced fatigue and sickness compared to eyes-engaged interaction. Additionally, we observed that participants voluntarily interacted with OoV menus eyes-free when Pro-Tact’s tactile feedback was provided in practical VR application usage contexts. This research contributes by introducing the novel interaction technique, Pro-Tact, and quantitatively evaluating its benefits in terms of performance, user experience, and user preference in OoV menu interactions.

The palmrest area of laptops has the potential as an additional input space, considering its consistent palm contact during keyboard interaction. We propose Palmrest+, leveraging shear force exerted on the palmrest area. We suggest two input techniques: Palmrest Shortcut, for instant shortcut execution, and Palmrest Joystick, for continuous value input. These allow seamless and subtle input amidst keyboard typing. Evaluation of Palmrest Shortcut against conventional keyboard shortcuts revealed faster performance for applying shear force in unimanual and bimanual-manner with a significant reduction in gaze shifting. Additionally, the assessment of Palmrest Joystick against the laptop touchpad demonstrated comparable performance in selecting one- and two- dimensional targets with low-precision pointing, i.e., for short distances and large target sizes. The maximal hand displacement significantly decreased for both Palmrest Shortcut and Palmrest Joystick compared to conventional methods. These findings verify the feasibility and effectiveness of leveraging the palmrest area as an additional input space on laptops, offering promising enhanced typing-related user interaction experiences.

The ball-shaped device is considered an intuitive means for 3D interaction, but its effectiveness in object manipulation has not been fully explored yet. In this study, we explored the use of an in-hand ball-shaped controller for object manipulation in virtual reality. We developed a ball-shaped controller with pressure-sensing capabilities featuring specifically designed interactions for object manipulation, including selection, translation, rotation, and scaling. Evaluations are conducted on 6-DOF docking tasks involving translation and rotation, and on 7-DOF tasks that additionally include scaling. These tasks involved manipulating objects at both close and distant ranges. The results indicated that, while the performance of the ball-shaped controller for direct object manipulation was comparable to other methods, it significantly improved completion times and reduced task load for distant object manipulation. Furthermore, the ball-shaped controller effectively reduced wrist and arm movements and is the most preferred method.

The mouse wheel may induce physical fatigue by requiring users to clutch the wheel repeatedly when scrolling long distances. To address this problem, some mouse products provide a free-spinning mode in which users can spin the wheel to initiate inertial scrolling. However, the time and effort required to toggle the free-spinning mode may lower the usability of the mouse wheel. To enable inertial scrolling without the overhead of mode switching, we present TouchWheel, a physical mouse wheel with touch sensitivity and a virtual wheel model that enables users to perform flick-and-stop operations as they do on touchscreens. A user experiment revealed that participants could actively use TouchWheel’s flick-and-stop operation according to their different scrolling needs. For tasks that frequently required long-distance scrolling, TouchWheel outperformed a normal mouse wheel in task completion time and required fewer clutching and mode-switching actions than the normal mouse wheel and a spin-enabled mouse wheel. For tasks that required short-distance scrolling, TouchWheel performed similarly to other baseline options.

We propose a new interface concept; Proprioceptive Menu with Tactons, an eyes- and ears-free menu in a proprioceptive space that uses a set of Tactons to describe its menu items. While previous proprioceptive menus utilized vibrotactile feedback only to express the boundary between menu items, we spot an unexplored possibility that distinct vibrotactile stimuli, Tactons, can be used to describe the status of each menu item. As an instance of this concept, we present WristMenu with Tactons, an eyes- and ears-free menu using a one-dimensional proprioceptive space of the axial wrist rotation. Users can grasp an overview of each menu item’s status through Tactons delivered on their wrist skin and eventually decide to select the desired item. Our evaluation study showed that users could find a target item among five candidates and recognize its status in 5 seconds on average at the accuracies of 93% and 86% in seated and walking conditions, respectively. Through our evaluation, we could confirm the feasibility of the proposed concept.

The conventional handwriting interface on pen computers uses two parts of the screen. The user must put a text cursor where they want to insert the text and then write on a separate pop-up window that recognizes ink. The separation of the two spaces may cause user fatigue and discomfort because it requires additional gaze and hand movement, which may cause a decrease in text editing efficiency and additional cognitive burden. As an alternative to alleviate these potential problems, we revisited and explored the concept of direct-writing interfaces as used in the past. In this study, we designed and implemented a direct- and indirect-writing text editor. Subsequently, we conducted an experiment to compare the usability of both text editors. The results showed that users’ gaze movement distance and frequency could be reduced, and constant time could be saved for each text editing operation using the direct-writing interface. In addition, participants preferred the direct-writing interface when simple and frequent text editing was required. However, the overall throughput of the text editing task was not affected. Lastly, we provided guidelines for further development of a direct-writing interface by combining empirical results, comments, and observations.

We present ExpanStick, a novel thumbstick that augments video game interaction using the force that the user applies at the physical limit of the gamepad. Gamers frequently manipulate a thumbstick leaning on the limit, or physical boundary around it, either to apply maximum input or to secure physical support, which makes the contact force an easily accessible input channel. The ExpanStick preserves the form factor and usability of the thumbstick, while expanding its input space. We built the ExpanStick and inserted it in a commercial gamepad as a proof-of-concept prototype. Using the prototype, we conducted a design workshop with console gamers to design the ExpanStick game interaction. Through demonstration of the interactions, we found the early use experience with the ExpanStick is positive and well accepted by the game players.

We present Lattice Menu, a gaze-based marking menu utilizing a lattice of visual anchors that helps perform accurate gaze pointing for menu item selection. Users who know the location of the desired item can leverage target-assisted gaze gestures for multilevel item selection by looking at visual anchors over the gaze trajectories. Our evaluation showed that Lattice Menu exhibits a considerably low error rate (~1%) and a quick menu selection time (1.3-1.6 s) for expert usage across various menu structures (4 × 4 × 4 and 6 × 6 × 6) and sizes (8, 10 and 12°). In comparison with a traditional gaze-based marking menu that does not utilize visual targets, Lattice Menu showed remarkably (~5 times) fewer menu selection errors for expert usage. In a post-interview, all 12 subjects preferred Lattice Menu, and most subjects (8 out of 12) commented that the provisioning of visual targets facilitated more stable menu selections with reduced eye fatigue.

Virtual reality (VR) environments are expected to become future workspaces. An effective bimanual 3D manipulation technique would be essential to support this vision. A ball-shaped tangible input device that can be rolled in a hand is known to be useful for 3D object manipulation because such devices allow users to utilize their finger dexterity. In this study, we further explored the potential of a rollable in-hand controller. First, we evaluated the effects of its form factor on user behavior and performance. Although the size and shape of a rollable controller are expected to influence user behavior and performance, their effects have not been empirically explored in prior works. Next, we evaluated a rollable controller on bimanual 3D assembly tasks. A rollable controller may incur a high mental load as it requires users to use finger dexterity; therefore, the benefit of using such a device in each hand is not obvious. We found that a 5 cm-diameter ball-shaped controller was the most effective among the sizes and forms that we considered, and that a pair of in-hand rollable controllers showed significantly faster completion time than a pair of VR controllers for complex bimanual assembly tasks involving frequent rotations.

Users with visual impairments find it difficult to enjoy real-time 2D interactive applications on the touchscreen. Touchscreen applications such as sports games often require simultaneous recognition of and interaction with multiple moving targets through vision. To mitigate this issue, we propose ThroughHand, a novel tactile interaction that enables users with visual impairments to interact with multiple dynamic objects in real time. We designed the ThroughHand interaction to utilize the potential of the human tactile sense that spatially registers both sides of the hand with respect to each other. ThroughHand allows interaction with multiple objects by enabling users to perceive the objects using the palm while providing a touch input space on the back of the same hand. A user study verified that ThroughHand enables users to locate stimuli on the palm with a margin of error of approximately 13 mm and effectively provides a real-time 2D interaction experience for users with visual impairments.

Handheld controllers are an essential part of VR systems. Modern sensing techniques enable them to track users’ finger movements to support natural interaction using hands. The sensing techniques, however, often fail to precisely determine whether two fingertips touch each other, which is important for the robust detection of a pinch gesture. To address this problem, we propose AtaTouch, which is a novel, robust sensing technique for detecting the closure of a finger pinch. It utilizes a change in the coupled impedance of an antenna and human fingers when the thumb and finger form a loop. We implemented a prototype controller in which AtaTouch detects the finger pinch of the grabbing hand. A user test with the prototype showed a finger-touch detection accuracy of 96.4%. Another user test with the scenarios of moving virtual blocks demonstrated low object-drop rate (2.75%) and false-pinch rate (4.40%). The results and feedback from the participants support the robustness and sensitivity of AtaTouch.

Beyond a simple notification of incoming calls or messages, more complex information such as alphabets and digits can be delivered through spatiotemporal tactile patterns (STPs) on a wrist-worn tactile display (WTD) with multiple tactors. However, owing to the limited skin area and spatial acuity of the wrist, frequent confusions occur between closely located tactors, resulting in a low recognition accuracy. Furthermore, the accuracies reported in previous studies have mostly been measured for a specific posture and could further decrease with free arm postures in real life. Herein, we present Heterogeneous Stroke, a design concept for improving the recognition accuracy of STPs on a WTD. By assigning unique vibrotactile stimuli to each tactor, the confusion between tactors can be reduced. Through our implementation of Heterogeneous Stroke, the alphanumeric characters could be delivered with high accuracy (93.8% for 26 alphabets and 92.4% for 10 digits) across different arm postures.

Non-verbal behavior is essential for embodied agents like social robots, virtual avatars, and digital humans. Existing behavior authoring approaches including keyframe animation and motion capture are too expensive to use when there are numerous utterances requiring gestures. Automatic generation methods show promising results, but their output quality is not satisfactory yet, and it is hard to modify outputs as a gesture designer wants. We introduce a new gesture generation toolkit, named SGToolkit, which gives a higher quality output than automatic methods and is more efficient than manual authoring. For the toolkit, we propose a neural generative model that synthesizes gestures from speech and accommodates fine-level pose controls and coarse-level style controls from users. The user study with 24 participants showed that the toolkit is favorable over manual authoring, and the generated gestures were also human-like and appropriate to input speech. The SGToolkit is platform agnostic, and the code is available at https://github.com/ai4r/SGToolkit.

Smartwatches are a convenient substitute for smartphones in hand-busy situations, but there is limited interactivity to one-handed use of wrist-worn devices. In this paper, we propose WristDial, a one-handed integer input technique which uses quantization of the angular distance of the wrist rotation with nonvisual feedback. Using tactile feedback, we present four techniques based on counting of tactile stimuli. With one of them, 94.44% average accuracy and 2.81 s average completion time were acquired in the task of entering a number from the range 1 to 10. We also evaluate the quantitative performance of the combination of interruptible speech and non-speech auditory feedback. Our results show that when speech feedback exists, adding a continuous change of pitch or clicking sound at the boundaries does not have a significant effect on the performance. Using only speech feedback, 95.47% accuracy and 1.45 s completion time were acquired.

Promisingly, driving is adapting to a Voice User Interface (VUI) that lets drivers utilize diverse applications with little effort. However, the VUI has innate usability issues, such as a turn-taking problem, a short-term memory workload, inefficient controls, and difficulty correcting errors. To overcome these weaknesses, we explored supplementing the VUI with tactile interaction. As an early result, we present the Voice+Tactile interactions that augment the VUI via multi-touch inputs and high-resolution tactile outputs. We designed various Voice+Tactile interactions to support different VUI interaction stages and derived four Voice+Tactile interaction themes: Status Feedback, Input Adjustment, Output Control, and Finger Feedforward. A user study showed that the Voice+Tactile interactions improved the VUI efficiency and its user experiences without incurring significant additional distraction overhead on driving. We hope these early results open new research questions to improve in-vehicle VUI with a tactile channel.

There are needs for pen interaction on a laptop, and the market sees many pen-enabled laptop products. Many of these laptops can be transformed into tablets, when pen interaction is needed. In a real situation, however, a workflow often requires both keyboard and pen interactions, and such a convertible feature may not be effective. In this study, we introduce MirrorPad, a novel interface device contained in a laptop for direct pen interaction. It is both a normal touchpad and a viewport for pen interaction with a mirrored region on the screen. We report findings and decisions obtained from the design iterations that we conducted with users to refine MirrorPad toward the final design. In the user study, MirrorPad showed the same performance as that of the laptop configuration during keyboard interaction and a performance similar to that of the tablet configuration during pen interaction. The user study results confirmed that MirrorPad effectively supports a workflow, which requires interspersed keyboard and pen interactions, thereby achieving its initial goal.

Completing tasks on smartwatches often requires multiple gestures due to the small size of the touchscreens and the lack of sufficient number of touch controls that are easily accessible with a finger. We propose to increase the number of functions that can be triggered with the touch gesture by enabling a smartwatch to identify which finger is being used. We developed MagTouch, a method that uses a magnetometer embedded in an off-the-shelf smartwatch. It measures the magnetic field of a magnet fixed to a ring worn on the middle finger. By combining the measured magnetic field and the touch location on the screen, MagTouch recognizes which finger is being used. The tests demonstrated that MagTouch can differentiate among the three fingers used to make contacts at a success rate of 95.03%.

The space around the body not only expands the interaction space of a mobile device beyond its small screen, but also enables users to utilize their kinesthetic sense. Therefore, body-centric peephole interaction has gained considerable attention. To support its practical implementation, we propose OddEyeCam, which is a vision-based method that tracks the 3D location of a mobile device in an absolute, wide, and continuous manner with respect to the body of a user in both static and mobile environments. OddEyeCam tracks the body of a user using a wide-view RGB camera and obtains precise depth information using a narrow-view depth camera from a smartphone close to the body. We quantitatively evaluated OddEyeCam through an accuracy test and two user studies. The accuracy test showed the average tracking accuracy of OddEyeCam was 4.17 and 4.47cm in 3D space when a participant is standing and walking, respectively. In the frst user study, we implemented various interaction scenarios and observed that OddEyeCam was well received by the participants. In the second user study, we observed that the peephole target acquisition task performed using our system followed Fitts? law. We also analyzed the performance of OddEyeCam using the obtained measurements and observed that the participants completed the tasks with suffcient speed and accuracy.

Near-surface multi-finger tracking (NMFT) technology expands the input space of touchscreens by enabling novel interactions such as mid-air and finger-aware interactions. We present DeepFisheye, a practical NMFT solution for mobile devices, that utilizes a fisheye camera attached at the bottom of a touchscreen. DeepFisheye acquires the image of an interacting hand positioned above the touchscreen using the camera and employs deep learning to estimate the 3D position of each fingertip. We created two new hand pose datasets comprising fisheye images, on which our network was trained. We evaluated DeepFisheye's performance for three device sizes. DeepFisheye showed average errors with approximate value of 20 mm for fingertip tracking across the different device sizes. Additionally, we created simple rule-based classifiers that estimate the contact finger and hand posture from DeepFisheye's output. The contact finger and hand posture classifiers showed accuracy of approximately 83 and 90%, respectively, across the device sizes.

Force feedback has not been fully explored in modern gaming environments where a gamepad is the main interface. We developed various game interaction scenarios where force feedback through the thumbstick of the gamepad can be effective, and categorized them into five themes. We built a haptic device and control system that can support all presented interactions. The resulting device, FS-Pad, has sufficient fidelity to be used as a haptic game interaction design tool. To verify the presented interactions and effectiveness of the FS-Pad, we conducted a user study with game players, developers, and designers. The subjects used an FS-Pad while playing a demo game and were then interviewed. Their feedback revealed the actual needs for the presented interactions as well as insight into the potential design of game interactions when applying FS-Pad.

For human-like agents, including virtual avatars and social robots, making proper gestures while speaking is crucial in human-agent interaction. Co-speech gestures enhance interaction experiences and make the agents look alive. However, it is difficult to generate human-like gestures due to the lack of understanding of how people gesture. Data-driven approaches attempt to learn gesticulation skills from human demonstrations, but the ambiguous and individual nature of gestures hinders learning. In this paper, we present an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures. By incorporating a multimodal context and an adversarial training scheme, the proposed model outputs gestures that are human-like and that match with speech content and rhythm. We also introduce a new quantitative evaluation metric for gesture generation models. Experiments with the introduced metric and subjective human evaluation showed that the proposed gesture generation model is better than existing end-to-end generation models. We further confirm that our model is able to work with synthesized audio in a scenario where contexts are constrained, and show that different gesture styles can be generated for the same speech by specifying different speaker identities in the style embedding space that is learned from videos of various speakers. All the code and data is available at https://github.com/ai4r/Gesture-Generation-from-Trimodal-Context.

Prototyping electronic circuits is an increasingly popular activity, supported by researchers, who develop toolkits to improve the design, debugging, and fabrication of electronics. Although past work mainly dealt with circuit topology, in this paper we propose a system for determining or tuning the values of the circuit components. Based on the results of a formative study with seventeen makers, we designed VirtualComponent, a mixed-reality tool that allows users to digitally place electronic components on a real breadboard, tune their values in software, and see these changes applied to the physical circuit in real-time. VirtualComponent is composed of a set of plug-and-play modules containing banks of components, and a custom breadboard managing the connections and components' values. Through demonstrations and the results of an informal study with twelve makers, we show that VirtualComponent is easy to use and allows users to test components' value configurations with little effort.

In countries where languages with non-Latin characters are prevalent, people use a keyboard with two language modes namely, the native language and English, and often experience mode errors. To diagnose the mode error problem, we conducted a field study and observed that 78% of the mode errors occurred immediately after application switching. We implemented four methods (Auto-switch, Preview, Smart-toggle, and Preview & Smart-toggle) based on three strategies to deal with the mode error problem and conducted field studies to verify their effectiveness. In the studies considering Korean-English dual input, Auto-switch was ineffective. On the contrary, Preview significantly reduced the mode errors from 75.1% to 41.3%, and Smart-toggle saved typing cost for recovering from mode errors. In Preview & Smart-toggle, Preview reduced mode errors and Smart-toggle handled 86.2% of the mode errors that slipped past Preview. These results suggest that Preview & Smart-toggle is a promising method for preventing mode errors for the Korean-English dual-input environment.

Co-speech gestures enhance interaction experiences between humans as well as between humans and robots. Most existing robots use rule-based speech-gesture association, but this requires human labor and prior knowledge of experts to be implemented. We present a learning-based co-speech gesture generation that is learned from 52 h of TED talks. The proposed end-to-end neural network model consists of an encoder for speech text understanding and a decoder to generate a sequence of gestures. The model successfully produces various gestures including iconic, metaphoric, deictic, and beat gestures. In a subjective evaluation, participants reported that the gestures were human-like and matched the speech content. We also demonstrate a co-speech gesture with a NAO robot working in real time.

A watch-back tactile display (WBTD) is an attractive output option due to its always-available nature. However, employing commonly-used vibration modality on a WBTD may result in a low efficiency since its stimulation area is relatively wide compared with the small contact area of a watch-back. We considered using a more localized tactile stimulus, a poke, to improve the efficiency of a WBTD. We built a WBTD consisting of overlapping 3×3 poke and vibrotactile tactor arrays so that it may be used either as a poke display or as a vibrotactile display. An experiment was conducted to optimize the parameters of poke stimuli, and its results revealed that four directional patterns were best recognized when poking depth was deepest (3 mm) and sensory saltation was exploited. In the next two experiments, we compared the information transfer capacities of the poke and vibrotactile displays. The information transfer capacity of the poke display (1.55 bits) was shown to be higher than that of the vibrotactile display (1.32 bits) in a simulated environment with the mental load of a primary task. This result confirmed our expectation that using a more localized tactile stimulus would improve the efficiency of a WBTD.

Indirect writing interface is currently widely used for handwriting text entry with a pen, however, it may cause additional eye and hand movement. Gu and Lee suggested a revival of the concept of direct writing and they showed the potential of the direct writing interface in editing text. In this demo, we demonstrate an In-Place Ink text editor prototype which is a notepad-like simple text editor with direct writing feature.

There are needs for pen interaction on a laptop, and the market sees many pen-enabled laptop products. Many of them can be transformed into tablets when pen interaction is needed. In a real situation, however, users use multiple applications simultaneously, and such a convertible feature may not be useful. We introduce MirrorPad, a novel interface device contained in a laptop for direct pen interaction. It is both a normal touchpad and a viewport for pen interaction with a mirrored region on the screen. MirrorPad allows users to perform pen annotation work and keyboard work interspersed in the same workflow while the device remains in the laptop-like configuration.

Text entry is expected to be a common task for smart glass users, which is generally performed using a touchpad on the temple or by a promising approach using eye tracking. However, each approach has its own limitations. For more efficient text entry, we present the concept of gaze-assisted typing (GAT), which uses both a touchpad and eye tracking. We initially examined GAT with a minimal eye input load, and demonstrated that the GAT technology was 51% faster than a two-step touch input typing method (i.e.,M-SwipeBoard: 5.85 words per minute (wpm) and GAT: 8.87 wpm). We also compared GAT methods with varying numbers of touch gestures. The results showed that a GAT requiring five different touch gestures was the most preferred, although all GAT techniques were equally efficient. Finally, we compared GAT with touch-only typing (SwipeZone) and eye-only typing (adjustable dwell) using an eye-trackable head-worn display. The results demonstrate that the most preferred technique, GAT, was 25.4% faster than the eye-only typing and 29.4% faster than the touch-only typing (GAT: 11.04 wpm, eye-only typing: 8.81 wpm, and touch-only typing: 8.53 wpm).

Interpersonal touch, one of the most primitive social languages, is an excellent design element frequently used in interaction design. In this paper, we present a richer understanding of it by using spatial factors and social relations among people, which has rarely been explored in interactive systems. We designed an interactive installation called "TouchBranch" where players can move light between branches placed at various distances by connecting their bodies. The user studies were conducted with 21 groups consisting of intimates, acquaintances, and strangers. We observed a change in the interpersonal touch pattern and touch tolerance according to each factor. Interestingly, the effect of the social relation was dramatic, but that of the spatial factor was not quantitatively significant. Nevertheless, we discovered that spatial factor can influence the interpersonal touch experience. Based on the results, we discuss in this paper the influence of two factors on the interpersonal touch that stands out in the context of interactive systems.

We present a new handheld haptic device, Thor's Hammer, which uses propeller propulsion to generate ungrounded, 3-DOF force feedback. Thor's Hammer has six motors and propellers that generates strong thrusts of air without the need for physical grounding or heavy air compressors. With its location and orientation tracked by an optimal tracking system, the system can exert forces in arbitrary directions regardless of the device's orientation. Our technical evaluation shows that Thor's Hammer can apply up to 4 N of force in arbitrary directions with less than 0.11 N and 3.9° of average magnitude and orientation errors. We also present virtual reality applications that can benefit from the force feedback provided by Thor's Hammer. Using these applications, we conducted a preliminary user study and participants felt the experience more realistic and immersive with the force feedback.

A tactile display on the back of a smartwatch is an attractive output option; however, its channel capacity is limited owing to the small contact area. In order to expand the channel capacity, we considered using two perceptually distinct types of stimuli, wind and vibration, together on the same skin area. The result is a multimodal tactile display that combines wind and vibration to create "colored" tactile sensations on the wrist. As a first step toward this goal, we conducted in this study four user experiments with a wind-vibration tactile display to examine different ways of combining wind and vibration: Individual, Sequential, and Simultaneous. The results revealed the sequential combination of wind and vibration to exhibit the highest potential, with an information transfer capacity of 3.29 bits. In particular, the transition of tactile modality was perceived at an accuracy of 98.52%. The current results confirm the feasibility and potential of a multimodal tactile display combining wind and vibration.

As smartwatch functions expand, target selection among many items will probably become a common task. List search interfaces (LSIs) for a smartwatch use a prefix matching query to search for items, and need two modes because of the small screen size: query input mode and list navigation mode. Despite the modes, LSIs may be more efficient than list interfaces (LIs), which involve no text querying, for large pools. Actually, we could show that the LSI outperformed the LI for pool sizes over 60. However, we also found that the LSI users experience overhead when deciding whether to switch the modes or not. To reduce the overhead, we designed two auto-switching LSIs: input-length-based auto-switching LSI (IA-LSI) and list-size-based auto-switching LSI (LA-LSI). We could show both auto-switching LSIs outperformed the conventional LSI for pool sizes over 60. We also conducted experiments with the auto-switching LSIs for various pool sizes, and provided their results and guidelines for the optimal switching criteria for the LSIs.

Tap-typing with a pen on a virtual keyboard is often used for intermittent text entry on a pen-equipped tablet; however, this has usability issues owing to the large size of the virtual keyboard. We speculated that tap-typing with a pen on a tablet may not require such a large keyboard, but there was no empirical study to help us to determine the best virtual keyboard size for it. Therefore, we conducted two experiments. In the first experiment, we compared virtual keyboards with different key sizes (3, 4, 5, 6, 8, 10, and 13 mm) using a text entry task. The results showed that keyboards with 6-, 8-, and 10-mm keys were the best when text entry speed, task load, and preference were considered. In the second experiment, a keyboard with 6-mm keys was compared with other conventional pen text entry options using a presentation slide editing task with a pen. The results showed that the keyboard with 6-mm keys performed better than the others in terms of preference, System Usability Scale (SUS) scores, and ease-of-learning scores in a Usefulness, Satisfaction, and Ease of use (USE) questionnaire.

The QWERTY keyboard has a wide form factor, whereas smartwatches frequently have round or square form factors. Even if extra space for an input field is considered, the aspect ratio of the QWERTY keyboard may not be optimal for a round or square smartwatch. We surmised that a narrower keyboard would be better able to utilize the screen area of a round or square smartwatch, thereby enabling larger keys and improved performance. Larger keys, however, may not necessarily result in better performance because of the reduced familiarity of a modified keyboard layout. To investigate this, we designed DiaQwerty keyboards, which are QWERTY variants with an aspect ratio of 10:7, and compared them with a plain QWERTY keyboard and a SplitBoard on round and square smartwatches. For a 33 mm round watch, DiaQwerty was comparable with QWERTY and was faster than SplitBoard. For a 24 mm x 30 mm square watch, DiaQwerty was faster than both QWERTY and SplitBoard. In the latter case, DiaQwerty achieved a 15% improvement over QWERTY. We concluded that the advantages of the enlarged keys outweighed the disadvantages of the reduced familiarity of the modified layout on a square watch.

As tablet computers are hosting more productivity applications, efficient text entry is becoming more important. A soft keyboard, which is the primary text entry interface for tablets, however, often competes with applications for the limited screen space. A promising solution to this problem may be a semi-transparent soft keyboard (STK), which can share the screen with an application. A few commercial STK products are already available, but research questions about the STK design have not been explored in depth yet. Therefore, we conducted three experiments to answer 1) the effect of the transparency level on usability, 2) exploration of diverse design options for an STK, and 3) the effect of an STK on the different text caret positions. The results imply that STKs with 50% opacity showed a balanced performance; well-designed STKs were acceptable in both content reading and typing situations; which could reach 90-100% of an opaque keyboard in terms of overall performance; and the text caret could intrude the STK down to the number row.

Despite the advantages of instant messaging (IM) such as simple and fast messaging, IM cannot be considered as effective as face-to-face actual conversation. The current message-by-message interface shows a gap with actual conversation in terms of temporal structure and information dimension levels. Our study investigated the real-time text-displaying interface as a way of shortening this gap. A comparative user study was conducted with three types of IM interfaces to compare and analyze the duration of silence (short pause when nobody types through the IM) quantitatively. Varying chatting experiences in each IM interface were qualitatively measured through a focus group interview. We report the study results regarding the reduced duration of silence, non-verbal expression channel, and new texting techniques found in the real-time text interface. We also discuss design implications for improving the conversing experience for IM interfaces.

We explored new interaction scenarios that can be realized when a touchpad outputs fast and high-resolution spatio-temporal tactile patterns to the touch-sensitive skin on the fingertips of a user. We first constructed a special tactile multi-touch touchpad called PinPad, which was capable of outputting fast and high-resolution tactile patterns using a 40 x 25 array of actuated pins. We then developed various interaction scenarios that could be realized using the prototype: 1) Tactile Target, 2) Guide and Constraint, 3) Multi-finger Output, and 4) Dynamic Partition. To evaluate the PinPad scenarios, we implemented demo applications, and conducted interviews with users to collect feedback about their experiences with PinPad and the PinPad scenarios. The participants confirmed the effectiveness of spatio-temporal outputs of PinPad in the scenarios. In particular, they provided diverse feedback regarding the unique tactile experiences of the fast and high-resolution outputs of PinPad.

This paper reviews and categorizes the most common types of selfie sticks available in the market and discusses their design and usability. Through a theoretical analysis, it demonstrates how most commercial selfie sticks ignore important human factors, including ergonomics. It, then, presents results of an online survey, where selfie stick users (N=105), predominantly from the Republic of Korea, rated the usability of their selfie sticks. Results of the survey provided an insight into users' selfie stick usage behavior and suggested that most commercial selfie sticks are unergonomic, causing the users short-term fatigues that could "in theory" turn into chronic over the time with extensive use. Finally, based on the survey results, the paper makes recommendations for future design.

While smart glasses make information more accessible in mobile scenarios, entering text on these devices is still difficult. In this paper, we suggest using a smartwatch as an indirect input device for smart glasses text entry. With the watch-glasses combination, users do not need to lift the arm to touch the glasses nor need to carry a special external input device. To prove the feasibility of the suggested combination, we implemented two text entry methods: a modified version of SwipeBoard, which we adapted for the suggested combination, and HoldBoard, which we newly designed and implemented specifically for the suggested combination. We evaluated the performances of the two text entry methods through two user studies, and could show that they are faster than prior art for smart glasses text entry in a seated condition. A further study showed that they are competitive with the prior art also in a walking condition.

Small touchscreen interfaces such as a smartwatch have usability problems due to the small screen. One solution to these problems is to utilize the space around the smartwatch as continuous input space for the touchscreen interface. We defined four steps for a gesture that starts on the touchscreen and continues in the air. The goal of this definition was to bring the experience of large touchscreen devices into a smartwatch usage. We compared design options for the four steps and made decisions for the options based on the results of four user experiments. We expect that gestures designed based on these decisions will be both easy to learn and robust.

As an option for remote text entry, the possibility of bimanual text entry was explored using a touchpad that is called two-thumb touchpad typing (4-T). The design of a 4-T interface presents a few important design options to examine, and one of them is a target selection method. In this study, two 4-T interfaces were designed, called SlideType and FlyType. These interfaces use two different target selection methods, slide-and-click and fly-and-touch, respectively, and their prototypes were implemented using an optical hover-tracking touchpad. An experiment with the prototypes showed that FlyType was faster than SlideType and could achieve a text entry speed over 24 words per minute after practicing 2 hr. In addition, a microanalysis of thumb movements during the experiment showed that there are clear quantitative differences between the two methods in the degree of ballisticity of finger motions, in the concurrency of the movements of the two thumbs, and in the degree of integration between reaching and selecting actions, and that FlyType supports a more integral, ballistic, and concurrent motor skill than SlideType.

Two-finger operations, such as rotation operations and zoom operations, are now standard touchscreen operations. However, these two-finger operations may pose an ergonomic problem when they are performed with one hand. FingerSkate is an interaction technique to mitigate the ergonomic problem of two-finger operations adopting the concept of gesture relaxation. A preliminary study showed that the FingerSkate technique was accepted well by the participants. In particular, a micro-analysis of finger movements during the study showed that FingerSkate was utilized well for large rotations. In a subsequent user study, the effects of two FingerSkate design options, Concurrency and Pivot, were examined. The Concurrency option had a statistically significant effect on performance and task workload for a docking task. The Pivot option had a statistically significant effect on task workload for a puzzle task. More participants preferred the sequential Concurrency option, and most participants preferred the center-of-object Pivot option.

We present a new smart-restorable backspace technique to facilitate correction of "overlooked" errors on touchscreen-based tablets. We conducted an empirical study to compare the new backspace technique with the conventional one. Results of the study revealed that the new technique improves the overall text entry performance, both in terms of speed and operations per character, by significantly reducing error correction efforts. In addition, results showed that most users preferred the new technique to the one they use on their tablets, and found it easy to learn and use. Most of them also felt that it improved their overall text entry performance, thus wanted to keep using it.

We introduce TapBoard 2, a touchpad-based keyboard that solves the problem of typing and pointing disambiguation. The pointing interaction design of TapBoard 2 is nearly identical to natural touchpad interaction, and its shared workspace naturally invites bimanual pointing interaction. To implement TapBoard 2, we developed a novel gesture representation scheme for a systematic design and gesture recognizer. A user evaluation showed that TapBoard 2 successfully supports collocated pointing and typing interaction. It was able to disambiguate typing and pointing actions with an accuracy of greater than 95%. In addition, the typing and pointing performance of TapBoard 2 were comparable to that of a separate keyboard and mouse. In particular, the bimanual pointing operations of TapBoard 2 are highly efficient and strongly favored by participants.

An indirect handwriting interface using a separate window is a de facto standard interface in most of major pen computing systems. In this paper, we study the potential of the other less explored handwriting interfaces, which we call In-Place-Ink (IPI) interfaces, where a user writes directly on a target text input area. We first conducted a preliminary experiment to determine the optimal input area dimensions for direct handwriting input, and based on the results designed two IPI interfaces: a discrete IPI (DIPI) interface and a continuous IPI (CIPI) interface. We then conducted a main experiment using form-filling tasks to evaluate their efficiencies and collect user feedback about them in comparison with a conventional indirect writing (IW) interface. The experiment results showed that CIPI required significantly lower completion time for a form-filling task than other interfaces, and was comparable to the IW interface in terms of user preference. We hope our report of designing and evaluating the two IPI interfaces may stimulate future research efforts toward more direct handwriting interfaces.

Traditional wearable tactile displays transfer tactile stimulations through a firm contact between the stimulator and the skin. We conjecture that a firm contact may not be always possible and acceptable. Therefore, we explored the concept of a non-contact wearable tactile display using an airflow, which can transfer information without a firm contact. To secure an empirical ground for the design of a wearable airflow display, we conducted a series of psychophysical experiments to estimate the intensity thresholds, duration thresholds, and distance thresholds of airflow perception on various body locations, and report the resulting empirical data in this paper. We then built a 4-point airflow display, compared its performance with that of a vibrotactile display, and could show that the two tactile displays are comparable in information transfer performance. User feedback was also positive and revealed many unique expressions describing airflow-based tactile experiences. Lastly, we demonstrate the feasibility of an airflow-based wearable tactile display with a prototype using micro-fans.

The QWERTY keyboard has been a de facto standard for computer text entry and continues to be one for mobile text entry such as for smartphones. It is not clear, however, that it will continue to be an option for text entry for much smaller devices such as smartwatches. In a series of user experiments, we examined the performance of the QWERTY keyboard when it is reduced to fit a small smartwatch screen. At the same time, we examined whether the ZoomBoard and the SplitBoard, which are QWERTY keyboards augmented by zooming and panning strategies, respectively, would be effective in comparison with a plain QWERTY keyboard. In Experiment 1, we evaluated the text entry performance of new users on the three QWERTY keyboards. In Experiment 2, we evaluated the relative performance of the three keyboards for three different screen sizes. In Experiment 3, we further observed how the keyboard performance changed when used in a mobile situation. Main results are: (i) users could adapt to a plain QWERTY keyboard even in the smallest screen cases. (ii) The SplitBoard consistently showed a better performance than other keyboards in all tested sizes. (iii) The SplitBoard showed a better performance than other keyboards in a mobile condition (treadmill) and was preferred most by participants.

The two main tasks of a smart TV GUI are menu navigation and free pointing. Traditional remotes with directional keys are suitable for menu navigation but may not be so for free pointing. More recent remotes with a two-dimensional (2D) pointing device are suitable for free pointing but may not be so for menu navigation. To support both types of tasks well, we devised a new input device called TouchRoller. We expect that it can support both types of tasks well because it has a separable control structure and a continuous input property. A comparative user study showed that the performance of TouchRoller is comparable to that of directional keys for menu navigation and 2D pointing devices for free pointing. In addition, it was most favored by the participants, and NASA TLX test results showed that TouchRoller demands the lowest task load.

Text entry on a smartwatch is a challenging problem due to the device's limited screen area. In this paper, we introduce the SplitBoard, which is a soft keyboard designed for a smartwatch. As the user flicks left or right on the keyboard, it switches between the left and right halves of a QWERTY keyboard. We report the results of two user experiments where the SplitBoard was compared to an ordinary QWERTY keyboard, the ZoomBoard, SlideBoard, and Qwerty-like keypad. We measured the initial performance with new users for each method. The SplitBoard outperformed all other techniques in the experiments. The SplitBoard is expected to be a viable option for smartwatch text entry because of its light processing requirements, good performance, and immediate learnability.

A watch-back tactile display (WBTD) is expected to be a viable supplement to the user interface limitations of a smartwatch. However, its design requires that many design parameters such as tactor types and stimulus patterns be determined. We conducted a series of experiments to explore the design space of a WBTD consisting of 3×3 tactors. We demonstrated that tactor types and the temporal patterns and locus of a stimulus produce statistically significant effects on the efficiency of a WBTD. The experimental results can act as a practical guideline for the design of an efficient WBTD.

A page transition operation on touch interfaces is a common and frequent subtask when one conducts a drag-like operation such as selecting text and dragging an icon. Traditional page transition gestures such as scrolling and flicking gestures, however, cannot be conducted while conducting the drag-like operation since they have a confliction. We proposed Push-Push that is a new drag-like operation not in conflict with page transition operations. Thus, page transition operations could be conducted while performing Push-Push. To design Push-Push, we utilized the hover and pressed states as additional input states of touch interfaces. The results from two experiments showed that Push-Push has an advantage on increasing performance and qualitative opinions of users while reducing the subjective overload.

In recent years, touch screens have emerged and matured as the main input interface for mobile and tablet computers calling for extended touch input possibilities. In this paper, we explore the use of consecutive distant taps to expand the touch screen input vocabulary. We analyzed time intervals and distances between consecutive taps during common applications on a tablet and verified that consecutive distant taps can be used conflict-free with existing touch gestures. We designed the two interaction techniques Ta-tap and Ta-Ta-tap that utilize consecutive distant taps. Ta-tap uses two consecutive distant taps to invoke alternative touch operations for multi-touch emulation, whereas Ta-Ta-tap uses a series of consecutive distant taps to define a spatial gesture. We verified the feasibility of both interaction techniques through a series of experiments and a user study. The high recognition rate of Ta-tap and Ta-Ta-tap gestures, the few conflicts with existing gestures, and the positive feedback from the participants assert the potential of consecutive distant taps as a new design space to enrich touch screen interactions.

Although reliefs are frequently used to add patterns to product surfaces, there is a lack of interaction techniques to model reliefs on the surface of virtual objects. We adopted the repoussé and chasing artwork techniques in an alternative interaction technique to model relief on virtual surfaces. To support this interaction technique, we developed the double-sided touchpad Trampoline that can detect the position and force of a finger touch on both sides. Additionally, Trampoline provides users with elastic feedback, as its surface consists of a stretchable fabric. We implemented a relief application with this device and the developed interaction technique. An informal user study showed that the proposed system can be a promising solution to create reliefs.

In this paper, we explore the possibility of a long touchpad that utilizes the entire area below the keyboard of a laptop computer. An essential prerequisite for such a touchpad is a robust palm rejection method, which we satisfy using a proximity-sensing touchpad. We developed LongPad, a proximity-sensing optical touchpad that is as wide as a laptop keyboard, and implemented a palm rejection algorithm that utilizes proximity images from LongPad. In a user study conducted, we observed that LongPad rejected palm touches almost perfectly while participants were repeating typing and pointing tasks. We also summarize the new design space enabled by LongPad and demonstrate a few of the interaction techniques it facilitates.

The possibility of using shear forces is being explored recently as a method to enrich touch screen interaction. However, most of the related studies are restricted to the case of single-point shear forces, possibly owing to the difficulty of independently sensing shear forces at multiple touch points. In this paper, we propose indirect methods to estimate shear forces using the movement of contact areas. These methods enable multi-point shear force estimation, where the estimation is done for each finger independently. We show the feasibility of these methods through an informal user study with a demo application utilizing these methods.

A physical keyboard key has three states, whereas a touch screen usually has only two. Due to this difference, the state corresponding to the touched state of a physical key is missing in a touch screen keyboard. This touched state is an important factor in the usability of a keyboard. In order to recover the role of a touched state in a touch screen, we propose the TapBoard, a touch screen software keyboard that regards tapping actions as keystrokes and other touches as the touched state. In a series of user studies, we validate the effectiveness of the TapBoard concept. First, we show that tapping to type is in fact compatible with the existing typing skill of most touch screen keyboard users. Second, users quickly adapt to the TapBoard and learn to rest their fingers in the touched state. Finally, we confirm by a controlled experiment that there is no difference in text-entry performance between the TapBoard and a traditional touch screen software keyboard. In addition to these experimental results, we demonstrate a few new interaction techniques that will be made possible by the TapBoard.

Our goal is to show that it is possible to automatically infer social relationship types among people who stay together in an organization by analyzing communication patterns. We collected indoor co-location data and instant messenger data from 22 participants for one month. Based on the data, we designed and explored several indicators which are considered to be useful for mining social relationship types. We applied machine learning techniques using the indicators and found that it is possible to develop an intelligent method to infer social relationship types.

In this paper, we present a haptic feedback method for a virtual button based on the force-displacement curves of a physical button. The original feature of the proposed method is that it provides haptic feedback, not only for the "click" sensation but also for the moving sensation before and after transition points in a force-displacement curve. The haptic feedback is by vibrotactile stimulations only and does not require a force feedback mechanism. We conducted user experiments to show that the resultant haptic feedback is realistic and distinctive. Participants were able to distinguish among six different virtual buttons, with 94.1% accuracy even in a noisy environment. In addition, participants were able to associate four virtual buttons with their physical counterparts, with a correct answer rate of 79.2%.

We explored the possibility of touch-screen-like interaction with a remote control in the TV-viewing environment. A shadow representing the user's thumb touches the screen, presses a button, flicks a cover-flow list, and draws a simple stroke, while the thumb stays and moves on and above the touchpad. In order to implement the concept we developed an optical touchpad for tracking the thumb hovering over its surface, and designed a TV application to demonstrate possible new interaction styles. Throughout two iterations of prototyping, we corrected some of our false expectations, and also verified its potential as a viable option for a TV remote control. This paper presents technical issues and requirements for the hover-tracking touchpad and a complete report of our user studies to explore touch-screen-like interaction for the TV.

We introduce an interaction technique that increases the touch screen input vocabulary by distinguishing a strong tap from a gentle tap without the use of additional hardware. We have designed and validated an algorithm that detects different types of screen touches by combining data from the built-in accelerometer with position data from the touch screen. The proposed technique allows a touch screen input to contain not only the position of a finger contact, but also its type, i.e., whether the contact is a 'Tap' or a 'ForceTap.' To verify the feasibility of the proposed technique we have implemented our detection algorithm in experiments that test cases of single-handed, two-handed, immersive, and on the move usage. Based on the experimental results, we investigate the advantages of using two types of touch inputs and discuss emerging issues. Finally, we suggest a design guideline for applying the proposed technique to touch screen applications, and present possible application scenarios.

Six-degrees-of-freedom (6-DOF) trackers, which were mainly for professional computer applications, are now in demand by everyday consumer applications. With the requirements of consumer electronics in mind, we designed an optical 6-DOF tracker where a few photo-sensors can track the position and orientation of an LED cluster. The operating principle of the tracker is basically source localization by solving an inverse problem. We implemented a prototype system for a TV viewing environment, verified the feasibility of the operating principle, and evaluated the basic performance of the prototype system in terms of accuracy and speed. We also examined its application possibility to different environments, such as a tabletop computer, a tablet computer, and a mobile spatial interaction environment.

Force gestures are touch screen gestures augmented by the normal and tangential forces on the screen. In order to study the feasibility of the force gestures on a mobile touch screen, we implemented a prototype touch screen device that can sense the normal and tangential forces of a touch gesture on the screen. We also designed two example applications, a web browser and an e-book reader, that utilize the force gestures for their primary actions. We conducted a user study with the prototype and the applications to study the characteristics of the force gestures and the effectiveness of their mapping to the primary actions. In the user study we could also discover interesting usability issues and collect useful user feedback about the force gestures and their mapping to GUI actions.

We designed an algorithm to build a speed-call list adaptively based on mobile phone call logs. Call logs provide the time-dependent calling patterns of mobile phone users, and therefore a speed-call list based on them will be more successful in recommending a desired number than a speed-call list based on recent calls only. This paper presents the design process of our algorithm for an adaptive speed-call list, its verification result with recorded call logs, and in-situ evaluation results of the algorithm using an Experience Sampling Method (ESM) system.

A mobile phone with a camera enabled people to capture moments to remember in the right time and place. Due to a limited user interface, however, a mobile phone is not yet a platform for enjoying the captured moments. We explored possible application scenarios for promoting utilization of user-created photos on a mobile phone. For the realization of the scenarios, we designed context-aware photo selection algorithms that take into consideration mobile phone contexts such as the current location and recent calls. A user study was conducted with a mobile phone prototype for the evaluation of the photo selection algorithms and also for user feedback about the photo consumption scenarios.

We present a real-time simulation of spray painting incorporated into a VR environment as an alternative training system for ship-building industries. The system allows the user to try out a painting work on life-size structures with a spray gun. Our goal is to provide a trainee realistic painting experience in real-time as well as to represent the thickness of the deposited paint on the surface for evaluation of his performance. The Gaussian model is used for a painting deposition model, and texture mapping technique is utilized to provide efficient visual feedback. We also present effective collision detection methods for a volume of spray paint particles

This paper introduces an interactive device called "UbiRo" that is a convenient device for object registration and recognition of service robots. To register the objects with computer vision, user needs to cut the object off from its background on an image that is taken for the registration process. Taking images and segmenting the object from the images by user can be a time taking and unpleasant work. To solve this problem, we use a new computer vision interface that combines a user's simple gesture and computer vision. We call it `interactive computer vision' and the technique is applied to UbiRo. With UbiRo, users can effectively segment the object from the images taken and control the mouse pointer on display of robot to make the registration easy. The same interactive computer vision is also effective for object recognition. We present here the results on two experiments of object recognition and an additional usability test for controlling mouse pointer with UbiRo

The bifurcating neuron (BN) is an integrate-and-fire model neuron that exhibits bistability and attractor-merging crisis. The bifurcating neuron network 1, a network of BNs, was shown to be an associative memory that is robust against spurious minima problems. It was noted that the BN can in fact have multi-stability, which would make the BN a chaotic, multi-state, integrate-and-fire model neuron. This paper is a report of our preliminary study on the tri-stability of the BN, the behaviors of interacting such BNs, and a design of a network of BNs that interact via "mutually-exclusive" coupling, thereby solving a 3-coloring problem.

"Tilt scrolling" and "peephole display" are popular user interface ideas for small computers, and inertial sensors were often the choice for the realization of such ideas. Inertial sensors, however, have two fundamental limitations; the frame of reference is not the user but the earth, and drifting errors are difficult to overcome. A possibly better solution, free from such limitations, is machine vision. Machine vision was a luxury for small computers but is becoming a practical solution because a camera is now a common component of small computers and the required vision algorithm is already running in optical mice. The vision algorithm of our prototype, which we called ISeeU, tracks simple features in the user's face and calculates lateral displacements and changes in distance, which in turn are used to control scrolling and zooming. An informal test with scrolling tasks indicates that its performance is comparable with a user interface using arrow keys.

The first requirement of a "spatial mouse" is the ability to identify the object that it is aiming at. Among many possible technologies that can be employed for this purpose, possibly the best solution would be object recognition by machine vision. The problem, however, is that object recognition algorithms are not yet reliable enough or light enough for hand-held devices. This paper demonstrates that a simple object recognition algorithm can become a practical solution when augmented by interactivity. The user draw a circle around a target using a spatial mouse, and the mouse captures a series of camera frames. The frames can be easily stitched together to give a target image separated from the background, with which we need only additional steps of feature extraction and object classification. We present here results from two experiments with a few household objects.