Projectors and large displays are effective for presenting information to large, unspecified audiences without additional devices such as head-mounted displays. However, they are unable to deliver content to individual viewers. The ability to display varied visual content based on individual user states facilitates rapid and appropriate access to necessary information. We propose DirectionalIllusion, a novel image display technique that uses high-speed projection to present different images depending on the user’s gaze movement direction. Our method projects stripe-based frames at high refresh rates; when a user moves their gaze in a specific direction, hidden visual content becomes perceptible due to temporal interference patterns. By changing the direction of gaze, users can selectively reveal different images embedded within the same visual field, enabling direction-dependent information access. Our results demonstrated that directional gaze shifts reliably trigger the intended image perception. We also propose application scenarios for both intentional and unintentional gaze movements, demonstrating how each can be utilized in different interaction contexts.

Vocal training is difficult because the muscles that control pitch, resonance, and phonation are internal and invisible to learners. This paper investigates how Electromyography (EMG) and ultrasonic imaging (UI) can make these muscles observable for training purposes. We report three studies. First, we analyze the EMG and UI data from 16 singers (beginners, experienced & professionals), revealing differences among three vocal groups of the muscle control proficiency. Second, we use the collected data to create a system that visualizes an expert’s muscle activity as reference. This system is tested in a user study with 12 novices, showing that EMG highlighted muscle activation nuances, while UI provided insights into vocal cord length and dynamics. Third, to compare our approach to traditional methods (audio analysis and coach instructions), we conducted a focus group study with 15 experienced singers. Our results suggest that EMG is promising for improving vocal skill development and enhancing feedback systems. We conclude the paper with a detailed comparison of the analyzed modalities (EMG, UI and traditional methods), resulting in recommendations to improve vocal muscle training systems.

In sports training, individualized skill assessment and feedback are essential for athletes to master complex movements and enhance performance. Existing approaches for generating coaching comments primarily rely on externally captured pose information, which limits their applicability in outdoor sports such as skiing that involve large-scale movement. To address this challenge, we propose a method for presenting athletes’ postures and generating coaching feedback solely based on foot pressure and IMU data collected from insole sensors. In our approach, a large language model directly interprets foot pressure signals to provide actionable coaching, thereby supporting independent practice. Through model evaluation and user studies, we demonstrate that the proposed method generates expert-level feedback and outperforms pose-based approaches. Furthermore, the user study shows that the feedback helps athletes identify body parts requiring correction and enhances their motivation for training.

We present Chromotion, a high-speed projection method that renders intended colors along the motion trajectories of moving objects. When an object moves across a high speed temporally multiplexed frame sequence, its occlusion of the projected patterns can, through persistence of vision, produce motion dependent colors along its path. Chromotion exploits this phenomenon by decomposing each static image into a short sequence in which target color frames are interleaved with a single complementary color frame. This temporal design allows moving objects to sample the sequence so that the perceived color along their motion paths converges to the target color, while stationary regions still integrate to the original static color. We built a prototype and conducted a camera based technical evaluation together with a user evaluation. The results show that Chromotion reliably produces the target color on motion trajectories without degrading static color fidelity. Because the approach requires no body or gaze tracking and no decoding of embedded information, it scales to public settings and supports multiuser and multimodal interactions. We also discuss limitations, and outline application scenarios such as public, ambient displays that blend into the environment.

Precise kinematic and dynamic representations are equally critical for fine-grained motor skills such as golf, yet the latter remains underexplored. This work introduces MuscleGolfAR, an augmented reality (AR) training system integrating swing motions and muscle activities. To support cost-effective inference of muscle activations, we construct a multimodal dataset encompassing posture, electromyography (EMG), and plantar pressure. The system displays inferred EMG through embodied (first-person perspective) and detached (third-person perspective) visualizations. User studies are subsequently conducted to evaluate the impact of these strategies on training effectiveness. Results reveal that embodied visualizations enhance ownership over augmented feedback, while detached visualizations facilitate a holistic comprehension of the whole body. Furthermore, individuals' preferences for these strategies correlate with their practice habits and skill proficiencies, offering new insights for the design of AR-based motor skill training systems.

Muscle coordination is fundamental when humans interact with the world. Reliable estimation of hand muscle engagement can serve as a source of internal feedback, supporting the development of embodied intelligence and the acquisition of dexterous skills. However, contemporary electromyography (EMG) sensing techniques either require prohibitively expensive devices or are constrained to gross motor movements, which inherently involve large muscles. On the other hand, EMGs exhibit dependency on individual anatomical variability and task-specific contexts, resulting in limited generalization. In this work, we preliminarily investigate the latent pose-EMG correspondence using a general EMG gesture dataset. We further introduce a multimodal dataset, PianoKPM Dataset, and a hand muscle estimation framework, PianoKPM Net, to facilitate high-fidelity EMG inference. Subsequently, our approach is compared against reproducible competitive baselines. The generalization and adaptation across unseen users and tasks are evaluated by quantifying the training set scale and the included data amount.

We have proposed a novel image display technique that uses high-speed projection to present different images depending on the user’s gaze movement direction. This method projects stripe-based frames at high refresh rates; when a user moves their gaze in a specific direction, previously hidden visual content becomes perceptible due to temporal interference patterns. In this method, the optimal gaze velocity is determined from parameters such as the decomposition parameters of the striped patterns and the projection speed of these images. However, it is known that in the human visual system, gaze velocity lags behind the target object’s velocity during pursuit eye movements. Therefore, we conducted an experiment with human subjects to compare the theoretically optimal gaze velocity with the actual measured velocity. The results suggest that the velocity of the guide point used to induce gaze movement should be set to approximately 1.5 times the theoretical optimal gaze velocity.

Efficient space division is increasingly important as space is decreasing for both apartments and offices, where the average person spends most of their time. Therefore, it is essential to explore additional ways to separate and cover space without preventing its accessibility, while also providing possibilities for personalization by the individual. To achieve this, we present StringWall, a tangible, projection-based user interface created from low-cost materials that can be integrated into any space or size. It consists of rubber strings tracked by a depth camera that can be adapted to the user's needs in terms of size and orientation. To evaluate our system, we conducted visibility tests for the projection and background under different conditions. Furthermore, we conducted two preliminary studies to explore the users' preferences in terms of different interaction methods and use cases, as well as the haptic feedback capabilities of our system. The results show that users found the system and proposed use cases engaging, and that the system is suitable for providing extended haptic feedback when physical restraints align with the user's expectations. We further provide a demonstration of two versions of a shelf prototype incorporating the StringWall and its interactions.

Sports performance is often hindered by unintentional habits, particularly in golf, where achieving a consistent and correct swing is crucial yet challenging due to ingrained swing path habits. This study explores redirection approaches in virtual reality (VR) to correct golfers' swing paths through strategic ball shifting. By initiating a forward ball shift just before impact, we aim to prompt golfers to react and modify their swing motion, thereby eliminating undesirable swing habits. Building on recent research, our VR-based methods incorporate a gradual transformation of visuomotor associations to enhance motor skill learning. In this study, we develop three ball shift patterns, including a novel pattern that employs gradual ball shifts with interspersed normal conditions, designed to retain learning effects post-training. A preliminary study, including expert interviews, assesses the feasibility of various ball-shifting directions. Subsequently, a comprehensive user study measures the learning effects across different ball shift modes. The results indicate that our proposed redirection mode effectively corrects swing paths and yields a sustained learning effect.

The training of microsurgical suturing techniques in neurosurgery requires a considerable amount of time, and there is a need for training methods that enhance efficiency. Especially for novices, imitation of examples and receiving feedback from professionals are crucial elements in improving training efficiency. This paper proposes MR MANE, an MR application that applies the learning method of imitating examples and receiving real-time feedback to the training of microsurgical suturing techniques, aiming to enhance the efficiency of skill acquisition for novices. We conducted a user experiment involving 10 participants to verify whether imitating examples and real-time feedback enhances the efficiency of novices’ training in microsurgical suturing. The results demonstrated that imitating examples and real-time feedback effectively improves the skills of novices in a short period.

We propose a gaze-guiding method called MOSion to adjust the guiding strength reacted to observers’ motion based on a high-speed projector and the afterimage effect in the human vision system. Our method decomposes the target area into mosaic patterns to embed visual cues in the perceived images. The patterns can only direct the attention of the moving observers to the target area. The stopping observer can see the original image with little distortion because of light integration in the visual perception. The precomputation of the patterns provides the adaptive guiding effect without tracking devices and computational costs depending on the movements. The evaluation and the user study show that the mosaic decomposition enhances the perceived saliency with a few visual artifacts, especially in moving conditions. Our method embedded in white lights works in various situations such as planar posters, advertisements, and curved objects.

Woven beads, a structured fabric category, comprises interconnected rows of beads joined by fiber strands. While the stiffness of woven beads can be adjusted by relying on fiber tension during fabrication, the resulting shape and stiffness properties remain fixed. This study explores the potential of tunable shape and stiffness in woven beads, offering adaptability in comfort, functionality, and form factor. By leveraging Pneumatic Artificial Muscles (PAMs), we employ a state-of-the-art technique for dynamically modulating fabric stiffness through mechanical constraints in bead form. This approach enables a modular and scalable fabrication process, fostering programmability in mechanical properties. Our investigation encompasses diverse bead iterations and stitching patterns to broaden their applicability in fabric behavior including degree of freedom, stretchability, permeability, and textures. We evaluate the mechanical properties to differentiate design capabilities, and present techniques for locally adjusting stiffness. We showcase the versatility through applications, including variable stiffness wearables and shape-changing everyday objects.

The integration of advanced technologies in healthcare necessitates the development of systems accommodating the daily routines in medical practices. Neurosurgeons, in particular, require extensive practice in microsurgical suturing in the long term, even in the busy routine of a medical practice. This study collaboratively developed a Mixed Reality system with neurosurgeons to support self-training in microscopic suturing. Based on the neurosurgeons’ opinions, we implemented a level-appropriate microsurgical suture training system. For novices, the system offers shadow-matching training to support the practice of precise movements under the high-sensitivity environment of the microscope. For intermediates, it provides a real-time feedback system, which allows users to practice attention to details. Evaluation involved testing the novice system on students with no medical background and the intermediate system on neurosurgery residents. The effectiveness of the system was demonstrated through the experimental results and subsequent discussion.

We propose SolePoser, a real-time 3D pose estimation system that leverages only a single pair of insole sensors. Unlike conventional methods relying on fixed cameras or bulky wearable sensors, our approach offers minimal and natural setup requirements. The proposed system utilizes pressure and IMU sensors embedded in insoles to capture the body weight’s pressure distribution at the feet and its 6 DoF acceleration. This information is used to estimate the 3D full-body joint position by a two-stream transformer network. A novel double-cycle consistency loss and a cross-attention module are further introduced to learn the relationship between 3D foot positions and their pressure distributions. We also introduced two different datasets of sports and daily exercises, offering 908k frames across eight different activities. Our experiments show that our method’s performance is on par with top-performing approaches, which utilize more IMUs and even outperform third-person-view camera-based methods in certain scenarios.

Alpine skiing is a popular winter sport, and several systems have been proposed to enhance training and improve efficiency. However, many existing systems rely on simulation-based environments, which suffer from drawbacks such as a gap between real skiing and the lack of body ownership. To address these limitations, we present ARpenSki, a novel augmented reality (AR) ski training system that employs a see-through head mounted display (HMD) to deliver augmented visual training cues that may be applied on real slopes. The proposed AR system provides a transparent view of the lower half of the field of vision, where we implemented three different AR-based direct and indirect postural visualization methods. We conducted an user study to investigate the influence of different visual cues in the AR environment. Our results indicate that a simple AR visualization of the user’s spine (Figure 1.2) yields the most favorable training performance, surpassing conventional visualizations by 7% improvement in the user’s posture. Building upon these promising findings, we further tested our system on real slopes and showed the potential of a real AR skiing application.

An omni-directional (360°) camera captures the entire viewing sphere surrounding its optical center. Such cameras are growing in use to create highly immersive content and viewing experiences. When such a camera is held by a user, the view includes the user’s hand grip, finger, body pose, face, and the surrounding environment, providing a complete understanding of the visual world and context around it. This capability opens up numerous possibilities for rich mobile input sensing. In OmniSense, we explore the broad input design space for mobile devices with a built-in omni-directional camera and broadly categorize them into three sensing pillars: i) near device ii) around device and iii) surrounding device. In addition we explore potential use cases and applications that leverage these sensing capabilities to solve user needs. Following this, we develop a working system to put these concepts into action, by leveraging these sensing capabilities to enable potential use cases and applications. We studied the system in a technical evaluation and a preliminary user study to gain initial feedback and insights. Collectively these techniques illustrate how a single, omni-purpose sensor on a mobile device affords many compelling ways to enable expressive input, while also affording a broad range of novel applications that improve user experience during mobile interaction.

Motor skill acquisition involves learning from spatiotemporal discrepancies between target and self-generated motions. However, in dexterous skills with numerous degrees of freedom, understanding and correcting these motor errors are challenging. This issue becomes crucial for experienced individuals who seek for mastering and sophisticating their skills, where even subtle errors need to be minimized. To enable efficient optimization of body posture in piano learning, we present PianoSyncAR, an augmented reality system that superimposes the time-varying complex hand postures of a teacher over the hand of a learner. Through a user study with 12 pianists, we demonstrate several advantages of the proposed system over conventional tablet-screen, which implicate the potential of AR training as a complementary tool for video-based skill learning in piano playing.

While language-guided image manipulation has made remarkable progress, the challenge of how to instruct the manipulation process faithfully reflecting human intentions persists. An accurate and comprehensive description of a manipulation task using natural language is laborious and sometimes even impossible, primarily due to the inherent uncertainty and ambiguity present in linguistic expressions. Is it feasible to accomplish image manipulation without resorting to external cross-modal language information? If this possibility exists, the inherent modality gap would be effortlessly eliminated. In this paper, we propose a novel manipulation methodology, dubbed ImageBrush, that learns visual instructions for more accurate image editing. Our key idea is to employ a pair of transformation images as visual instructions, which not only precisely captures human intention but also facilitates accessibility in real-world scenarios. Capturing visual instructions is particularly challenging because it involves extracting the underlying intentions solely from visual demonstrations and then applying this operation to a new image. To address this challenge, we formulate visual instruction learning as a diffusion-based inpainting problem, where the contextual information is fully exploited through an iterative process of generation. A visual prompting encoder is carefully devised to enhance the model's capacity in uncovering human intent behind the visual instructions. Extensive experiments show that our method generates engaging manipulation results conforming to the transformations entailed in demonstrations. Moreover, our model exhibits robust generalization capabilities on various downstream tasks such as pose transfer, image translation and video inpainting.

To adopt the soft hand exoskeleton to support activities of daily livings, it is necessary to control finger joints precisely with the exoskeleton. The problem of controlling joints to follow a given trajectory is called the tracking control problem. In this study, we focus on the tracking control problem of a human finger attached with thin McKibben muscles. To achieve precise control with thin McKibben muscles, there are two problems: one is the complex characteristics of the muscles, for example, non-linearity, hysteresis, uncertainties in the real world, and the other is the difficulty in accessing a precise model of the muscles and human fingers. To solve these problems, we adopted DreamerV2, which is a model-based reinforcement learning method, but the target trajectory cannot be generated by the learned model. Therefore, we propose Tracker, which is an extension of DreamerV2 for the tracking control problem. In the experiment, we showed that Tracker can achieve an approximately 81% smaller error than PID for the control of a two-link manipulator that imitates a part of human index finger from the metacarpal bone to the proximal bone. Tracker achieved the control of the third joint of the human index finger with a small error by being trained for approximately 60 minutes. In addition, it took approximately 15 minutes, which is less than the time required for the first training, to achieve almost the same accuracy by fine-tuning the policy pre-trained by the user’s finger after taking off and attaching thin McKibben muscles again as the accuracy before taking off.

We address the problem of face modeling, which is still challenging in achieving high-quality reconstruction results efficiently. Neither previous regression-based nor optimization-based frameworks could well balance between the facial reconstruction fidelity and efficiency. We notice that the large amount of in-the-wild facial images contain diverse appearance information, however, their underlying knowledge is not fully exploited for face modeling. To this end, we propose our Reconstruction by Encoding Facial Priors (ReEnFP) pipeline to exploit the potential of unconstrained facial images for further improvement. Our key is to encode generative priors learned by a style-based texture generator on unconstrained data for fast and detail-preserving face reconstruction. With our texture generator pre-trained using a differentiable renderer, faces could be encoded to its latent space as opposed to the time-consuming optimization-based inversion. Our generative prior encoding is further enhanced with a pyramid fusion block for adaptive integration of input spatial information. Extensive experiments show that our method reconstructs photo-realistic facial textures and geometric details with precise identity recovery.

Hand pose analysis is a key step to understanding dexterous hand performances of many high-level skills, such as playing the piano. Currently, most accurate hand tracking systems are using fabric-/marker-based sensing that potentially disturbs users’ performance. On the other hand, markerless computer vision-based methods rely on a precise bare-hand dataset for training, which is difficult to obtain. In this paper, we collect a large-scale high precision 3D hand pose dataset with a small workload using a marker-removal network (MR-Net). The proposed MR-Net translates the marked-hand images to realistic bare-hand images, and the corresponding 3D postures are captured by a motion capture thus few manual annotations are required. A baseline estimation network PiaNet is introduced and we report the accuracy of various metrics together with a blind qualitative test to show the practical effect.

This article examines the attentional mechanism of in-person collaboration by means of System Dynamics-based simulations using an eye tracking experiment. Three experimental conditions were tested: in-person collaboration, remote collaboration, and single user. We hypothesized that collaboration focuses users’ attention on key information facilitating decision-making. Collaborating participants dwelt longer on key elements of the simulation than single users. Moreover, in-person collaboration and single users yielded a strategy of decision-making similar to an optimal strategy. Finally, in-person collaboration was less cognitively demanding and of higher quality. The contribution of this article is a deeper understanding of how in-person collaboration on a large display can help users focus their visual attention on the most important areas. With this novel understanding, we believe collaborative systems designers will be better equipped to design more effective attention-guiding mechanisms in remote collaboration systems. The present work has the potential to advance the study of collaborative, interactive technologies.

Projector-camera (ProCam) systems have been widely used to develop the interactive spatial augmented reality (SAR) environment. However, since the projection angle of the existing ProCams is narrow, it is necessary to deploy multiple ProCams to make an omnidirectional SAR environment. These ProCams need to be calibrated and cannot be moved after calibration. To solve this limitation, we introduce OmniProCam, which is a portable coaxial ProCam combining with an ultra-fisheye lens. The ultra-fisheye lens enables omnidirectional projection and interaction, and the coaxial ProCam enables calibration-free. In order to prove the availability of the OmniProcam system, we develop a prototype named OmniLantern, a portable and omnidirectional device in the shape of Lantern. In this paper, we describe the implementation details of the OmniLantern prototype that bases on the retroreflective markers, distortion correction, gesture recognition, and touch detection. We also introduce some applications which show the advantages of OmniLantern. Finally, we summarize the issue of the OmniLantern and the future potential of the OmniProCam system.

Previous studies have explored generating accurately lip-synced talking faces for arbitrary targets given audio conditions. However, most of them deform or generate the whole facial area, leading to non-realistic results. In this work, we delve into the formulation of altering only the mouth shapes of the target person. This requires masking a large percentage of the original image and seamlessly inpainting it with the aid of audio and reference frames. To this end, we propose the Audio-Visual Context-Aware Transformer (AV-CAT) framework, which produces accurate lip-sync with photo-realistic quality by predicting the masked mouth shapes. Our key insight is to exploit desired contextual information provided in audio and visual modalities thoroughly with delicately designed Transformers. Specifically, we propose a convolution-Transformer hybrid backbone and design an attention-based fusion strategy for filling the masked parts. It uniformly attends to the textural information on the unmasked regions and the reference frame. Then the semantic audio information is involved in enhancing the self-attention computation. Additionally, a refinement network with audio injection improves both image and lip-sync quality. Extensive experiments validate that our model can generate high-fidelity lip-synced results for arbitrary subjects.

Distance learning for skill learning is still inadequate because the perceptual information provided to the user is limited. This study proposed a framework for a distance learning system with real-time feedback to share physical information between the student and the teacher. As an initial trial to use this framework, we developed a prototype of a distance learning system for skiing including visual feedback, and verified its operation. The result indicated the system could be applied enough to a distance learning system.

What can we picture solely from a clip of speech? Previous research has shown the possibility of directly inferring the appearance of a person's face by listening to a voice. However, within human speech lies not only the biometric identity signal but also the identity-irrelevant information such as the talking content. Our goal is to extract as much information from a clip of speech as possible. In particular, we aim at not only inferring the face of a person but also animating it. Our key insight is to synchronize audio and visual representations from two perspectives in a style-based generative framework. Specifically, contrastive learning is leveraged to map both the identity and speech content information within the speech to visual representation spaces. Furthermore, the identity space is strengthened with class centroids. Through curriculum learning, the style-based generator is capable of automatically balancing the information from the two latent spaces. Extensive experiments show that our approach encourages better speech-identity correlation learning while generating vivid faces whose identities are consistent with given speech samples. Moreover, by leveraging the same model, these inferred faces can be driven to talk by the audio.

Learning an advanced skill in sports requires a huge amount of practice and players also have to overcome both physical difficulties and the dullness of repetitive training. Returning a fast spin shot in table tennis could be taken as an example, as athletes need to judge the spin type and decide the racket pose within a second, which is difficult for beginners. Therefore, in this paper, we show how to design an intuitive training system to acquire this specific skill using different cues in Virtual Reality (VR). Using VR, we can easily provide visual information, attach haptic devices, and distort the speed of time, however, it is difficult to decide which types of information could benefit the training. In an initial study, by comparing real world training with VR training, we showed the effect of VR training and obtained some insights about augmentation for training spin shots. The training system was then improved by adding three new conditions using different visualizations and temporal distortions, as well as a haptic racket for creating realistic feedback. Finally, we performed a detailed experiment, which suggest a significant improvement of skill for each condition compared to the baseline, while a qualitative evaluation indicates that both users' motivation and their understanding of spin are increased by using our system.

In this paper, we present a novel collaboration tool, OmniGlobeVR, which is an asymmetric system that supports communication and collaboration between a VR user (occupant) and multiple non-VR users (designers) across the virtual and physical platform. OmniGlobeVR allows designer(s) to explore the VR space from any point of view using two view modes: a 360° first-person mode and a third-person mode. In addition, a shared gaze awareness cue is provided to further enhance communication between the occupant and the designer(s). Finally, the system has a face window feature that allows designer(s) to share their facial expressions and upper body view with the occupant for exchanging and expressing information using nonverbal cues. We conducted a user study to evaluate the OmniGlobeVR, comparing three conditions: (1) first-person mode with the face window, (2) first-person mode with a solid window, and (3) third-person mode with the face window. We found that the first-person mode with the face window required significantly less mental effort, and provided better spatial presence, usability, and understanding of the partner's focus. We discuss the design implications of these results and directions for future research.

Allowing projections on moving objects is associated with a problem that a projection might shift due to the delay between tracking and projection. In the present paper, we proposed a new prediction model based on deep neural networks that can be used to predict both pose and position of the target object. As a result, we developed a real-time tracking and projection system named"MirAIProjection that employs motion-capture cameras and common projectors. We conducted several experiments to evaluate the effectiveness of the proposed system and demonstrated that the proposed system could reduce the slipping and increase the accuracy and robustness of the projection.

Introducing interactive components into furniture has proven difficult due to the different lifespans of furniture and digital devices. We present Foxels, a modular, smart furniture concept that allows users to create their own interactive furniture on demand by simply snapping together individual building blocks. The modular design makes the system flexible to accommodate a variety of interactive furniture setups, making it particularly well-suited for re-configurable spaces. Considering the trade-off between ease-of-use and high versatility, we explored a number of interaction methods that can be applied to modular interactive furniture, thereby extending the well-known tangible programming paradigm. After explaining our implementation, we demonstrate the validity of the proposed concepts by presenting how Foxels can be used in an ideation workshop along with many additional real-world examples.

We present MonoEye, a multimodal human motion capture system using a single RGB camera with an ultra-wide fisheye lens, mounted on the user's chest. Existing optical motion capture systems use multiple cameras, which are synchronized and require camera calibration. These systems also have usability constraints that limit the user's movement and operating space. Since the MonoEye system is based on a wearable single RGB camera, the wearer's 3D body pose can be captured without space and environment limitations. The body pose, captured with our system, is aware of the camera orientation and therefore it is possible to recognize various motions that existing egocentric motion capture systems cannot recognize. Furthermore, the proposed system captures not only the wearer's body motion but also their viewport using the head pose estimation and an ultra-wide image. To implement robust multimodal motion capture, we design three deep neural networks: BodyPoseNet, HeadPoseNet, and CameraPoseNet, that estimate 3D body pose, head pose, and camera pose in real-time, respectively. We train these networks with our new extensive synthetic dataset providing 680K frames of renderings of people with a wide range of body shapes, clothing, actions, backgrounds, and lighting conditions. To demonstrate the interactive potential of the MonoEye system, we present several application examples from common body gestural to context-aware interactions.

The automatic recognition of how people use their hands and fingers in natural settings -- without instrumenting the fingers -- can be useful for many mobile computing applications. To achieve such an interface, we propose a vision-based 3D hand pose estimation framework using a wrist-worn camera. The main challenge is the oblique angle of the wrist-worn camera, which makes the fingers scarcely visible. To address this, a special network that observes deformations on the back of the hand is required. We introduce DorsalNet, a two-stream convolutional neural network to regress finger joint angles from spatio-temporal features of the dorsal hand region (the movement of bones, muscle, and tendons). This work is the first vision-based real-time 3D hand pose estimator using visual features from the dorsal hand region. Our system achieves a mean joint-angle error of 8.81 degree for user-specific models and 9.77 degree for a general model. Further evaluation shows that our system outperforms previous work with an average of 20% higher accuracy in recognizing dynamic gestures, and achieves a 75% accuracy of detecting 11 different grasp types. We also demonstrate 3 applications which employ our system as a control device, an input device, and a grasped object recognizer.

We present MoVNect, a lightweight deep neural network to capture 3D human pose using a single RGB camera. To improve the overall performance of the model, we apply the teacher-student learning method based knowledge distillation to 3D human pose estimation. Real-time post-processing makes the CNN output yield temporally stable 3D skeletal information, which can be used in applications directly. We implement a 3D avatar application running on mobile in real-time to demonstrate that our network achieves both high accuracy and fast inference time. Extensive evaluations show the advantages of our lightweight model with the proposed training method over previous 3D pose estimation methods on the Human3.6M dataset and mobile devices.

Video communication systems have been suffered from the narrow field of view. To solve this limitation, one study proposed symmetric 360° video communication system by combining an omnidirectional camera and a hemispherical display. However, the system still had several issues, e.g., the invisibility of hemisphere which was at the opposite side from a user caused the inconvenience of observing the remote environment. To solve these issues, we introduce OmniGlobe, a novel symmetric full 360° video communication system which incorporates an omnidirectional camera, a full spherical display, and several visual or interactive techniques. Based on an experiment, we could indicate that our system is effective in reducing the inconvenience of observing the remote environment and increased the remote space awareness and user's gaze awareness to support remote collaboration. We also discuss the takeaways, limitations and application areas in our system which help improve the system.

To better explore the incorporation of pointing and gesturing into ubiquitous computing, we introduce WRIST, an interaction and sensing technique that leverages the dexterity of human wrist motion. WRIST employs a sensor fusion approach which combines inertial measurement unit (IMU) data from a smartwatch and a smart ring. The relative orientation difference of the two devices is measured as the wrist rotation that is independent from arm rotation, which is also position and orientation invariant. Employing our test hardware, we demonstrate that WRIST affords and enables a number of novel yet simplistic interaction techniques, such as (i) macro-micro pointing without explicit mode switching and (ii) wrist gesture recognition when the hand is held in different orientations (e.g., raised or lowered). We report on two studies to evaluate the proposed techniques and we present a set of applications that demonstrate the benefits of WRIST. We conclude with a discussion of the limitations and highlight possible future pathways for research in pointing and gesturing with wearable devices.

We introduce a vision-based technique to recognize static hand poses and dynamic finger tapping gestures. Our approach employs a camera on the wrist, with a view of the opisthenar (back of the hand) area. We envisage such cameras being included in a wrist-worn device such as a smartwatch, fitness tracker or wristband. Indeed, selected off-the-shelf smartwatches now incorporate a built-in camera on the side for photography purposes. However, in this configuration, the fingers are occluded from the view of the camera. The oblique angle and placement of the camera make typical vision-based techniques difficult to adopt. Our alternative approach observes small movements and changes in the shape, tendons, skin and bones on the opisthenar area. We train deep neural networks to recognize both hand poses and dynamic finger tapping gestures. While this is a challenging configuration for sensing, we tested the recognition with a real-time user test and achieved a high recognition rate of 89.4% (static poses) and 67.5% (dynamic gestures). Our results further demonstrate that our approach can generalize across sessions and to new users. Namely, users can remove and replace the wrist-worn device while new users can employ a previously trained system, to a certain degree. We conclude by demonstrating three applications and suggest future avenues of work based on sensing the back of the hand.

We generate synthetic annotated data for learning 3D human pose estimation using an egocentric fisheye camera. Synthetic humans are rendered from a virtual fisheye camera, with a random background, random clothing, random lighting parameters. In addition to RGB images, we generate ground truth of 2D/3D poses and location heat-maps. Capturing huge and various images and labeling manually for learning are not required. This approach will be used for the challenging situation such as capturing training data in sports.

Wearable cameras have the potential to be used in various ways in combination with egocentric views such as action recognition, gesture input method for augmented/virtual reality (AR/VR) as well as lifelogger. Particularly, the pose of the camera wearer is one of the interesting factors of the egocentric view and various eccentric view-based pose estimation systems have been proposed; however, there is no balance between recognizable poses and enough egocentric views. In this work, we propose MonoEye, a system to provide wearer's estimated 3D pose and wide egocentric view. Our system's chest-mounted camera, equipped with the ultra-wide fisheye lens, covers the wearer's limbs and wide egocentric view; our pose estimation network estimates 3D body pose of the wearer from the camera's egocentric view. The proposed system not only can be used as an input interface of AR and VR through estimation of a various pose of the wearer but also has a potential to be used for action recognition by providing a wide egocentric view.

We are interested in utilizing egocentric view from a wearable camera and are working on MonoEye system, a novel system to estimate the wearer's motion using a chest-mounted camera equipped with an ultra-wide fisheye lens. Because our system has a wide field of view, it provides a balanced capacity of recognizable pose types and broad egocentric view. The prototype deep neural network estimates camera wearer's 3D pose and acquires motion without complex configuration like conventional motion capture systems.

In this work, we propose a real-time golf learning system using visual feedback. The system projects the visual feedback which is imitating shadow on the ground in front of the learner. It makes it possible for the learner to receive the feedback without the posture collapse. The learner can receive the feedback with the fixed face orientation as the actual golf swing, which could not be solved in existing systems projecting feedbacks on the wall. Furthermore, a posture difference is provided by projecting the expert's outline on the learner's virtual shadow. In addition, because shadow was also used in golf practice conventionally, the cost of adaptation to this learning system which uses virtual shadow as visual feedback may be low.

The training of skiing is difficult because of environmental requirements and teaching methods. Therefore, we propose a virtual reality ski training system using an indoor ski simulator. The system is based on a simple indoor ski simulator with two trackers to capture the motion of skis. Users can control the skis in the virtual ski slope we provided and train their skills with a replay of a professional skier. The training system consists of three modules: a coach replay system for reviewing pro-skiers's motion; a time control system that can be used to watch the detailed motion of both the coach and the user; and a visualization of the angle of the skis to compare the difference of motions between the users and the coach.

We introduce a novel soft exoskeleton glove, which is capable of generating human-like finger joint movements with little constraints on volitional motions. Four pneumatic artificial muscles (approx. 2.5 mm in diameter weight and less than 2 g) were attached to each finger, which consists of two antagonistic pairs of the muscles (i.e. flexor and extensor) and thereby enables to control different joints of each finger independently. Implementation of this structure for all five digits resulted in the hand exoskeleton with 20 DOFs for one hand. This architecture was designed similar to the human anatomy of the forearm muscle, which eventually ensured supporting the natural, unconstrained hand motion. Furthermore, as functions of our system, we showed that it is possible to generate a pressing force of approx. 10 N and to manipulate a finger to perform high-speed tapping at approx. 10 Hz. These results suggest that the system can be used not only as a rehabilitation device but also as an instruction device for playing an instrument or sports, which requires dexterous motion control and quick performance.

This paper propose a real-time human motion forecasting system which visualize the future pose in virtual reality using a RGB camera. Our system consists of three parts: 2D pose estimation from RGB frames using a residual neural network, 2D pose forecasting using a recurrent neural network, and 3D recovery from the predicted 2D pose using a residual linear network. To improve the prediction learning quantity of temporal feature, we propose a special method using lattice optical flow for the joints movement estimation. After fitting the skeleton, a predicted 3d model of target human will be built 0.5s in advance in a 30-fps video.

Novices can learn sports in a variety of ways ranging from guidance from an instructor to watching video tutorials. In each case, subsequent and repeated self-directed practice sessions are an essential step. However, during such self-directed practice, constant guidance and feedback is absent. As a result, the novices do not know if they are making mistake or if there are any areas for improvement. In this position paper, we propose using wearable devices to augment such self-directed practice sessions by providing augmented guidance and feedback. In particular, a head-worn display can provide real-time guidance whilst wrist-worn devices can provide real-time tracking and monitoring of various states. We envision this approach being applied to various sports, and in particular this is suitable for sports that utilize precise hand motion such as snooker, billiards, golf, archery, cricket, tennis and table tennis.

In this paper, we propose a novel mixed reality martial arts training system using deep learning based real-time human pose forecasting. Our training system is based on 3D pose estimation using a residual neural network with input from a RGB camera, which captures the motion of a trainer. The student wearing a head mounted display can see the virtual model of the trainer and his forecasted future pose. The pose forecasting is based on recurrent networks, to improve the learning quantity of the motion's temporal feature, we use a special lattice optical flow method for the joints movement estimation. We visualize the real-time human motion by a generated human model while the forecasted pose is shown by a red skeleton model. In our experiments, we evaluated the performance of our system when predicting 15 frames ahead in a 30-fps video (0.5s forecasting), the accuracies were acceptable since they are equal to or even outperforms some methods using depth IR cameras or fabric technologies, user studies showed that our system is helpful for beginners to understand martial arts and the usability is comfortable since the motions were captured by RGB camera.

Field-of-view limitation has been a long-standing issue in video communication systems. With the advancement of omnidirectional panoramic technology, the omnidirectional camera, which can provide a 360° field of view, has become increasingly popular in the last few years. Previous research indicated that one-way video communication systems with a wider field of view improve task efficiency. Therefore, we propose to utilize omnidirectional cameras in a symmetrical video communication system and study how this configuration affects remote collaboration. In this study, we conducted experiments based on two conditions, which are an omnidirectional camera with a spherical display and an omnidirectional camera with a horizontally placed 2D flat display. Under these conditions, we analyzed how the display types affected remote collaboration. Our results show that participants marginally preferred the spherical display to the 2D flat display. We also show the advantages and disadvantages of each display. The findings contribute to our understanding of how to design an environment for remote collaboration that captures and shows a 360° panoramic view of a remote site.

In Web or digital signage, content providers want to guide users' attention to the intended regions. Using active visual stimuli, such as animated or flashing objects, is effective for gaze guidance; however, it has been reported that such an approach often results in unpleasant feelings for users. This paper proposed a new method for gaze guidance using artificial color shifts that does not induce unpleasant feelings in the user. We created an image filter that separates the image in three layers, i.e., cyan, magenta, and yellow, and slightly shifted each layer the left, right, and down, respectively. The filter was applied to the entire image except the region where the user's gaze was to be guided. We conducted experiments using a gaze tracker. The experimental results showed that the proposed method can guide the user's gaze to a particular region with less unpleasant feelings.

Recently, large horizontal interactive surfaces have begun to be developed. In these systems, an overhead camera is often used to detect the position of objects on the surface even if they are not in contact with the surface. However the issues that these systems face are that they are expensive and a camera cannot be easily attached on top of the surface in some situations. This paper proposes a framework that uses a camera on a drone (UAV) as an overhead camera unit to convert arbitrary horizontal rectangular regions into interactive surfaces. Although commercially available drones that are equipped with cameras have high latencies and are difficult to use in real-time interactive systems, we solved this latency issue using a small PC that performs primitive image processing tasks onboard. First, we describe a drone unit that has an infrared camera and a small PC for real-time image processing, such as surface detection and object detection. Second, we describe novel infrared markers for the robust detection of the four corners of a rectangular region and the objects within that region. Finally, we describe an interactive sports coaching application in which a drone unit is used as an overhead camera both for a large playing field and small tabletop.

In this work, we propose a golf training system using real-time visual feedback. The system projects the virtual shadow of the user on the ground in front of the user; this shadow provides feedback to the user without form collapse. Additionally, an expert's contour is overlaid on the virtual shadow of the user to make them aware of the difference between their form and that of the expert. In this system, the user can receive feedback with a fixed face orientation according to the actual golf swing; such feedback cannot be realized in systems projecting feedbacks on a wall. Moreover, by imitating the shadows already used in conventional golf training, the cost of adaptation to this training system can be reduced.

We propose OmniEyeball (OEB), which is a novel interactive 360° image I/O system combining a spherical display system with an omnidirectional camera. We also present our experimental design of a user interface on the OEB, including a vision-based touch detection technique as well as several visual and interactive features. Our proposed techniques may contribute to solving the weak awareness of the opposite side of the spherical display as well as the workload caused by walking around in the 360° symmetric video communication.

We propose an omnidirectional projection system embedding a projector with an ultra wide-angle lens in the table. AR markers are attached on the target surfaces so that the system can track them with an omnidirectional camera. Finally, we developed a prototype of the system and introduced some applications to show the effectiveness of our method.

We propose MlioLight, a projector-camera unit-based projection-mapping system for overlaying multiple images on a screen or on real world objects using multiple flashlight-type devices. We focus on detecting the areas of overlapping lights in a multiple light source scenario and overlaying multi-layered information on real world objects in these areas. To blend multiple images, we developed methods for light identification and overlapping area detection using wireless synchronization between a high-speed camera and multiple flashlight devices. In this study, we describe the concept of MlioLight as well as its prototype implementation and applications. In addition, we present an evaluation of the proposed prototype.

Recently, there has been a renewed interest in Fluidized Bed Interface. It can give us the haptic feedback of sand and fluid reversibly, but exhibits difficulty in visually changes. So, we proposed a novel spatial augmented reality technique to provide the visual liquidness impression. Our system gives visual feedback to users by projecting liquid images on the surface of the interface. The image is generated on the basis of human visual cognition and is different from a mere fluid simulation. As the result, the visual liquidness of our system was emphasized compared to the Fluidized Bed Interface. In addition, the effect of emphasizing visual liquidness is confirmed in the fluidized bed phenomenon, and it is found out that our system is superior in displaying a liquid image. Our system is expected not only to be applied in entertainments but also to contribute to the elucidation of human liquid cognitive mechanisms.

This paper proposes a method for gaze navigation in the real world by projecting an image onto a real object and changing its appearance. In the proposed method, a camera captures an image of objects in the real world. Next all the pixels in the image but those in a specified region are slightly shifted to left and right. Then the obtained image is projected onto the original objects. As a result, the objects not in the specified region looks blurred. We conducted user experiments and showed that the users' gaze were navigated to the specified region.

This paper describes image processing that can be used to develop immersive 360-degree videos using capsule endoscopy procedures. When viewed through a head-mounted display (HMD), doctors are able to inspect the human gastrointestinal tract as if they were inside the patient's body. Although the endoscopy capsule has two tiny fisheye cameras, the images captured by these cameras cannot be converted to equirectangular images which is the basic format used to produce 360-degree videos. This study proposes a method to generate a pseudo-omnidirectional video from the original images and stabilizes the video to prevent virtual reality (VR) sickness.

There are real-time training systems to learn the correct golf swing form by providing visual feedback to the users. However, real-time visual feedback requires the users to see the display during their motion that leads to the wrong posture. This paper proposed a real-time golf-swing training system using sonification and sound image localization. The system provides real-time audio feedback based on the difference between the pre-recorded model data and real-time user data, which consists of the roll, pitch, and yaw angles of a golf club shaft. The system also used sound image localization so that the user can hear the audio feedback from the direction of the club head. The user can recognize the current posture of the club without moving their gaze.

Balloonygen, an extended tabletop display embedded with a balloon-like deformable spherical screen, is a display that can seamlessly expose a spherical screen for three-dimensional contents, such as omnidirectional images, in a conventional flat display. By continuously morphing between a two-dimensional shape called tabletop and a three-dimensional shape called sphere, we render the benefits of a flat display and a spherical display to coexist and propose a smoother approach for information sharing. Balloonygen dynamically provides an optimal way to display the contents by inflating the rubber membrane installed at the center of a tabletop display and morphing between the two- and three-dimensional shapes. In this study, by prototyping and designing the application scenario, we discuss the advantages and disadvantages of this display and possible interactions involved.

We propose a real-time human motion forecasting system which visualize the future pose in virtual reality using a RGB camera. Our system consists of three parts: 2D pose estimation from RGB frames using a residual neural network, 2D pose forecasting using a recurrent neural network, and 3D recovery from the predicted 2D pose using a residual linear network. To improve the prediction learning quantity of temporal feature, we propose a special method using lattice optical flow for the joints movement estimation. After fitting the skeleton, a predicted 3d model of target human will be built 0.5s in advance in a 30-fps video.

Active involvement of users with disabilities is difficult to employ during the iterative stages of the design process due to high costs and effort associated with user studies. This research proposes a user centered design (UCD) strategy to incorporate the use of gaze-contingent tunnel vision simulation with sighted individuals to facilitate rapid prototyping of accessible interfaces. Through three types of validation studies, we examined how our simulation techniques can provide the opportunity for continued evaluation and refinement of the design. Our simulation approach was effective in emulating scanning behaviors caused by tunnel vision along with grasping user feedback to recognize user interface and usability criteria early in the design cycle.

“SHIN-TAI” is an upper humanoid body composed of artificial muscles and artificial fats covered with artificial skin. The artificial muscles reproduce natural changes in shape and stiffness of real human muscles by a pressure-regulating system, and the amount of artificial fats can be increased or decreased by controlling liquid flow rate into liquid bags under the skin. By controlling these two programmable functions, SHIN-TAI can assume various body types, namely, “slim,” “muscular,” “voluptuous,” or “overweight.” Even its gender can be changed by region-selective control of the fats. It is demonstrated that a controllable character of a humanoid can be expressed by using our method instead of conversations or body gestures, which depend on a static human-like housing as in recent humanoid implementations.

We present a sphere-shaped interactive display system, named Qoom, as a new input and output device. Unlike existing sphere-shaped displays, Qoom is a perfectly spherical ball that can be rotated, thrown, or even kicked. First, we discuss how spherical displays can be used in daily life and describe how users interact with spheres. Then, we show how we developed the Qoom prototype that uses touch and rotation detection, real-time object tracking, and spherical projection mapping. We implemented actions including touching, rotating, bouncing and throwing as controls. We also developed applications for Qoom that utilize the unique advantages of ball displays.

In information media such as TV programs, digital signage, or web pages, information content providers often want to guide viewers' attention to a particular location of the display. However, "active" methods, such as flashing displays, using animation, or changing colors, often interrupt viewers' concentration and makes viewers feel annoyed. This paper proposes a method for guiding viewers' attention without viewers noticing. By focusing on a characteristic of the human visual system, we propose a dynamic blur control method. Our method gradually blurs the image on the display to the threshold at which viewers are aware of the modulation of the display, while the region where viewers' attention should be guided remains unblurred. Two subjective experiments were conducted to show the effectiveness of our method. In the first, viewers' attention was guided to the unblurred region using blur control. In the second, a threshold was found at which viewers were aware of the modulation, and viewers' gaze is guided below this threshold. This means that the viewers' attention can be guided without them noticing.

Projections on moving objects have a problem in that the projection may slip because of the delay between tracking and projection. Here, we propose a new prediction method combining a Kalman filter and a three-frame feedback model that switches between these models according to the ball's state of motion. We developed a real-time tracking and projection system named "Ballumiere", which uses motion capture cameras for tracking and multiple projectors for spherical projection. We conducted a comparative experiment with an existing prediction model and showed that our method minimizes slipping and increases the accuracy of the projection.

This paper presents an unconstrained gaze estimation method using an online learning algorithm. We focus on a desktop scenario, where a user operates a personal computer, and use the mouse-clicked positions to infer, where on the screen the user is looking at. Our method continuously captures the user's head pose and eye images with a monocular camera, and each mouse click triggers learning sample acquisition. In order to handle head pose variations, the samples are adaptively clustered according to the estimated head pose. Then, local reconstruction-based gaze estimation models are incrementally updated in each cluster. We conducted a prototype evaluation in real-world environments, and our method achieved an estimation accuracy of 2.9°.

In the field of interface surface research, the idea of the 'softness' of a surface medium is one significant factor in determining a suitable means of interaction with the user. With direct touch input, for example, the degree of surface softness allows for the generation various touch sensations and tactile feedback. Additionally, the softness also affects the shape of the surface: a soft surface will allow the user to deform the surface at will while a hard surface will maintain its shape easier. In many traditional flexible surfaces to date, this element has been considered static and thus unchangeable. This project, in contrast, considers the softness of a surface to be dynamic and thus further explores the interaction possibilities with this type of surface. We demonstrate the possibilities of dynamically changing surfaces and their derived user interaction.

In this paper, we propose a novel interface design for "human-robot" communication by focusing on visual and tactual transformation of the muscles. Since recent humanoids may appear as humanoid figures using human-like body gestures and behavior, it is hard to say that they have enough elements to cover the complex composition that is a human. The muscles that constitute the human body work by not only turning joints and generating limb and body movements, but also control skin surface shape and firmness, allowing the various levels of touch response. Therefore, we attempt to approach the creation of sensitive and subtle expression by a humanoid robot using organic constructs. In this project, we produce "SHIRI", which represents emotions with organic movements of the Gluteus Maximus Actuator (GMA). In addition, we also implement user interaction for SHIRI and consider how perceptions the user can obtain by communicating with SHIRI.

There are many kinds of three dimensional displays that have been developed to date. Most of them provide 3D visual sensation to the users, but they do not provide 3D haptic feedback. On the other hand, an interactive surface system using transparent gels enables users to touch a 3D surface. The main issue of the system, however, is that the image is distorted due to "lens effect" of the gels. This paper describes a method to solve the image distortion through the use of a light field display (LFD) which is discussed in detail in section 2. By combining the LFD and transparent gel interface, it becomes possible to show correct 3D images on gels from any viewing position.

We present our new interaction technique for tabletop system and video game application using it. This technique is for recognizing a pinching gesture performed with the thumb and forefinger and tapping gestures with them by a ceiling camera above the table. The positions and orientations of multiple gestures are recognized at 200Hz, and the users around the table are tracked by their arm positions. This technique is useful for large-sized tabletop system for multiple users because it does not require any additional equipments to them nor registration process before using the system. Some pilot studies demonstrated the robustness and accuracy of the proposed technique with displaying appropriate guide on the display. We also developed a video game application called "Pac-pac". A player can shoot a bullet along the orientation of his hand by tapping, from any side of the table. We exhibited Pac-pac several time and well-received.

In this paper we proposed a new interface for interacting with large displays via small video devices such as a cell phone. We estimate the location of the camera relative to the display using a matrix of transparent markers embedded on display. As a result, our interface allows the user to interact with digital contents without being distracted by opaque visual markers. Our interface enables intuitive interactions such as pointing, rotating, dragging and dropping. Moreover, our use of a small hand-held camera device allows for interaction with large scale displays without the need for direct contact with the display surface. Thus our system is well suited for interactions when there is some distance between the user and the display. Our proposed system has applications to large scale advertisement displays and can enable interactions between individuals and large scale digital content.

The goal of super-resolution (SR) techniques is to enhance the resolution of low-resolution (LR) images. How to evaluate the performance of an SR algorithm seems to be forgotten when researchers keep producing algorithms. This paper presents a task-oriented method for evaluating SR techniques. Our method includes both objective and subjective measures and is designed from the viewpoint of how SR impacts many essential image processing and vision tasks. We evaluate some state-of-the-art SR algorithms and the results suggest that different SR algorithms should be utilized for different applications. In general, they reflect the consistency and conflict between objective and subjective measures as well as computer vision systems and human vision systems do.

We propose a simple and practical calibration technique that effectively estimates camera parameters from just five points on two orthogonal 1-D objects, each which has three collinear points, one of which is shared. We derive the basic equations needed to realize camera calibration from just five points observed on a single image that captures the objects. We describe a new camera calibration algorithm that estimates the camera parameters based on the basic equations and optimizes them by the bundle adjustment technique. Our method is validated by both computer simulated data and real images. The results show that the camera parameters yielded by our method are close to those yielded by existing methods. The tests demonstrate that our method is both effective and practical.

Tabletop systems are currently being focused on and many applications using these systems are being developed. In such tabletop systems, how to recognize real objects on the table is an essential and important issue. In existing tabletop systems, 2-D markers have been often used. However, their black-and-white pattern, which means nothing to humans, spoils the appearance of the object. We developed transparent markers on a liquid crystal display (LCD) tabletop system by using the polarization features of the LCD and optical lms. In particular, through experiments with various kinds of optical films, we found that two halfwave plates make the markers rotation invariant. By using the transparent markers, tangible transparent Magic Lenses(TM) applications were developed.

PhotoelasticTouch is a novel tabletop system designed to intuitively facilitate touch-based interaction via real objects made from transparent elastic material. The system utilizes vision-based recognition techniques and the photoelastic properties of the transparent rubber to recognize deformed regions of the elastic material. Our system works with elastic materials over a wide variety of shapes and does not require any explicit visual markers. Compared to traditional interactive surfaces, our 2.5 dimensional interface system enables direct touch interaction and soft tactile feedback. In this paper we present our force sensing technique using photoelasticity and describe the implementation of our prototype system. We also present three practical applications of PhotoelasticTouch, a force-sensitive touch panel, a tangible face application, and a paint application.

We propose a novel technique for estimating the number of people in a video sequence; it has the advantages of being stable even in crowded situations and needing no ground-truth data. By analyzing the geometrical relationships between image pixels and their intersection volumes in the real world quantitatively, a foreground image can be directly indicate the number of people. Because foreground detection can be done even in crowded situations, the proposed method can be applied to such situations. Also it can estimate the number of people in an a-priori manner, so it needs no ground-truth data which is necessary for existing feature-based estimating techniques. Experiments show the validity of the proposed method.

In this paper, we present a new approach based on Markov Chain Monte Carlo(MCMC) for the stable monocular tracking of variable interacting targets in 3D space. The crucial problem with monocular tracking multiple targets is that mutual occlusions on the 2D image cause target conflict (change ID, merge targetshellip). We focus on the fact that multiple targets cannot occupy the same position in 3D space and propose to track multiple interacting targets using relative position of targets in 3D space. Experiments show that our system can stably track multiple humans that are interacting with each other.

We propose an online anomal movement detection method using incremental unsupervised learning. As the feature for discrimination, we extract the principal component of the spatio-temporal feature by incremental PCA. We then detect anomal movements by an incremental 1-class SVM. In order to use principal component as the feature for discrimination while supporting incrementation of the subspace, we modify the SVM kernel function to take account of the difference in distance scale between the principal component feature vectors and that of the feature vectors after the subspace is incremented. This allows us to efficiently conduct the relearning process even though the dimension of the original input spatio-temporal feature is high. Experiments show that anomal scenes can be detected without the cost of preparing a lot of labeled data for preliminary learning.

An effective Internet cyber threat monitoring system detects cyber threats using network sensors deployed at particular points on the Internet, statistically analyses the time of attack, source of attack, and type of attack, and then visualizes the result of this analysis. Existing systems, however, simply visualize country-by-country statistics of attacks or hourly changes of attacks. Using these systems, it is difficult to understand the source of attack, the diffusion of the attack, or the relation between the target and the source of the attack. This paper described a method for visualizing cyber threats by using 2-dimensional matrix representation of IP addresses. The advantages of this method are that: (I) the logical distance of IP addresses is represented intuitively, (2) Internet address space is visualized economically, (3) macroscopic information (site level) and microscopic information (local level) are visualized simultaneously. By using this visualization framework, propagation of the Welchia worm and the Sasser.D worm are visualized.

An Internet cyber threat monitoring system detects cyber threats using network sensors deployed at particular points on the Internet, statistically analyzes the time of attack, source of attack, and type of attack, and then visualizes the result of this analysis. Existing systems, however, simply visualize country-by-country statistics of attacks or hourly changes of attacks. Using these systems, it is difficult to understand the source of attack, the diffusion of the attack, or the relation between the target and the source of the attack. This paper described a method for visualizing cyber threats by using 2-dimensional matrix representation of IP addresses. The advantages of this method are that: (1) the logical distance of IP addresses is represented intuitively; (2) Internet address space is visualized economically; (3) macroscopic information (Internet level) and microscopic information (local level) are visualized simultaneously. By using this visualization framework, propagation of the Welchia worm and the Sasser.D worm are visualized.

False detection is a major issue in deploying and maintaining Network-based Intrusion Detection Systems (NIDS). Traditionally, it is recommended to customize its signature database (DB) to reduce false detections. However, it requires quite deep knowledge and skills to appropriately customize the signature DB. Inappropriate customization causes the increase of false negatives as well as false positives. In this paper, we propose a visualization system of a NIDS log, named SnortView, which supports administrators in analyzing NIDS alerts much faster and much more easily. Instead of customizing the signature DB, we propose to utilize visualization to recognize not only each alert but also false detections. The system is based on a 2-D time diagram and alerts are shown as icons with different styles and colors. In addition, the system introduces some visualization techniques such as overlayed statistical information, source-destination matrix, and so on. The system was used to detect real attacks while recognizing some false detections.

This paper describes a two-handed drawing tool developed on our augmented desk system. Using our real-time finger tracking method, a user can draw and manipulate objects interactively by his/her own finger/hand. Based on the former work on two-handed interaction, different roles are assigned to each hand. The right hand is used to draw and to manipulate objects. Using gesture recognition, primitive objects can be drawn by users' handwriting. On the other hand, the left hand is used to manipulate menus and to assist the right hand. By closing all left hand fingers, users can initiate the appearance of structural radial menus around their left hands, and can select appropriate items by using a left hand finger. The left hand is also used to assist in the performance of drawing tasks, e.g., specifying the center of a circle or top-left corner of a rectangle, or specifying the object to be copied.

Computer security breaches are already a major problem in using computers. The most basic defense against it is to monitor and audit the computer logs. Computer logs, however have a huge amount of textual data. It is, therefore, almost impossible to inspect them manually using current systems. We propose a log visualization system called "Tudumi". Tudumi consists of several functions which assist system administrators to perform such tasks manually. These functions are information visualization, log summarization and reflecting known rules into the visualization method. Tudumi makes it easier to detect anomalous user activities, such as intrusion, from a huge amount of computer logs.

Introduces a method for tracking a user's hand in 3D and recognizing the hand's gesture in real time without the use of any invasive devices attached to the hand. Our method uses multiple cameras for determining the position and orientation of a user's hand moving freely in a 3D space. In addition, the method identifies pre-determined gestures in a fast and robust manner by using a neural network which has been properly trained beforehand. This paper also describes results of user study of our proposed method and several types of applications, including 3D object handling for a desktop system and a 3D walkthrough for a large immersive display system.

This article describes a design and implementation of an augmented desk system, named EnhancedDesk, which smoothly integrates paper and digital information on a desk. The system provides users an intelligent environment that automatically retrieves and displays digital information corresponding to the real objects (e.g., books) on the desk by using computer vision. The system also provides users direct manipulation of digital information by using the users' own hands and fingers for more natural and more intuitive interaction. Based on the experiments with our first prototype system, some critical issues on augmented desk systems were identified when trying to pursue rapid and fine recognition of hands and fingers. To overcome these issues, we developed a novel method for realtime finger tracking on an augmented desk system by introducing a infrared camera, pattern matching with normalized correlation, and a pan-tilt camera. We then show an interface prototype on EnhancedDesk. It is an application to a computer-supported learning environment, named Interactive Textbook. The system shows how effective the integration of paper and digital information is and how natural and intuitive direct manipulation of digital information with users' hands and fingers is.

This paper describes two interface prototypes which we have developed on our augmented desk interface system, EnhancedDesk. The first application is Interactive Textbook, which is aimed at providing an effective learning environment. When a student opens a page which describes experiments or simulations, Interactive Textbook automatically retrieves digital contents from its database and projects them onto the desk. Interactive Textbook also allows the student hands-on ability to interact with the digital contents. The second application is the Interactive Venn Diagram, which is aimed at supporting effective information retrieval. Instead of keywords, the system uses real objects such as books or CDs as keys for retrieval. The system projects a circle around each book; data corresponding the book are then retrieved and projected inside the circle. By moving two or more circles so that the circles intersect each other, the user can compose a Venn diagram interactively on the desk. We also describe the new technologies introduced in EnhancedDesk which enable us to implement these applications.

We introduce a fast and robust method for tracking positions of the centers and the fingertips of both right and left hands. Our method makes use of infrared camera images for reliable detection of a user's hands, and uses a template matching strategy for finding fingertips. This method is an essential part of our augmented desk interface in which a user can, with natural hand gestures, simultaneously manipulate both physical objects and electronically projected objects on a desk, e.g., a textbook and related WWW pages. Previous tracking methods which are typically based on color segmentation or background subtraction simply do not perform well in this type of application because an observed color of human skin and image backgrounds may change significantly due to protection of various objects onto a desk. In contrast, our proposed method was shown to be effective even in such a challenging situation through demonstration in our augmented desk interface. This paper describes the details of our tracking method as well as typical applications in our augmented desk interface.

Version control and module management are very important in practical software development. In UNIX, RCS or SCCS is used in general as version control tools. They, however, have a couple of drawbacks. This paper proposed a solution for these issues by applying 3D visualization. The prototype system, VRCS, was developed. In our system, version information stored in the RCS history file is displayed as a 2D tree by taking the z-axis as time. Other 2D trees are laid out in 3D space in the same way. In our visualization, files which compose a certain release of the software are connected by a line called a relation link. By using GUIs, users can check in/out each version easily and interactively. More importantly, just by choosing the relation link, a certain release is rebuilt automatically. Three comparative experiments between VRCS and RCS were conducted to know the effectiveness of VRCS. The result shows that VRCS is faster in checking in/out than RCS.

article Free Access Share on The role of another spatial dimension in software visualization Author: Hideki Koike View Profile Authors Info & Claims ACM Transactions on Information SystemsVolume 11Issue 3July 1993 pp 266–286https://doi.org/10.1145/159161.155370Online:01 July 1993Publication History 46citation928DownloadsMetricsTotal Citations46Total Downloads928Last 12 Months32Last 6 weeks6 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF