Hugging provides a sense of psychological security and is an essential form of support contributing to stress relief. Studies are being conducted to reproduce this effect using robots in human-robot interaction. In this study, we developed a prototype that acquired the user's biometric information using the huggable robot Moffuly-MS and tested its operation. The robot was equipped with a touch sensor, IMU, ultrasonic sensor, and temperature and humidity sensor. We confirmed that it was possible to use these to acquire information on the user's behavior, movements during the hug, and physiological changes. In the future, it is hoped that by improving the system's accuracy and integrating it with dialogue systems, new possibilities will be opened for stress relief and psychological care in the medical and welfare fields.

The ‘keyword method’ is an effective technique for learning vocabulary of a foreign language. It involves creating a memorable visual link between what a word means and what its pronunciation in a foreign language sounds like in the learner’s native language. However, these memorable visual links remain implicit in the people’s mind and are not easy to remember for a large number of words. To enhance the memorisation and recall of the vocabulary, we developed an application that combines the keyword method with text-to-image generators to externalise the memorable visual links into visuals. These visuals represent additional stimuli during the memorisation process. To explore the effectiveness of this approach we first run a pilot study to investigate how difficult it is to externalise the descriptions of mental visualisations of memorable links, by asking participants to write them down. We used these descriptions as prompts for text-to-image generator (DALL-E 2) to convert them into images and asked participants to select their favourites. Next, we compared different text-to-image generators (DALL-E 2, Midjourney, Stable and Latent Diffusion) to evaluate the perceived quality of the generated images by each. Despite heterogeneous results, participants mostly preferred images generated by DALL-E 2, which was used also for the final study. In this study, we investigated whether providing such images enhances the retention of vocabulary being learned, compared to the keyword method alone. Our results indicate that people did not encounter difficulties describing their visualisations of memorable links and that providing corresponding images significantly increases memory retention.

Flood risk communication in disaster-prone communities often relies on traditional tools (e.g., paper and browser-based hazard/flood maps) that struggle to engage community stakeholders and reflect intuitive flood situations. In this paper, we applied the transitional mixed reality (MR) interface concept from pioneering work and extended it for flood risk communication scenarios through co-design with community stakeholders to help vulnerable residents understand flood risk and facilitate preparedness. Starting with an initial transitional MR prototype, we conducted three iterative workshops - each dedicated to device usability, visualization techniques, and interaction methods. We collaborated with diverse community stakeholders in flood-prone areas, collecting feedback to refine the system according to community needs. Our preliminary evaluation indicates that this co-designed system significantly improves user understanding and engagement compared to traditional tools, though some older residents faced usability challenges. We detailed this iterative co-design process, critical insights and design implications, offering our work as a practical case of mixed reality application in strengthening flood risk communication. We also discuss the system's potential to support community-driven collaboration in flood preparedness.

As a pivotal application of Augmented Reality (AR) technology, AR games empower players to bridge reality with virtuality, offering a distinct and immersive experience set apart from traditional games. However, when creating AR games, one of the most formidable challenges faced by designers pertains to the unpredictability of intricate real-world environments, which hinders crafting naturally integrated scenes where virtual objects harmoniously blend with the players' surroundings. In this paper, we introduce EverywhereAR, a system that is capable of flexibly realizing the designer's idea in various real-world scenes. It provides a designer-friendly Game Scene Template development interface, for designers to quickly graphify their inspirations. To achieve the best AR game scene, this work proposes a highly customizable integration method. According to the integrated AR scene graph, the system will arrange each virtual object in a reasonable position to make the generated game scene look natural. We conducted an experiment to evaluate our system's performance across various game scene templates and real-world environments. Results from the experiment indicated that our system was able to generate AR game scenes matching the quality of scenes manually created by professional designers. In addition, we conducted another experiment to assess the effectiveness and usability of the proposed interface. The experiment results showed that the interface was intuitive and efficient, allowing users to create a simple game scene within one minute.

Performing tasks in a close range using augmented content or instructions visualized on a 2D display can be difficult because of missing visual information in the third dimension. This is because the world on the screen is rendered from the perspective of a single camera, typically on the device itself. However, when performing tasks using hands, haptic feedback supports vision, and prior knowledge and visual context affect task performance. This study rendered the world on a display from the user's perspective to re-enable depth cues from motion parallax and compared it with the conventional device perspective during haptic interactions. We conducted a user study involving 20 subjects and two experiments. First, the accuracy of touchpoint and depth estimation was measured under the conditions of a visual context and perspective rendering on a magic-lens display. We found that user-perspective rendering slightly improved the touch accuracy of targets on a physical surface; however, it significantly improved interactions without tactile feedback. This effect is relatively large when contextual information from the environment is absent, and it diminishes with increased haptic interactions. In the second experiment, we used a user-perspective magic lens to validate the proposed method in a practical needle injection scenario and confirm that the initial injections to virtual targets were more accurate. The results indicate that user-perspective rendering on magic lenses improves immediate performance in haptic tasks, suggesting they are particularly advantageous for frequently changing environments or short-duration tasks.

Augmented reality (AR) magic-lens (ML) displays, such as handheld devices, offer a convenient and accessible way to enrich our environment using virtual imagery. Several display technologies, including conventional monocular, less common stereoscopic, and varifocal displays, are currently being used. Vergence and accommodation effects on depth perception, as well as vergence-accommodation conflict, have been studied, where users interact only with the content on the display. However, little research exists on how vergence and accommodation influence user performance and cognitive-task load when users interact with the content on a display and its surroundings in a short timeframe. Examples of this are validating augmented instructions before making an incision and performing general hand-eye coordinated tasks such as grasping augmented objects. To improve interactions with future AR displays in such scenarios, we must improve our understanding of this influence. To this end, we conducted two fundamental visual-acuity user studies with 28 and 27 participants, while investigating eye vergence and accommodation distances on four ML displays. Our findings show that minimizing the accommodation difference between the display and its surroundings is crucial when the gaze between the display and its surroundings shifts rapidly. Minimizing the difference in vergence is more important when viewing the display and its surroundings as a single context without shifting the gaze. Interestingly, the vergence-accommodation conflict did not significantly affect the cognitive-task load nor play a pivotal role in the accuracy of interactions with AR ML content and its physical surroundings.

Augmented Reality (AR) has long been expected to help users improve their working efficiency. However, due to the absence of intelligent systems, existing AR applications are greatly affected by the virtual content interference with real-world activities. Unlike existing work, which focuses more on hiding virtual content to reduce interference, in this work, we propose an innovative AR Task Support System where virtual contents actively guide users with task completion. During task execution, our system proactively searches for and tracks key objects in the scene, and uses this context information to automatically select appropriate virtual content and display positions. Through introducing open-world prompt-based visual models, our system can effectively retrieve few-shot or even zero-shot objects that are uncommon in the dataset. This approach extends the use of AR Task Support System beyond controlled industrial settings to more uncontrolled daily scenarios, overcoming the limitations of existing systems. It also significantly reduces development costs for developers. We demonstrate the advantages of our system over traditional virtual content management systems through a series of experiments that are closer to users' real usage situations.

Abstract Lenslet array near-eye displays are a revolutionary technology that generates a virtual image in the field of view of the observer. Although this technology is advantageous in creating compact near-eye displays, undesirable artifacts occur when the user pupil moves outside of the pupil practical movable region (PPMR). Even with dynamic image updating based on eye-tracking techniques, artifacts can still be perceived when human eyes turn rapidly. To enlarge PPMR, we proposed a new rendering method in previous work. To improve the rendering speed in the eye tracking system, look-up tables are used. The disadvantage of the onboard system is the large memory consumption. In this study, we analyzed the system parameters of the incident pupil and pupil margin light columns, the feasibility of the optimized system, and evaluated the optimized system that can adapt to the maximum velocity of the saccadic pupil movement. We optimized the rendering method to reduce memory consumption in the process of generating microdisplay images. In addition, we provide GPU rendering method to improve system speed and reduce system latency to meet the maximum human eye rotation speed. We conducted user studies to evaluate the effect of the method using the optimized rendering method combined with eye tracking to reduce artifacts for fast eye rotation on different images and videos. Results showed that our method effectively reduced artifacts via the optimized rendering method with eye tracking, which adapted to faster human eye movements.

This study proposes a communication robot that plays video games together with the user to create a fun atmosphere for a daily-use robot. In recent years, the lack of conversation in daily life has become a problem, and as a solution to this problem, a dialogue robot have attracted attention as a substitute for a human and as a means of encouraging daily speech. In this study, we propose a dialogue robot that plays a video game with a user in order to produce enjoyable daily communication with the robot. The proposed method not only controls the robot’s speech, but also controls the game development by manipulating the game characters. The proposed robot enhances a user’s enjoyment by adjusting the level of game manipulation to produce a competitive match and by speaking according to the game scene. A robot that produces speech that makes a player feel positive emotions is called a friendly robot. In order to examine the effects of these robots on a user’s enjoyment, we conducted an experiment with 30 subjects in which they were subjectively evaluated by questionnaires in two types of matches: with a friendly robot and alone. The results showed that robots were more enjoyable than playing alone.

Previous research has shown that many factors in a virtual environment influence the way, how we perceive interaction with humanoid agents. It is still unknown if visual stereotypes also translate into human-robot interaction scenarios. To investigate this relationship, we conducted a two-part user study with 26 participants using the Stereotype Content Model (SCM) to identify different stereotypical avatars and visual features in terms of their effect on the perception of robot touch. We found that humanoid avatars which resemble the robot and are perceived as warm in the SCM are more liked and preferred than cold avatars. We also found different effects of visual appearance on touch perception. This suggests that a humanoid-stereotpyical appearance of a touch-care robot in virtual reality can change the overall perception of the interaction. We propose the use of stereotypical warm avatars as described in the SCM for humanity-based applications.

In this study, we analyzed the human "gentle stroke with speech behavior" for comfortable motion by implementing the model into a robot. The results of 19 subjects’ human analysis showed that there was a positive correlation between stroke speed and speech speed. The faster the speech speed, the larger the variance of each feature. Moreover, it was also estimated regardless of the length of each sentence, people tend to fold their hands at "breaks in sentences" or "points just before breaks in sentences". Based on this human-modeled knowledge, we evaluated a human comfort experiment in 36 subjects using a robot arm to reproduce the "stroke and speech" behavior at different speeds. Results of the questionnaire showed that subjects felt more comfortable when stroke speed and speech speed were positively correlated. This result suggests that people consciously adjust their stroking and speaking styles when they perform "comfortable stroke with speech behavior".

Recent augmented reality (AR) advancements have enabled the development of effective training systems, especially in the medical, rehabilitation, and industrial fields. However, it is unclear from the literature what the intrinsic value of AR to training is and how it differs across multiple application fields. In this work, we gathered and reviewed the prototypes and applications geared towards training the intended user's knowledge, skills, and abilities. Specifically, from IEEE Xplore plus other digital libraries, we collected 64 research papers present in high-impact publications about augmented reality training systems (ARTS). All 64 papers were then categorized according to the training method used, and each paper's evaluations were identified by validity. The summary of the results shows trends in the training methods and evaluations that incorporate ARTS in each field. The narrative synthesis illustrates the different implementations of AR for each of the training methods. In addition, examples of the different evaluation types of the current ARTS are described for each of the aforementioned training methods. We also investigated the different training strategies used by the prevailing ARTS. The insights gleaned from this review can suggest standards for designing ARTS regarding training strategy, and recommendations are provided for the implementation and evaluation of future ARTS.

Nowadays, decreasing opportunities for people to have daily conversations due to the increase in the number of withdrawn young people and households living alone. The lack of daily conversation has been pointed out as a risk that can lead to mental problems such as depression, and serious health problems such as dementia for the elderly. Efforts to encourage daily communication by having communication robots that act as talking partners is attracting attention to solve those problems. One of the challenges of communication robots is the difficulty of maintaining users' motivation to continue using robots. In this study, we propose the communication robot that plays a video game together with a user as an approach to keep the user's motivation high to use the robot. The proposed method not only controls the dialogue content of the robot but also controls the video game situation by manipulating the video game character. In this system, we aim to create an atmosphere where users can enjoy playing video games with the robot together, and the proposed game communication robot can keep their motivation high to use the robot.

Projector-camera systems have long been used in measuring three-dimensional shapes. Most projector-camera systems can only be used in dark rooms because frame-based cameras are not robust against strong ambient light and are difficult to obtain correspondence to the image pixels of the projector. Recently, event cameras, which can detect the direction of luminance change, have received attention in the field of computer vision. When considering the many advantages of event cameras, this study focuses on their wide dynamic range (120 vs. 40 dB of a frame-based camera) and their ability to detect fast luminance changes. Our objective is to realize a projector-camera system that combines the event camera with a projector under the strong ambient light. Specifically, this study proposes a new structured light that combines different frequencies of flickers to acquire the correspondence between the image pixels of the event camera and the projector. This method does not rely on the co-axial frame-based measurement and synchronization mechanism between projector and camera and is thus applicable to most general event cameras. Experiments confirm that the proposed method obtains the correspondence robustly with reasonable accuracy in a bright room (up to 2,600 lux) under general indoor lighting and additional light projection.

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user’s ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user’s performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training.

Triangle meshes are used in many important shape-related applications including geometric modeling, animation production, system simulation, and visualization. However, these meshes are typically generated in raw form with several defects and poor-quality elements, obstructing them from practical application. Over the past decades, different surface remeshing techniques have been presented to improve these poor-quality meshes prior to the downstream utilization. A typical surface remeshing algorithm converts an input mesh into a higher quality mesh with consideration of given quality requirements as well as an acceptable approximation to the input mesh. In recent years, surface remeshing has gained significant attention from researchers and engineers, and several remeshing algorithms have been proposed. However, there has been no survey article on remeshing methods in general with a defined search strategy and article selection mechanism covering the recent approaches in surface remeshing domain with a good connection to classical approaches. In this article, we present a survey on surface remeshing techniques, classifying all collected articles in different categories and analyzing specific methods with their advantages, disadvantages, and possible future improvements. Following the systematic literature review methodology, we define step-by-step guidelines throughout the review process, including search strategy, literature inclusion/exclusion criteria, article quality assessment, and data extraction. With the aim of literature collection and classification based on data extraction, we summarized collected articles, considering the key remeshing objectives, the way the mesh quality is defined and improved, and the way their techniques are compared with other previous methods. Remeshing objectives are described by angle range control, feature preservation, error control, valence optimization, and remeshing compatibility. The metrics used in the literature for the evaluation of surface remeshing algorithms are discussed. Meshing techniques are compared with other related methods via a comprehensive table with indices of the method name, the remeshing challenge met and solved, the category the method belongs to, and the year of publication. We expect this survey to be a practical reference for surface remeshing in terms of literature classification, method analysis, and future prospects.

Accurately placing virtual objects in a scene is a challenging tasks in handheld augmented reality (HAR). To add and arrange virtual objects in HAR, users must manipulate 6 degrees of freedom (DoFs) of the virtual object, namely: position (3) and orientation (3). However, it is difficult to manipulate all DoFs with the two-dimensional display of the handheld device. We present SlidAR+, a method for controlling the position and orientation of objects in HAR. SlidAR+ is an extension of SlidAR [1], a technique that allows users to control the position of a virtual object by manipulating only 1 DoF. We use the direction of gravity as a constraint to improve the user’s control and reduce the time it takes to adjust the orientation. Upon comparing it with a state-of-the-art object manipulation method, using SlidAR+, user were able to complete the tasks faster under our expected conditions and were also preferred by most participants.

We present a novel method that robustly estimates the reflectance, even in an environment with dynamically changing light. To control the appearance of an object by using a projector-camera system, an appropriate estimate of the object's reflectance is vital to the creation of an appropriate projection image. Most conventional estimation methods assume static light conditions; however, in practice, the appearance is affected by both the reflectance and environmental light. In an environment with dynamically changing light, conventional reflectance estimation methods require calibration every time the conditions change. In contrast, our method requires no additional calibration because it simultaneously estimates both the reflectance and environmental light. Our method is based on the concept of creating two different light conditions by switching the projection at a rate higher than that perceived by the human eye and captures the images of a target object separately under each condition. The reflectance and environmental light are then simultaneously estimated by using the pair of images acquired under these two conditions. We implemented a projector-camera system that switches the projection on and off at 120 Hz. Experiments confirm the robustness of our method when changing the environmental light. Further, our method can robustly estimate the reflectance under practical indoor lighting conditions.

Augmented reality (AR) systems support complex tasks like assembly by overlaying task-related content onto the real world. In recent years, the effort of designing and developing assembly task support systems in AR decreased with the availability of high potential head-mounted displays and provision of integrated development environments. Nevertheless, problems still arise when companies craft an effective AR task support system, particularly in the difficulty of selecting appropriate techniques and information-presentation methods, and the requirements that vary with each use case. In this study, we formulated a corresponding guideline, developed a selection aid tool that incorporates filtering based on the categorization of subtasks and the degree of freedom of available tracking, and evaluated their effectiveness in two experiments. First, to confirm effects on system design, we asked 18 participants to perform the design action of the AR system with the guideline for two tasks (PC assembly and rope work). Consequently, to verify the quality of the designed AR systems from Experiment 1, we asked another set of 20 participants to perform the same tasks with those systems. The results confirm that using the guideline can considerably lower efforts creating media and alleviate the error for a specific process. We envision our guideline and tool to be accessible as an online web page, assisting AR assembly task support system designer/developers worldwide.

Teaching robots new skills is considered as an important aspect of Human-Robot Collaboration (HRC). One challenge is that robots cannot communicate feedback in the same ways as humans do. This decreases the trust towards robots since it is difficult to judge, before the actual execution, if the robot has learned the task correctly. In this paper, we introduce an Augmented Reality (AR) based visualization tool that allows humans to verify the taught behavior before its execution. Our verification interface displays a virtual simulation embedded into the real environment, timely coupled with a semantic description of the current action. We developed three designs based on different interface/visualization-technology combinations to explore the potential benefits of enhanced simulations using AR over traditional simulation environments like RViz. We conducted a user study with 18 participants to assess the effectiveness of the proposed visualization tools regarding error detection capabilities. One of the advantages of the AR interfaces is that they provide more realistic feedback than traditional simulations with a lower cost of not having to model the entire environment.

With projection mapping, we can control the appearance of real-world objects by adding illumination. A projector can be used to control light reflected from an object, where the reflected light depends not only on the projection but also on the reflectance and environmental light. Because the resulting colors are affected by the reflectance and environmental light, the presentable color range of a projector is limited. The purpose of this work is to broaden this limited range by focusing on the perceived colors. Although our eyes capture reflected light to perceive the colors of an object, the colors perceived by humans are not always the same as the actual colors, and there are often significant differences between them because of the human visual system. To overcome the limitations of a projector based on human perception, we intentionally generate this difference by inducing a visual illusion, namely, color constancy. In this work, we designed an algorithm to determine the projected colors for presenting the desired colors perceptually by employing a color constancy effect. In addition, we conducted a user study and confirmed that our algorithm can (1) create a misperception regarding the color of illumination, (2) broaden the presentable color range of a projector, and (3) shift the perceptual colors in the desirable direction.

Humanitude is a multimodal communication care method that utilizes seeing, touching, and speaking. Moreover, a touch-care method is well known as an effective care method mainly focus on touch motion. These kinds of care techniques are effective in practical situations, however, it is difficult to provide such care therapy to all patients due to the lack of human resources. To address this problem, researchers try to develop a touch-care robot that can provide touch-care automatically. Conventional research of touch-care robot mainly focuses on the movement of stroke or the speech that only considers the impression of the contents of speech but not prosodic information. Therefore, in this research, we focus on the speech rate in the prosodic information with stroke motion. In this work, we investigate the effects of speech rate on the prosodic information and evaluate the relationship between stroke pace and speech rate to improve human comfort. We conducted a user study with 6 participants around 20 years old males. As a result of the list of the questionnaire suggests a correlation between stroke pace and speech rate that provides comfort.

Using projection mapping, we can control the appearance of realworld objects by projecting colored light onto them. Since a projector can only add illumination to the scene, limited color gamut can be presented through projection mapping. However, actual color and perceived color are not always the same, and there is often large difference between them. We intentionally generate this difference by inducing visual illusion for extending the controllable color gamut of a projector. In particular, we induce color constancy. We demonstrate changing object color with and without inducing color constancy. Audiences perceive the controlled colors with and without illusion as different color nevertheless these colors are physically completely same.

Using projection mapping, we can control the appearance of realworld objects by projecting colored light onto them. Because a projector can only add illumination to the scene, only a limited color gamut can be presented through projection mapping. In this paper we describe how the controllable color gamut can be extended by accounting for human perception and visual illusions. In particular, we induce color constancy to control what color space observers will perceive. In this paper, we explain the concept of our approach, and show first results of our system.

Work space simulations help trainees acquire skills necessary to perform their tasks efficiently without disrupting the workflow, forgetting important steps during a procedure, or the location of important information. This training can be conducted in Augmented and Virtual Reality (AR, VR) to enhance its effectiveness and speed. When the skills are transferred to the actual application, it is referred to as positive training transfer. However, thus far, it is unclear which training, AR or VR, achieves better results in terms of positive training transfer. We compare the effectiveness of AR and VR for spatial memory training in a control-room scenario, where users have to memorize the location of buttons and information displays in their surroundings. We conducted a within-subject study with 16 participants and evaluated the impact the training had on short-term and long-term memory. Results of our study show that VR outperformed AR when tested in the same medium after the training. In a memory transfer test conducted two days later AR outperformed VR. Our findings have implications on the design of future training scenarios and applications.

Virtual Reality (VR) Head-Mounted Displays (HMDs) are on the verge of becoming commodity hardware available to the average user and feasible to use as a tool for 3D work. Some HMDs include front-facing cameras, enabling Augmented Reality (AR) functionality. Apart from avoiding collisions with the environment, interaction with virtual objects may also be affected by seeing the real environment. However, whether these effects are positive or negative has not yet been studied extensively. For most tasks it is unknown whether AR has any advantage over VR. In this work we present the results of a user study in which we compared user performance measured in task completion time on a 9 degrees of freedom object selection and transformation task performed either in AR or VR, both with a 3D input device and a mouse. Our results show faster task completion time in AR over VR. When using a 3D input device, a purely VR environment increased task completion time by 22.5 percent on average compared to AR ( ). Surprisingly, a similar effect occurred when using a mouse: users were about 17.3 percent slower in VR than in AR ( ). Mouse and 3D input device produced similar task completion times in each condition (AR or VR) respectively. We further found no differences in reported comfort.

Inspection tasks focus on observation of the environment and are required in many industrial domains. Inspectors usually execute these tasks by using a guide such as a paper manual, and directly observing the environment. The effort required to match the information in a guide with the information in an environment and the constant gaze shifts required between the two can severely lower the work efficiency of inspector in performing his/her tasks. Augmented reality (AR) allows the information in a guide to be overlaid directly on an environment. This can decrease the amount of effort required for information matching, thus increasing work efficiency. AR guides on head-mounted displays (HMDs) have been shown to increase efficiency. Handheld AR (HAR) is not as efficient as HMD-AR in terms of manipulability, but is more practical and features better information input and sharing capabilities. In this study, we compared two handheld guides: an AR interface that shows 3D registered annotations, that is, annotations having a fixed 3D position in the AR environment, and a non-AR picture interface that displays non-registered annotations on static images. We focused on inspection tasks that involve high information density and require the user to move, as well as to perform several viewpoint alignments. The results of our comparative evaluation showed that use of the AR interface resulted in lower task completion times, fewer errors, fewer gaze shifts, and a lower subjective workload. We are the first to present findings of a comparative study of an HAR and a picture interface when used in tasks that require the user to move and execute viewpoint alignments, focusing only on direct observation. Our findings can be useful for AR practitioners and psychology researchers.

Entertainment is challenging to observe, especially with children, due to limited analytical tools. In response, we present a modified framework for entertainment computing, COMPASS -- COmbined Mental, PhysicAl, Social and Spatial factors, which we use to analyze augmented reality activities for cub scouts.

Over the last decades, visual representations of data has been a commonly used medium to bolster human cognition in performance evaluation of professional athletes. However, the current approaches to these visualizations still build upon the paper based principles of initial designs with solid backgrounds. Due to this situation, same visualizations usually fail to provide explicit information about the physical characteristics of the scenario that the data was captured, such as the form of athletes. In this work, we present a data visualization method which combines visual representations of cyclist's pedaling with correlated frames of indoor training videos. We designed a prototype system which allows us to superimpose various pedaling visualizations onto simultaneously captured training videos of cyclists. The results of user studies we conducted with twelve professional cyclists confirmed their interest in new possibilities emerging from intuitive data visualizations. We also received valuable feedback about the feasible benefits of our approach over traditional approaches, such as reduced cognitive overload in understanding visualizations. We conclude by discussing the future implementations and application areas of our approach and further need of adjusting it to distinct training scenarios.

Co-located haptic feedback in mixed and augmented reality environments can improve realism and user performance, but it also requires careful system design and calibration. In this poster, we determine the thresholds for perceiving co-location errors through two psychophysics experiments in a typical fine-motor manipulation task. In these experiments we simulate the two fundamental ways of implementing VHAR systems: first, attaching a real tool; second, augmenting a virtual tool. We determined the just-noticeable co-location errors for position and orientation in both experiments and found that users are significantly more sensitive to co-location errors with virtual tools. Our overall findings are useful for designing visuo-haptic augmented reality workspaces and calibration procedures.

A common factor among current generation optical see-through augmented reality systems is fixed focal distance to virtual content. In this work, we investigate the issue of focus blur, in particular, the blurring caused by simultaneously viewing virtual content and physical objects in the environment at differing focal distances. We examine the application of dynamic sharpening filters as a straight forward, system independent, means for mitigating this effect improving the clarity of defocused AR content. We assess the utility of this method, termed SharpView, by employing an adjustment experiment in which users actively apply varying amounts of sharpening to reduce the perception of blur in AR content. Our experimental results validate the ability of our SharpView model to improve the visual clarity of focus blurred content, with optimal performance at focal differences well suited for near field AR applications.

Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly.

Projection-based Augmented Reality commonly projects on rigid objects, while only few systems project on deformable objects. In this paper, we present Pseudo Printed Fabrics (PPF), which enables the projection on a deforming piece of cloth. This can be applied to previewing a cloth design while manipulating its shape. We support challenging manipulations, including heavy occlusions and stretching the cloth. In previous work, we developed a similar system, based on a novel marker pattern; PPF extends it in two important aspects. First, we improved performance by two orders of magnitudes to achieve interactive performance. Second, we developed a new interpolation algorithm to keep registration during challenging manipulations. We believe that PPF can be applied to domains including virtual-try on and fashion design.

Usability evaluations are important to the development of augmented reality systems. However, conducting large-scale longitudinal studies remains challenging because of the lack of inexpensive but appropriate methods. In response, we propose a method for implicitly estimating usability ratings based on readily available sensor logs. To demonstrate our idea, we explored the use of features of accelerometer data in estimating usability ratings in an annotation task. Results show that our implicit method corresponds with explicit usability ratings at 79% and 84%. These results should be investigated further in other use cases, with other sensor logs.

Augmented reality (AR) is an emerging technology for communicating learning contents. Several AR systems are designed for learning. However, studies that have investigated instructional strategies for applying AR are few. This investigation requires the implementation of prototypes that use state-of-the-art technology and sound learning theory. In this work, we implemented two prototypes for learning Filipino and German words by first developing a handheld AR platform. These prototypes demonstrate situated vocabulary learning. Using our AR system, students can learn words related to their current environment. We assessed the quality of these prototypes by conducting usability evaluations. For the theoretical grounding, we leveraged on multimedia learning theory to design the content. Through our handheld AR platform, we evaluated situated vocabulary learning by comparing our prototypes to a flash cards application. In the first evaluation, students scored significantly lower when using AR in an immediate post-test. However, this difference disappeared after taking into account the variability in usability scores via analysis of covariance. Taking account usability is fairer when comparing an emerging technology to traditional technology. Test scores were also not significantly different in a delayed post-test. In the second evaluation, although the post-test score and answering time of students did not differ, our results showed that they feel more satisfied and can keep their attention better when using AR. For the first time, we demonstrated situated vocabulary learning by using AR. Moreover, our preliminary study confirms the intuition that students can achieve the same score using AR, but with benefits such as ease in maintaining attention and increased satisfaction.

Augmented reality (AR) is an enabling technology for presenting information in relation to real objects or real environments. AR is situated multimedia or information that is positioned in authentic physical contexts. In this paper, we discuss how we address issues in creating AR content for educational settings. From the learning theory perspective, we explain that AR is a logical extension of multimedia learning theory. From the development perspective, we demonstrate how AR content can be created through our in situ authoring tool and our platform for handheld AR.

The idea of using Augmented Reality (AR) user interfaces (UIs) to create 3D media content, such as 3D models for movies and games has been repeatedly suggested over the last decade. Even though the concept is intuitively compelling and recent technological advances have made such an application increasingly feasible, very little progress has been made towards an actual real-world application of AR in professional media production. To this day, no immersive 3D UI has been commonly used by professionals for 3D computer graphics (CG) content creation. In this paper, we are first to publish a requirements analysis for our target application in the professional domain. Based on a survey that we conducted with media professionals, the analysis of professional 3D CG software, and professional training tutorials, we identify these requirements and put them into the context of AR UIs. From these findings, we derive several interaction design principles that aim to address the challenges of real-world application of AR to the production pipeline. We implemented these in our own prototype system while receiving feedback from media professionals. The insights gained in the survey, requirements analysis, and user interface design are relevant for research and development aimed at creating production methods for 3D media production.

For this demo, we present an Augmented Reality (AR) User Interface (UI) for the 3D design software Autodesk Maya, aimed at professional media creation. A user wears a head-mounted display (HMD) and thin cotton gloves which allow him to interact with virtual 3D models in the work area. Additional viewers can see the video stream on a projector and thus share the users view. Both head and hand positions are tracked from the HMD video stream, and an inertial measurement unit (IMU) and conductive materials on the gloves allow interaction with virtual objects. This system is built using Autodesk Maya — a professional 3D software package commonly used in the media industry — and aims to fulfill the requirements of professional 3D design work which we identified in our paper of the same title. While still an early prototype, it was already tested with media professionals to evaluate our approach.

We demonstrate the geometrically-correct projection-based texture mapping onto a deformable object like a cloth. This system can be used to simulate design that involves change in shape, such as sheets of malleable material. The geometrically-correct projection-based texture mapping onto a cloth is conducted using the measurement of object's 3D shape and the detection of the retro-reflective marker on the object's surface. Rapid prototyping is used as an example application of this projection technique.

Handheld augmented reality (HAR) applications must be carefully designed and improved based on user feedback to sustain commercial use. However, no standard questionnaire considers perceptual and ergonomic issues found in HAR. We address this issue by creating a HAR Usability Scale (HARUS).

In this paper, we propose a method for estimating the camera pose for an environment in which the intrinsic camera parameters change dynamically. In video see-through augmented reality (AR) technology, image-based methods for estimating the camera pose are used to superimpose virtual objects onto the real environment. In general, video see-through-based AR cannot change the image magnification that results from a change in the camera׳s field-of-view because of the difficulty of dealing with changes in the intrinsic camera parameters. To remove this limitation, we propose a novel method for simultaneously estimating the intrinsic and extrinsic camera parameters based on an energy minimization framework. Our method is composed of both online and offline stages. An intrinsic camera parameter change depending on the zoom values is calibrated in the offline stage. Intrinsic and extrinsic camera parameters are then estimated based on the energy minimization framework in the online stage. In our method, two energy terms are added to the conventional marker-based method to estimate the camera parameters: reprojection errors based on the epipolar constraint and the constraint of the continuity of zoom values. By using a novel energy function, our method can accurately estimate intrinsic and extrinsic camera parameters. We confirmed experimentally that the proposed method can achieve accurate camera parameter estimation during camera zooming.

Projection-based Augmented Reality commonly employs a rigid substrate as the projection surface and does not support scenarios where the substrate can be reshaped. This investigation presents a projection-based AR system that supports deformable substrates that can be bent, twisted or folded. We demonstrate a new invisible marker embedded into a deformable substrate and an algorithm that identifies deformations to project geometrically correct textures onto the deformable object. The geometrically correct projection-based texture mapping onto a deformable marker is conducted using the measurement of the 3D shape through the detection of the retro-reflective marker on the surface. In order to achieve accurate texture mapping, we propose a marker pattern that can be partially recognized and can be registered to an object’s surface. The outcome of this work addresses a fundamental vision recognition challenge that allows the underlying material to change shape and be recognized by the system. Our evaluation demonstrated the system achieved geometrically correct projection under extreme deformation conditions. We envisage the techniques presented are useful for domains including prototype development, design, entertainment and information based AR systems.

Augmented reality (AR) x-ray interaction is an enabling technology for providing students with virtual abstractions of the interior of an object. It provides students contextual visualization - the presentation of virtual information in the rich context of a real environment - thereby offering compelling experiences. According to experiential learning theory, such personal experiences are necessary for reaching different types of learners. AR x-ray is a novel interaction technique for education, thus, it is necessary to investigate how it affects the students' perception. We implemented AR x-ray using a state-of-the-art occlusion technique, and compared it to viewing 3D objects without occlusion. Results of two user studies (n=23 and n=47) show that there are no significant differences in realism, perception of depth, and visibility with occlusion and without occlusion, and that the current technique is usable for educational purposes. We also conducted interviews with both students (n=23) and teachers (n=12). Results indicate that AR x-ray is perceived to be useful for motivating and explaining to students. The teachers expressed willingness to adopt AR x-ray and to undergo training for using AR-based teaching materials.

Engineers and educators alike have prototyped a variety of augmented reality learning experiences (ARLEs). However, adapting ARLEs in educational practice would require an interdisciplinary approach that considers learning theory, pedagogy and instructional design. To address this requirement, we model ARLEs as learning objects by outlining the necessary components, and we propose a participatory design to demonstrate the authoring process of an augmented reality learning object (ARLO). ARLOs can be made useful in many scenarios if teachers are empowered to edit its context elements, content and instructional activity. Lastly, we point to the research questions entailed in modeling ARLEs as ARLOs.

In general, video see-through based augmented reality (AR) cannot change the magnification of camera zooming parameter due to the difficulty of dealing with changes in intrinsic camera parameters. To realize the usage of camera zooming in AR, we propose a novel simultaneous intrinsic and extrinsic camera parameter estimation method based on an energy minimization framework. Our method is composed of the online and offline stages. An intrinsic camera parameter change depending on the zoom values is calibrated in the offline stage. Intrinsic and extrinsic camera parameters are then estimated based on the energy minimization framework in the online stage. In our method, two energy terms are added to the conventional marker-based camera parameter estimation method. One is reprojection errors based on the epipolar constraint. The other is the constraint of continuity of zoom values. By using a novel energy function, our method can estimate accurate intrinsic and extrinsic camera parameters. In an experiment, we confirmed that the proposed method can achieve accurate camera parameter estimation during camera zooming.

A method for fast and incremental indexing, with both effective and efficient query processing, is proposed for XML element retrieval. When frequent document updates occur on the Web, they must be handled to maintain the effectiveness of the search system. When new topics are added and document statistics change drastically, search accuracy is also reduced. We therefore consider a method not only for updating indices efficiently but also for processing queries effectively and efficiently. We construct indices for fast updating and propose a method for computing accurate term weights even under dynamically changing statistics. Experimental results show that our proposed system can handle document updates at low cost and search documents accurately even when their statistics change.

The objective of this study is to investigate the relationship between the features of augmented reality (AR) and human memorization ability. The basis of this relation is derived from the following features. The AR feature is that AR can provide information associated with specific locations in the real world. The feature of human memory is that humans can easily memorize information if the information is visually associated with specific locations. To investigate this relation, we conduct a pilot user study in which blocks are picked from some drawers. As a result, significant differences are found between a situation in which visual information is displayed at the location of each drawer in the real world and that in which textual information is displayed at an unrelated location.

This paper presents an application of Augmented Reality (AR) in Rapid Prototyping (RP) of non-textured rigid curved surfaces. By enhancing the prototypes with AR, evaluation of its design and aesthetic concepts in real-time becomes easier, saving time and production costs. In our application, no fiducial markers are required and the CAD model used to build the prototype is applied in an edge-based tracking system specially designed to deal with curved shapes. Results from a pilot user study comparing the use of a 3D software and the proposed application are also presented.

In the research fields of Augmented Reality (AR) and Mixed Reality (MR), tracking and registration methods are still one of the most important topics. The tracking research field is highly active, and numerous methods appear on a regular basis. The TrakMark working group (WG) was established 2009 to create a benchmark test that permits objective and accurate evaluation of the tracking methods. This year, the workshop will cover a wide range of topics concerning AR/MR registration, tracking and benchmarking. Key areas include, but are not limited to: — Vision-based registration, camera localization — Visual SLAM, structure from motion, camera calibration, sensor fusion — Natural feature tracking, object tracking, feature detection, feature description — Comparison of methods, evaluation of methods, suggestion of new benchmarking scheme — Survey of tracking papers.

Computer-aided design (CAD) system conventionally have been widely used to support designers for creating, modifying, adding something to or removing something from objects by showing simulated objects on computer screen. These virtual, non-physical, objects have been, however, known as imperfect imitation of reality. The impression of shape is highly related to the second order derivative of geometric feature of the shape. Conventional CAD systems, including AutoCAD, usually have visualization feature of the first derivative (normal) and the second derivative (curvature) of given surfaces. There, however, still have been problems in curvature visualization on the screen. First, it lacks true feeling of physical objects. Second, even if designers were given a physical mock-up object in hand, they wouldn't precisely recognize minute change of curvatures — few designers can sense small differences of curvature and most others need a special device to check the curvature. For solving these problem, the authors propose a novel curvature visualization system based on mixed reality technology. The color mapping according to the Gaussian curvature calculated via a time-of-flight camera provides the observers with intuitively understanding the object's curvature information.

The unified technique for the irradiance correction and appearance enhancement for the real scene is proposed in this paper. The proposed method employed MPC algorithm for the projector camera system and enabled arbitrary appearance control such like photo retouching software in the real world. In the experiment, the appearance control of saturation enhancement, color removal, phase control, edge enhancement, image blur, makes unique brightness and the other enhancements for the real scene are shown.

This is a proposal to the ISMAR community from Japanese AR/MR researchers for the future progress of AR/MR technology. We hope to expand our activities over the ISMAR community and call for international participants who would take part in a number of voluntary works. At the same time, this paper presents a current view of the outcomes of these activities may have be in due course. We will focus here on the various tracking methods, one of the most active themes at the annual ISMAR symposiums. The main goal of our activities is to build a framework to comprehensively evaluate a variety of existing and future tracking methods. Strictly speaking, our targets should include all the geometric registration methods that merge the real and virtual world seamlessly. They can also be termed as real-time 3D matchmove. After initial registration (calibration) of two spaces is achieved, either object tracking or camera tracking is required when the subject or camera moves. In addition to methods suited to static registration, there exist also methods that focus solely on improving the performance of tracking, without calibration. In this paper, when we use the terms "registration and tracking" or simply "tracking" we will be referring to the general definition of tracking as given above.

The two papers in this special section are extended versions of papers originally presented at the International Symposium on Mixed and Augmented Reality (ISMAR) 2007. These two papers won awards at the symposium.

In this paper, a dynamic appearance enhancement method for the less saturated object is proposed. The appearance enhancement method proposed in this paper is realized with a projector camera dynamic feedback system. Therefore, the proposed method has an ability of the rapid and continuous appearance enhancement and it is useful for the human visual perception assistance. Also, the system not requires a pre-captured appearance of the target.

We describe an augmented reality system for superimposing three-dimensional (3-D) live content onto two-dimensional fiducial markers in the scene. In each frame, the Euclidean transformation between the marker and the camera is estimated. The equivalent virtual view of the live model is then generated and rendered into the scene at interactive speeds. The 3-D structure of the model is calculated using a fast shape-from-silhouette algorithm based on the outputs of 15 cameras surrounding the subject. The novel view is generated by projecting rays through each pixel of the desired image and intersecting them with the 3-D structure. Pixel color is estimated by taking a weighted sum of the colors of the projections of this 3-D point in nearby real camera images. Using this system, we capture live human models and present them via the augmented reality interface at a remote location. We can generate 384/spl times/288 pixel images of the models at 25 fps, with a latency of <100 ms. The result gives the strong impression that the model is a real 3-D part of the scene.

A real-time system for capturing humans in 3D and placing them into a mixed reality environment is presented in this paper. The subject is captured by nine cameras surrounding her. Looking through a head-mounted-display with a camera in front pointing at a marker, the user can see the 3D image of this subject overlaid onto a mixed reality scene. The 3D images of the subject viewed from this viewpoint are constructed using a robust and fast shape-from-silhouette algorithm. The paper also presents several techniques to produce good quality and speed up the whole system. The frame rate of our system is around 25 fps using only standard Intel processor-based personal computers. Besides a remote live 3D conferencing and collaborating system, we also describe an application of the system in art and entertainment, named Magic Land, which is a mixed reality environment where captured avatars of human and 3D computer generated virtual animations can form an interactive story and play with each other. This system demonstrates many technologies in human computer interaction: mixed reality, tangible interaction, and 3D communication. The result of the user study not only emphasizes the benefits, but also addresses some issues of these technologies.

This demonstration shows a city-planning system based on augmented reality with tangible user interface. Miniature models, illustrations and graphical computer displays have been used for the comparison and consideration in city-planning process. Augmented reality technology enables users to consider city plans more effectively and easily. One important issue of the augmented reality environment is how user can manipulate 3D structures that are displayed as virtual objects. It has to be intuitive and easy so that it may not disturb user's thought. We propose a new direct manipulation method based on the concept called tangible user interface. User holds a transparent cup upside down and can pick up, move or delete a virtual object by using it.

In this paper, we describe the effects of shadow representation of virtual objects in augmented reality. Optical consistency is important in order to create realistic augmented reality environments. We focus on providing accurate shadows and made two assumptions about the effects of shadow representation of virtual objects. First, that the shadow of virtual objects provides a stronger connection between the real world and virtual objects and so increases virtual object presence. Second, that the shadow of virtual objects provides depth cues and so makes three-dimensional perceptions easier for the users of the interface. We report on two experiments that show that these assumptions are correct. We also find that users report that a characteristic shadow shape provides more virtual object presence in spite of incorrect virtual light direction.

We describe a real-time 3-D augmented reality video- conferencing system. With this technology, an observer sees the real world from his viewpoint, but modified so that the image of a remote collaborator is rendered into the scene. We register the image of the collaborator with the world by estimating the 3-D transformation between the camera and a fiducial marker. We describe a novel shape- from-silhouette algorithm, which generates the appropriate view of the collaborator and the associated depth map at 30 fps. When this view is superimposed upon the real world, it gives the strong impression that the collaborator is a real part of the scene. We also demonstrate interaction in virtual environments with a live fully 3-D collaborator. Finally, we consider interaction between users in the real world and collaborators in a virtual space, using a tangible AR interface.

We describe a method for observing multiple targets in a wide-area spatial environment using a distributed vision system (DVS). The DVS is constructed of some 'watching stations' that consist of a camera, an image processor and a computer network that connects each systems. The system's goal is to track multiple people in a wide-area that cannot be watched by single visual sensor. Our approach is based on three algorithms; an algorithm for real-time human tracking, the task decision algorithms of individual watching stations, and the object-matching method used between stations. We also describe experimental results that show the validity of our approach.

This paper presents an interactive theatre based on an embodied mixed reality space and wearable computers. Embodied computing mixed reality spaces integrate ubiquitous computing, tangible interaction and social computing within a mixed reality space, which enables intuitive interaction with physical world and virtual world. We believe it has potential advantages to support novel interactive theatre experiences. Therefore, we explored the novel interactive theatre experience supported in the embodied mixed reality space, and implemented live 3D characters to interact with user in such a system.

We conducted two experiments comparing communication behaviors of co-located users in collaborative augmented reality (AR) interfaces. In the first experiment, we compared optical, stereo- and mono-video, and immersive head mounted displays (HMDs) using a target identification task. It was found that differences in the real world visibility severely affect communication behaviors. The optical see-through case produced the best results with the least extra communication needed. Generally, the more difficult it was to use non-verbal communication cues, the more people resorted to speech cues to compensate. In the second experiment, we compared three different combinations of task and communication spaces using a 2D icon design task with optical see-through HMDs. It was found that the spatial relationship between the task and communication spaces also severely affected communication behaviors. Placing the task space between the subjects produced the most active behaviors in terms of initiatory body languages and utterances with least miscommunications.

We present a complete system for live capture of 3D content and simultaneous presentation in augmented reality. The user sees the real world from his viewpoint, but modified so that the image of a remote collaborator is rendered into the scene. Fifteen cameras surround the collaborator, and the resulting video streams are used to construct a three-dimensional model of the subject using a shape-from-silhouette algorithm. Users view a two-dimensional fiducial marker using a video-see-through augmented reality interface. The geometric relationship between the marker and head-mounted camera is calculated, and the equivalent view of the subject is computed and drawn into the scene. Our system can generate 384 /spl times/ 288 pixel images of the models at 25 fps, with a latency of < 100 ms. The result gives the strong impression that the subject is a real part of the 3D scene. We demonstrate applications of this system in 3D videoconferencing and entertainment.

In its simplest form, reality is merely information that is presented or acquired. Mixed Reality (MR) is built around the integration of real world physical and computer generated virtual information. We do not use the term augmented reality (AR) because we view the merging of both worlds as a symbiosis, with desirable properties from each accentuated and complementing each other, rather than the enhancement of one with the other. Collaborative MR allows multiple participants to simultaneously share a physical space while being surrounded by a virtual space that is registered with the physical. Because the MR world inherits the properties of real and virtual worlds, it is rich with social context, spatial cues, and tangible objects from the real world as well as flexible digital information from virtual. We believe that Mixed Reality is a medium, largely unexplored, but very well suited for face-to-face collaboration.

We present a wide area human tracking method using distributed computer vision systems. Each vision system consists of a camera and an image processor and they are all connected through a computer network. In this paper, we propose a method for human tracking and for coordination of all the vision systems. The human tracking method works on each vision system and uses a type of model based template matching to track moving people at 15 frame/sec on a standard personal computer. Coordination between the vision systems is necessary to achieve consistent wide area tracking. We use a state transition map and several action rules to synchronize the image processing between systems. All the vision systems share the state transition map jointly and decide their own actions according to the action rules. We describe experimental results that show the validity of our approach.

A recognition method for roughly hand-drawn diagrams based on hybrid utilization of multilayered knowledge is described. This method is free from various drawing constraints which previous systems cannot avoid. It is based on two concepts: (1) no strict segmentation should be performed-various hypotheses of symbols should be considered; and (2) various kinds of knowledge should be utilized-ambiguity of a figure can thus be resolved. Experimental results show that this method is effective for the recognition of roughly hand-drawn logical diagrams.>