As a pivotal application of Augmented Reality (AR) technology, AR games empower players to bridge reality with virtuality, offering a distinct and immersive experience set apart from traditional games. However, when creating AR games, one of the most formidable challenges faced by designers pertains to the unpredictability of intricate real-world environments, which hinders crafting naturally integrated scenes where virtual objects harmoniously blend with the players' surroundings. In this paper, we introduce EverywhereAR, a system that is capable of flexibly realizing the designer's idea in various real-world scenes. It provides a designer-friendly Game Scene Template development interface, for designers to quickly graphify their inspirations. To achieve the best AR game scene, this work proposes a highly customizable integration method. According to the integrated AR scene graph, the system will arrange each virtual object in a reasonable position to make the generated game scene look natural. We conducted an experiment to evaluate our system's performance across various game scene templates and real-world environments. Results from the experiment indicated that our system was able to generate AR game scenes matching the quality of scenes manually created by professional designers. In addition, we conducted another experiment to assess the effectiveness and usability of the proposed interface. The experiment results showed that the interface was intuitive and efficient, allowing users to create a simple game scene within one minute.

Performing tasks in a close range using augmented content or instructions visualized on a 2D display can be difficult because of missing visual information in the third dimension. This is because the world on the screen is rendered from the perspective of a single camera, typically on the device itself. However, when performing tasks using hands, haptic feedback supports vision, and prior knowledge and visual context affect task performance. This study rendered the world on a display from the user's perspective to re-enable depth cues from motion parallax and compared it with the conventional device perspective during haptic interactions. We conducted a user study involving 20 subjects and two experiments. First, the accuracy of touchpoint and depth estimation was measured under the conditions of a visual context and perspective rendering on a magic-lens display. We found that user-perspective rendering slightly improved the touch accuracy of targets on a physical surface; however, it significantly improved interactions without tactile feedback. This effect is relatively large when contextual information from the environment is absent, and it diminishes with increased haptic interactions. In the second experiment, we used a user-perspective magic lens to validate the proposed method in a practical needle injection scenario and confirm that the initial injections to virtual targets were more accurate. The results indicate that user-perspective rendering on magic lenses improves immediate performance in haptic tasks, suggesting they are particularly advantageous for frequently changing environments or short-duration tasks.

Augmented reality (AR) magic-lens (ML) displays, such as handheld devices, offer a convenient and accessible way to enrich our environment using virtual imagery. Several display technologies, including conventional monocular, less common stereoscopic, and varifocal displays, are currently being used. Vergence and accommodation effects on depth perception, as well as vergence-accommodation conflict, have been studied, where users interact only with the content on the display. However, little research exists on how vergence and accommodation influence user performance and cognitive-task load when users interact with the content on a display and its surroundings in a short timeframe. Examples of this are validating augmented instructions before making an incision and performing general hand-eye coordinated tasks such as grasping augmented objects. To improve interactions with future AR displays in such scenarios, we must improve our understanding of this influence. To this end, we conducted two fundamental visual-acuity user studies with 28 and 27 participants, while investigating eye vergence and accommodation distances on four ML displays. Our findings show that minimizing the accommodation difference between the display and its surroundings is crucial when the gaze between the display and its surroundings shifts rapidly. Minimizing the difference in vergence is more important when viewing the display and its surroundings as a single context without shifting the gaze. Interestingly, the vergence-accommodation conflict did not significantly affect the cognitive-task load nor play a pivotal role in the accuracy of interactions with AR ML content and its physical surroundings.

Augmented Reality (AR) has long been expected to help users improve their working efficiency. However, due to the absence of intelligent systems, existing AR applications are greatly affected by the virtual content interference with real-world activities. Unlike existing work, which focuses more on hiding virtual content to reduce interference, in this work, we propose an innovative AR Task Support System where virtual contents actively guide users with task completion. During task execution, our system proactively searches for and tracks key objects in the scene, and uses this context information to automatically select appropriate virtual content and display positions. Through introducing open-world prompt-based visual models, our system can effectively retrieve few-shot or even zero-shot objects that are uncommon in the dataset. This approach extends the use of AR Task Support System beyond controlled industrial settings to more uncontrolled daily scenarios, overcoming the limitations of existing systems. It also significantly reduces development costs for developers. We demonstrate the advantages of our system over traditional virtual content management systems through a series of experiments that are closer to users' real usage situations.

Abstract Lenslet array near-eye displays are a revolutionary technology that generates a virtual image in the field of view of the observer. Although this technology is advantageous in creating compact near-eye displays, undesirable artifacts occur when the user pupil moves outside of the pupil practical movable region (PPMR). Even with dynamic image updating based on eye-tracking techniques, artifacts can still be perceived when human eyes turn rapidly. To enlarge PPMR, we proposed a new rendering method in previous work. To improve the rendering speed in the eye tracking system, look-up tables are used. The disadvantage of the onboard system is the large memory consumption. In this study, we analyzed the system parameters of the incident pupil and pupil margin light columns, the feasibility of the optimized system, and evaluated the optimized system that can adapt to the maximum velocity of the saccadic pupil movement. We optimized the rendering method to reduce memory consumption in the process of generating microdisplay images. In addition, we provide GPU rendering method to improve system speed and reduce system latency to meet the maximum human eye rotation speed. We conducted user studies to evaluate the effect of the method using the optimized rendering method combined with eye tracking to reduce artifacts for fast eye rotation on different images and videos. Results showed that our method effectively reduced artifacts via the optimized rendering method with eye tracking, which adapted to faster human eye movements.

This study proposes a communication robot that plays video games together with the user to create a fun atmosphere for a daily-use robot. In recent years, the lack of conversation in daily life has become a problem, and as a solution to this problem, a dialogue robot have attracted attention as a substitute for a human and as a means of encouraging daily speech. In this study, we propose a dialogue robot that plays a video game with a user in order to produce enjoyable daily communication with the robot. The proposed method not only controls the robot’s speech, but also controls the game development by manipulating the game characters. The proposed robot enhances a user’s enjoyment by adjusting the level of game manipulation to produce a competitive match and by speaking according to the game scene. A robot that produces speech that makes a player feel positive emotions is called a friendly robot. In order to examine the effects of these robots on a user’s enjoyment, we conducted an experiment with 30 subjects in which they were subjectively evaluated by questionnaires in two types of matches: with a friendly robot and alone. The results showed that robots were more enjoyable than playing alone.

Previous research has shown that many factors in a virtual environment influence the way, how we perceive interaction with humanoid agents. It is still unknown if visual stereotypes also translate into human-robot interaction scenarios. To investigate this relationship, we conducted a two-part user study with 26 participants using the Stereotype Content Model (SCM) to identify different stereotypical avatars and visual features in terms of their effect on the perception of robot touch. We found that humanoid avatars which resemble the robot and are perceived as warm in the SCM are more liked and preferred than cold avatars. We also found different effects of visual appearance on touch perception. This suggests that a humanoid-stereotpyical appearance of a touch-care robot in virtual reality can change the overall perception of the interaction. We propose the use of stereotypical warm avatars as described in the SCM for humanity-based applications.

In this study, we analyzed the human "gentle stroke with speech behavior" for comfortable motion by implementing the model into a robot. The results of 19 subjects’ human analysis showed that there was a positive correlation between stroke speed and speech speed. The faster the speech speed, the larger the variance of each feature. Moreover, it was also estimated regardless of the length of each sentence, people tend to fold their hands at "breaks in sentences" or "points just before breaks in sentences". Based on this human-modeled knowledge, we evaluated a human comfort experiment in 36 subjects using a robot arm to reproduce the "stroke and speech" behavior at different speeds. Results of the questionnaire showed that subjects felt more comfortable when stroke speed and speech speed were positively correlated. This result suggests that people consciously adjust their stroking and speaking styles when they perform "comfortable stroke with speech behavior".

Recent augmented reality (AR) advancements have enabled the development of effective training systems, especially in the medical, rehabilitation, and industrial fields. However, it is unclear from the literature what the intrinsic value of AR to training is and how it differs across multiple application fields. In this work, we gathered and reviewed the prototypes and applications geared towards training the intended user's knowledge, skills, and abilities. Specifically, from IEEE Xplore plus other digital libraries, we collected 64 research papers present in high-impact publications about augmented reality training systems (ARTS). All 64 papers were then categorized according to the training method used, and each paper's evaluations were identified by validity. The summary of the results shows trends in the training methods and evaluations that incorporate ARTS in each field. The narrative synthesis illustrates the different implementations of AR for each of the training methods. In addition, examples of the different evaluation types of the current ARTS are described for each of the aforementioned training methods. We also investigated the different training strategies used by the prevailing ARTS. The insights gleaned from this review can suggest standards for designing ARTS regarding training strategy, and recommendations are provided for the implementation and evaluation of future ARTS.

Nowadays, decreasing opportunities for people to have daily conversations due to the increase in the number of withdrawn young people and households living alone. The lack of daily conversation has been pointed out as a risk that can lead to mental problems such as depression, and serious health problems such as dementia for the elderly. Efforts to encourage daily communication by having communication robots that act as talking partners is attracting attention to solve those problems. One of the challenges of communication robots is the difficulty of maintaining users' motivation to continue using robots. In this study, we propose the communication robot that plays a video game together with a user as an approach to keep the user's motivation high to use the robot. The proposed method not only controls the dialogue content of the robot but also controls the video game situation by manipulating the video game character. In this system, we aim to create an atmosphere where users can enjoy playing video games with the robot together, and the proposed game communication robot can keep their motivation high to use the robot.

Projector-camera systems have long been used in measuring three-dimensional shapes. Most projector-camera systems can only be used in dark rooms because frame-based cameras are not robust against strong ambient light and are difficult to obtain correspondence to the image pixels of the projector. Recently, event cameras, which can detect the direction of luminance change, have received attention in the field of computer vision. When considering the many advantages of event cameras, this study focuses on their wide dynamic range (120 vs. 40 dB of a frame-based camera) and their ability to detect fast luminance changes. Our objective is to realize a projector-camera system that combines the event camera with a projector under the strong ambient light. Specifically, this study proposes a new structured light that combines different frequencies of flickers to acquire the correspondence between the image pixels of the event camera and the projector. This method does not rely on the co-axial frame-based measurement and synchronization mechanism between projector and camera and is thus applicable to most general event cameras. Experiments confirm that the proposed method obtains the correspondence robustly with reasonable accuracy in a bright room (up to 2,600 lux) under general indoor lighting and additional light projection.

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user’s ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user’s performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training.

Triangle meshes are used in many important shape-related applications including geometric modeling, animation production, system simulation, and visualization. However, these meshes are typically generated in raw form with several defects and poor-quality elements, obstructing them from practical application. Over the past decades, different surface remeshing techniques have been presented to improve these poor-quality meshes prior to the downstream utilization. A typical surface remeshing algorithm converts an input mesh into a higher quality mesh with consideration of given quality requirements as well as an acceptable approximation to the input mesh. In recent years, surface remeshing has gained significant attention from researchers and engineers, and several remeshing algorithms have been proposed. However, there has been no survey article on remeshing methods in general with a defined search strategy and article selection mechanism covering the recent approaches in surface remeshing domain with a good connection to classical approaches. In this article, we present a survey on surface remeshing techniques, classifying all collected articles in different categories and analyzing specific methods with their advantages, disadvantages, and possible future improvements. Following the systematic literature review methodology, we define step-by-step guidelines throughout the review process, including search strategy, literature inclusion/exclusion criteria, article quality assessment, and data extraction. With the aim of literature collection and classification based on data extraction, we summarized collected articles, considering the key remeshing objectives, the way the mesh quality is defined and improved, and the way their techniques are compared with other previous methods. Remeshing objectives are described by angle range control, feature preservation, error control, valence optimization, and remeshing compatibility. The metrics used in the literature for the evaluation of surface remeshing algorithms are discussed. Meshing techniques are compared with other related methods via a comprehensive table with indices of the method name, the remeshing challenge met and solved, the category the method belongs to, and the year of publication. We expect this survey to be a practical reference for surface remeshing in terms of literature classification, method analysis, and future prospects.

Accurately placing virtual objects in a scene is a challenging tasks in handheld augmented reality (HAR). To add and arrange virtual objects in HAR, users must manipulate 6 degrees of freedom (DoFs) of the virtual object, namely: position (3) and orientation (3). However, it is difficult to manipulate all DoFs with the two-dimensional display of the handheld device. We present SlidAR+, a method for controlling the position and orientation of objects in HAR. SlidAR+ is an extension of SlidAR [1], a technique that allows users to control the position of a virtual object by manipulating only 1 DoF. We use the direction of gravity as a constraint to improve the user’s control and reduce the time it takes to adjust the orientation. Upon comparing it with a state-of-the-art object manipulation method, using SlidAR+, user were able to complete the tasks faster under our expected conditions and were also preferred by most participants.

Augmented reality (AR) systems support complex tasks like assembly by overlaying task-related content onto the real world. In recent years, the effort of designing and developing assembly task support systems in AR decreased with the availability of high potential head-mounted displays and provision of integrated development environments. Nevertheless, problems still arise when companies craft an effective AR task support system, particularly in the difficulty of selecting appropriate techniques and information-presentation methods, and the requirements that vary with each use case. In this study, we formulated a corresponding guideline, developed a selection aid tool that incorporates filtering based on the categorization of subtasks and the degree of freedom of available tracking, and evaluated their effectiveness in two experiments. First, to confirm effects on system design, we asked 18 participants to perform the design action of the AR system with the guideline for two tasks (PC assembly and rope work). Consequently, to verify the quality of the designed AR systems from Experiment 1, we asked another set of 20 participants to perform the same tasks with those systems. The results confirm that using the guideline can considerably lower efforts creating media and alleviate the error for a specific process. We envision our guideline and tool to be accessible as an online web page, assisting AR assembly task support system designer/developers worldwide.

With projection mapping, we can control the appearance of real-world objects by adding illumination. A projector can be used to control light reflected from an object, where the reflected light depends not only on the projection but also on the reflectance and environmental light. Because the resulting colors are affected by the reflectance and environmental light, the presentable color range of a projector is limited. The purpose of this work is to broaden this limited range by focusing on the perceived colors. Although our eyes capture reflected light to perceive the colors of an object, the colors perceived by humans are not always the same as the actual colors, and there are often significant differences between them because of the human visual system. To overcome the limitations of a projector based on human perception, we intentionally generate this difference by inducing a visual illusion, namely, color constancy. In this work, we designed an algorithm to determine the projected colors for presenting the desired colors perceptually by employing a color constancy effect. In addition, we conducted a user study and confirmed that our algorithm can (1) create a misperception regarding the color of illumination, (2) broaden the presentable color range of a projector, and (3) shift the perceptual colors in the desirable direction.

Humanitude is a multimodal communication care method that utilizes seeing, touching, and speaking. Moreover, a touch-care method is well known as an effective care method mainly focus on touch motion. These kinds of care techniques are effective in practical situations, however, it is difficult to provide such care therapy to all patients due to the lack of human resources. To address this problem, researchers try to develop a touch-care robot that can provide touch-care automatically. Conventional research of touch-care robot mainly focuses on the movement of stroke or the speech that only considers the impression of the contents of speech but not prosodic information. Therefore, in this research, we focus on the speech rate in the prosodic information with stroke motion. In this work, we investigate the effects of speech rate on the prosodic information and evaluate the relationship between stroke pace and speech rate to improve human comfort. We conducted a user study with 6 participants around 20 years old males. As a result of the list of the questionnaire suggests a correlation between stroke pace and speech rate that provides comfort.

This paper proposes a system to enhance the degree of subjective deliciousness of food visually perceived by a person by automatically changing its appearance in a real environment. The system is called the Food Appearance Optimizer. The proposed system analyzes the appearance of food in a camera image and projects an appropriate image onto the food using a projector. The relationship between the degree of subjective deliciousness and four simple appearance features for each food category is modeled using data gathered via a crowdsourcing-based questionnaire. Using this model, the system generates the appropriate projection image to increase the deliciousness of the food.

Projection-based Augmented Reality commonly projects on rigid objects, while only few systems project on deformable objects. In this paper, we present Pseudo Printed Fabrics (PPF), which enables the projection on a deforming piece of cloth. This can be applied to previewing a cloth design while manipulating its shape. We support challenging manipulations, including heavy occlusions and stretching the cloth. In previous work, we developed a similar system, based on a novel marker pattern; PPF extends it in two important aspects. First, we improved performance by two orders of magnitudes to achieve interactive performance. Second, we developed a new interpolation algorithm to keep registration during challenging manipulations. We believe that PPF can be applied to domains including virtual-try on and fashion design.

We demonstrate the geometrically-correct projection-based texture mapping onto a deformable object like a cloth. This system can be used to simulate design that involves change in shape, such as sheets of malleable material. The geometrically-correct projection-based texture mapping onto a cloth is conducted using the measurement of object's 3D shape and the detection of the retro-reflective marker on the object's surface. Rapid prototyping is used as an example application of this projection technique.

Projection-based Augmented Reality commonly employs a rigid substrate as the projection surface and does not support scenarios where the substrate can be reshaped. This investigation presents a projection-based AR system that supports deformable substrates that can be bent, twisted or folded. We demonstrate a new invisible marker embedded into a deformable substrate and an algorithm that identifies deformations to project geometrically correct textures onto the deformable object. The geometrically correct projection-based texture mapping onto a deformable marker is conducted using the measurement of the 3D shape through the detection of the retro-reflective marker on the surface. In order to achieve accurate texture mapping, we propose a marker pattern that can be partially recognized and can be registered to an object’s surface. The outcome of this work addresses a fundamental vision recognition challenge that allows the underlying material to change shape and be recognized by the system. Our evaluation demonstrated the system achieved geometrically correct projection under extreme deformation conditions. We envisage the techniques presented are useful for domains including prototype development, design, entertainment and information based AR systems.

The objective of this study is to investigate the relationship between the features of augmented reality (AR) and human memorization ability. The basis of this relation is derived from the following features. The AR feature is that AR can provide information associated with specific locations in the real world. The feature of human memory is that humans can easily memorize information if the information is visually associated with specific locations. To investigate this relation, we conduct a pilot user study in which blocks are picked from some drawers. As a result, significant differences are found between a situation in which visual information is displayed at the location of each drawer in the real world and that in which textual information is displayed at an unrelated location.