Hydraulic excavators are widely used in construction work owing to their versatility. However, the general operation of these excavators is complex and novice operators require extensive training to operate them. In this study, we propose a virtual reality (VR)-based training system with three types of visual augmentations using pre-recorded expert operations to support the skill acquirement for handling a hydraulic excavator. To evaluate the effectiveness of the proposed visual augmentations in terms of skill improvement, we compared the scores of the trainees before and after training including combinations of visual augmentations. The results indicated that the display of the lever movement significantly improved the trajectory of the bucket tip, while ghost animation and slow motion did not show significant effects. Furthermore, by showing the lever input and excavator movement of the expert in slow motion, the task completion time increased because of the aftereffect. Our findings not only provide a design guideline for VR-based excavator operation training but can also be applied to augmented reality (AR)/mixed reality (MR) support systems for supporting practical excavator operations.

We proposed an interactive map applicationOTASCE Mapwhich presents map and route information. It utilizes customizable audio-tactile elements when the user traces the map on the touchscreen or the displayed route with their finger. However despite the unique interaction on the OTASCE Mapthe quantitative interaction situation and the operation requirements under screen-reading conditions remain unclear. Thereforein this studywe aimed to clarify the actual situation of route-tracing interaction on the smartphone-based map application and to derive efficiency improvement factors. At the exhibition for the visually impaired in Japan we evaluated the traceability of the route on the OTASCE Map. The result indicated that most participants could significantly trace the route and reach the destination when they did not perform release-retouch-trace (RRT) interaction. Herein we also discuss the cause of RRT interaction and how to decrease such interactions.

In this paper, we present a review of the competition track organized by us at IPIN 2017. Our competition track was named as “PDR challenge in warehouse picking”. Our pedestrian dead-reckoning (PDR) challenge focused on testing the PDR methods under a realistic scenario, which is tracking employees during the picking operation in a warehouse. Firstly, we introduce an overview of the PDR challenge including detailed information of all evaluation metrics. Secondly, we summarize the results and findings from the competition and list the challenges that should be tackled in the next competition. Thirdly, we introduce the design concept of an xDR challenge by reflecting the findings and feedback of design of the competition. The target of the PDR challenge was only the employees who move by foot during working in the warehouse. The main update in the xDR challenge is dealing with forklifts driven by employees as well as walking employees. We term dead-reckoning of a vehicle as vehicle dead-reckoning (VDR). The term xDR is derived from PDR plus VDR. Another update is providing reference data for the reconstructing situation of picking operation in the warehouse by using an IMU-based motion capture system.

Creating and animating realistic 3D human faces is an important element of virtual reality, video games, and other areas that involve interactive 3D graphics. In this paper, we propose a system to generate photorealistic 3D blendshape-based face models automatically using only a single consumer RGB-D sensor. The capture and processing requires no artistic expertise to operate, takes 15 seconds to capture and generate a single facial expression, and approximately 1 minute of processing time per expression to transform it into a blendshape model. Our main contributions include a complete end-to-end pipeline for capturing and generating photorealistic blendshape models automatically and a registration method that solves dense correspondences between two face scans by utilizing facial landmarks detection and optical flows. We demonstrate the effectiveness of the proposed method by capturing different human subjects with a variety of sensors and puppeteering their 3D faces with real-time facial performance retargeting. The rapid nature of our method allows for just-in-time construction of a digital face. To that end, we also integrated our pipeline with a virtual reality facial performance capture system that allows dynamic embodiment of the generated faces despite partial occlusion of the user's real face by the head-mounted display.

This paper presents a novel method for detecting the floor level of a smartphone user in an indoor environment based on the atmospheric pressure measured with sensors in the user's smartphone. Atmospheric pressure is one of the most informative factors when estimating altitude, because its relative variation has a strong correlation with changes in altitude. However, it is difficult to directly estimate the altitude based on an observation of the absolute pressure via smartphones, owing to individual differences in the sensors and changes in climate. In this paper, we propose a robust method for estimating the floor level by decomposing the observed pressure into three components. The first component is the change in altitude, and this component is the primary measure. The second is the global variation, such as changes in climate. The third concerns device-dependent variation. We utilize localization infrastructure, such as beacons or Wi-Fi access points, sparsely located in the environment, because such localization mechanisms can provide floor-level information. Exploiting this mechanism, time-series references to the pressure on each floor level can be generated collectively by aggregating readings from the pressure sensors of multiple users. Using these as a reference, each output of the pressure can be decomposed into device-specific offsets, environmental trends, and the altitude-dependent component. By extracting the altitude-dependent component, the floor level of the user can be estimated robustly without relying on fixed observations of atmospheric pressure.

We have been developing a mixed reality system to support road maintenance using overlaid visual aids. Such a system requires a positioning method that can provide sub-meter accuracy and function even if the appearance of the road surface changes significantly caused by many factors such as construction phase, time and weather. Therefore, we are developing a real-time worker positioning method that can be applied to these situation by integrating laser range finder (LRF) and pedestrian dead-reckoning (PDR) data. In the field, multiple workers move around the workspace. Therefore, it is necessary to determine corresponding pairs of PDR-based and LRF-based trajectories by identifying similar trajectories. In this study, we propose a method to calculate the similarity between trajectories and a procedure to integrate corresponding pairs of trajectories to acquire the position and movement direction of a worker.

Creating and animating a realistic 3D human face has been an important task in computer graphics. The capability of capturing the 3D face of a human subject and reanimate it quickly will find many applications in games, training simulations, and interactive 3D graphics. In this paper, we propose a system to capture photorealistic 3D faces and generate the blendshape models automatically using only a single commodity RGB-D sensor. Our method can rapidly generate a set of expressive facial poses from a single Microsoft Kinect and requires no artistic expertise on the part of the capture subject. The system takes only a matter of seconds to capture and produce a 3D facial pose and only requires 4 minutes of processing time to transform it into a blendshape model. Our main contributions include an end-to-end pipeline for capturing and generating face blendshape models automatically, and a registration method that solves dense correspondences between two face scans by utilizing facial landmark detection and optical flow. We demonstrate the effectiveness of the proposed method by capturing 3D facial models of different human subjects and puppeteering their models in an animation system with real-time facial performance retargeting.

We present a process for rendering a realistic facial performance with control of viewpoint and illumination. The performance is based on one or more high-quality geometry and reflectance scans of an actor in static poses, driven by one or more video streams of a performance. We compute optical flow correspondences between neighboring video frames, and a sparse set of correspondences between static scans and video frames. The latter are made possible by leveraging the relightability of the static 3D scans to match the viewpoint(s) and appearance of the actor in videos taken in arbitrary environments. As optical flow tends to compute proper correspondence for some areas but not others, we also compute a smoothed, per-pixel confidence map for every computed flow, based on normalized cross-correlation. These flows and their confidences yield a set of weighted triangulation constraints among the static poses and the frames of a performance. Given a single artist-prepared face mesh for one static pose, we optimally combine the weighted triangulation constraints, along with a shape regularization term, into a consistent 3D geometry solution over the entire performance that is drift free by construction. In contrast to previous work, even partial correspondences contribute to drift minimization, for example, where a successful match is found in the eye region but not the mouth. Our shape regularization employs a differential shape term based on a spatially varying blend of the differential shapes of the static poses and neighboring dynamic poses, weighted by the associated flow confidences. These weights also permit dynamic reflectance maps to be produced for the performance by blending the static scan maps. Finally, as the geometry and maps are represented on a consistent artist-friendly mesh, we render the resulting high-quality animated face geometry and animated reflectance maps using standard rendering tools.

We focus on a challenging trial of a Look-Change mixed reality (MR) space. Look refers to perceived ambience of an image created by illumination and color tone. Regarding the Look-Change, we have been working on a relighting technique which allows MR space to have additional virtual illumination. Furthermore, we demonstrate a photorealistic approach to create photorealistic virtual objects with free-form polarized reflectometry which requires no specialized hardware and is portable, thus making it easy apply to MR.

PreViz is a computer graphics movie that represents desired scenes in preproduction and is useful for sharing the director's imagination among crews. Therefore, film industry looks upon PreViz as one of the most important processes in filmmaking as the processes become complicated [The Previs Society 2011]. Stereoscopic 3D (S3D) movie production not only changed ways of expressiveness but also changed ways of production and tools. Consequently, it requires more careful planning and PreViz is still important. This paper describes a prototype of S3D MR-PreViz system for HD S3D PreViz shooting using mixed reality (MR) technologies in order to facilitate PreViz making in S3D. This system is designed as an extension of MR-PreViz for classic filmmaking [Ichikari et al. 2010].

In this paper, we introduce the outline of "The MR-PreViz Project" performed in Japan. In the pre-production process of filmmaking, Pre Viz, pre-visualizing the desired scene by CGI, is used as a new technique. As its advanced approach, we propose MR-PreViz that utilized mixed reality technology in current PreViz. MR-PreViz makes it possible to merge the real background and the computer-generated humans and creatures in open set or at outdoor location. The user can consider the camerawork and camera blocking efficiently by using MR-PreViz. This paper introduces the outline of the MR-PreViz project, the design of hardware configuration, camera-work authoring method and the results of prototyping.