To extend the coverage of people able to receive high-quality rehabilitation, remote rehabilitation is required in addition to traditional face-to-face rehabilitation. Although remote rehabilitation using video conferencing systems has been realized to date, communication through physical sensations such as detailed patient motoring information and manual instructions from the therapist has not yet been realized. Therefore, the ultimate goal of this study was to develop multimodal wearable sensor system to support remote rehabilitation with a somatosensory system. To this end, we conducted a basic study of sensing technology in multimodal wear. Multiple strain sensors were attached to the shoulder to digitize the detailed behavior of the shoulder girdle. It was confirmed that the movement of the scapula can be acquired by this strain sensor. Furthermore, it was confirmed that the combination of strain sensors and an inertia measurement unit can be applied for the motion estimation of the entire upper limb.

We proposed an interactive map applicationOTASCE Mapwhich presents map and route information. It utilizes customizable audio-tactile elements when the user traces the map on the touchscreen or the displayed route with their finger. However despite the unique interaction on the OTASCE Mapthe quantitative interaction situation and the operation requirements under screen-reading conditions remain unclear. Thereforein this studywe aimed to clarify the actual situation of route-tracing interaction on the smartphone-based map application and to derive efficiency improvement factors. At the exhibition for the visually impaired in Japan we evaluated the traceability of the route on the OTASCE Map. The result indicated that most participants could significantly trace the route and reach the destination when they did not perform release-retouch-trace (RRT) interaction. Herein we also discuss the cause of RRT interaction and how to decrease such interactions.

Several audio games have been developed for visually impaired players to enjoy only by using audio information. However, sighted persons find it challenging to play these games because they mainly use visual information. To bridge the gap between them, developing games that appropriately present multisensory modalities: To devise an auditory and tactile presentation method that allows the sighted to manipulate and identify features, enabling them to play without visual information. Therefore, we develop an inclusive game that both sighted and visually impaired people can play. Here, we developed a real-time sensory substitution method using auditory and tactile senses, including a tactile game controller (TactCon). Then, we implemented the method in a 2D side-scrolling action game and evaluated the game by either sighted or visually impaired people at a two-month exhibition: not a controlled experiment. Results show that immediate auditory and tactile presentation can be important for visually impaired and sighted gamers to play action games requiring real-time response. Furthermore, we considered how the presentation of moving objects can be understood in an inclusive action game.

In this paper, we present a review of the competition track organized by us at IPIN 2017. Our competition track was named as “PDR challenge in warehouse picking”. Our pedestrian dead-reckoning (PDR) challenge focused on testing the PDR methods under a realistic scenario, which is tracking employees during the picking operation in a warehouse. Firstly, we introduce an overview of the PDR challenge including detailed information of all evaluation metrics. Secondly, we summarize the results and findings from the competition and list the challenges that should be tackled in the next competition. Thirdly, we introduce the design concept of an xDR challenge by reflecting the findings and feedback of design of the competition. The target of the PDR challenge was only the employees who move by foot during working in the warehouse. The main update in the xDR challenge is dealing with forklifts driven by employees as well as walking employees. We term dead-reckoning of a vehicle as vehicle dead-reckoning (VDR). The term xDR is derived from PDR plus VDR. Another update is providing reference data for the reconstructing situation of picking operation in the warehouse by using an IMU-based motion capture system.

The present paper addresses the problem of analyzing the work of forklift trucks. To solve the problem, a forklift truck working model based on a state transition model is constructed using human activity sensing data, where the key point is using forklift truck operating data to determine human intention. The novel contribution of the model is its use of data that express the intention of the forklift truck operator to categorize the work of the forklift truck. By using a graph model, the proposed method can be extended to analyzing the human characteristics of forklift truck work. With this technique, some features of the graph model can be used to distinguish the skill level of the forklift truck operator.

This paper presents a novel method for detecting the floor level of a smartphone user in an indoor environment based on the atmospheric pressure measured with sensors in the user's smartphone. Atmospheric pressure is one of the most informative factors when estimating altitude, because its relative variation has a strong correlation with changes in altitude. However, it is difficult to directly estimate the altitude based on an observation of the absolute pressure via smartphones, owing to individual differences in the sensors and changes in climate. In this paper, we propose a robust method for estimating the floor level by decomposing the observed pressure into three components. The first component is the change in altitude, and this component is the primary measure. The second is the global variation, such as changes in climate. The third concerns device-dependent variation. We utilize localization infrastructure, such as beacons or Wi-Fi access points, sparsely located in the environment, because such localization mechanisms can provide floor-level information. Exploiting this mechanism, time-series references to the pressure on each floor level can be generated collectively by aggregating readings from the pressure sensors of multiple users. Using these as a reference, each output of the pressure can be decomposed into device-specific offsets, environmental trends, and the altitude-dependent component. By extracting the altitude-dependent component, the floor level of the user can be estimated robustly without relying on fixed observations of atmospheric pressure.

We have been developing a mixed reality system to support road maintenance using overlaid visual aids. Such a system requires a positioning method that can provide sub-meter accuracy and function even if the appearance of the road surface changes significantly caused by many factors such as construction phase, time and weather. Therefore, we are developing a real-time worker positioning method that can be applied to these situation by integrating laser range finder (LRF) and pedestrian dead-reckoning (PDR) data. In the field, multiple workers move around the workspace. Therefore, it is necessary to determine corresponding pairs of PDR-based and LRF-based trajectories by identifying similar trajectories. In this study, we propose a method to calculate the similarity between trajectories and a procedure to integrate corresponding pairs of trajectories to acquire the position and movement direction of a worker.

In this paper, we propose to use staying area data toward the estimation of serving time for customers. To classify utterances enables us to estimate conversation types between speakers. However, its performance becomes lower in real environments. We propose a method using area data with sound data to solve this problem. We also propose a method to estimate the conversation types using the decision trees. They were tested with the data recorded in a Japanese restaurant. In the experiment to classify utterances, the proposed method performed better than the method using only sound data. In the experiment to estimate the conversation types, we succeeded to recover 70% of the mis-classified conversations using both of sound and area data.

This paper presents a study of a method for estimating the position and orientation of a photo-shoot in indoor environments for augmented reality applications. Our proposed localization method is based on registered frame data of virtualized reality models, which are photos with known photo-shoot positions and orientations, and depth data. Because registered frame data are secondary product of modeling process, additional works are not necessary to create registered frame data especially for the localization. In the method, a photo taken by a mobile camera is compared to registered frame data for the localization. Since registered frame data are linked with photo-shoot position, orientation, and depth data, 3D coordinates of each pixel on the photo of registered frame data is available. We conducted experiments with employing five techniques of the estimation for comparative evaluations.

In the study of AR/MR field, tracking and geometric registration methods are very important topics that are actively discussed. Especially, the study on tracking is flourishing and many algorithms are being proposed every year. With this trend in mind, we, the TrakMark WG, had proposed benchmark test schemes for geometric registration and tracking in AR/MR at ISMAR 2009 [1]. This paper is an intermediate report of the TrakMark WG, which describes its activities and the first proposal on benchmarking image sequences.

We have developed a GUI tool and a mobile MR system for reliving experiences at museums. The GUI tool was developed to provide services based on content made from activity logs and to enable effective analysis of visitors' behaviors using the logs. In addition, by running the GUI tool on a mobile MR system, users can browse visitors' activities with a sensation of realism on the spot. This paper describes the GUI tool and a pilot user study that was conducted to evaluate it.

In this paper, we propose a method of personal positioning for a wearable augmented reality (AR) system that allows a user to freely move around indoors and outdoors. The user is equipped with self-contained sensors, a wearable camera, an inertial head tracker and display. The method is based on sensor fusion of estimates for relative displacement caused by human walking locomotion and estimates for absolute position and orientation within a Kalman filtering framework. The former is based on intensive analysis of human walking behavior using self-contained sensors. The latter is based on image matching of video frames from a wearable camera with an image database that was prepared beforehand.

In this paper, we present a wearable augmented reality (AR) system with personal positioning based on walking locomotion analysis that allows a user to freely mover around indoors and outdoors. The user is equipped with self-contained sensors, a wearable camera, an inertial head tracker and display. The system is based on the sensor fusion of estimates for relative displacement caused by human walking locomotion and estimates for absolute position and orientation within a Kalman filtering framework. The former is based on intensive analysis of human walking behavior using self-contained sensors. The latter is based on image matching of video frames from a wearable camera with an image database that was prepared beforehand.

We propose a novel fiducial-less 3-D object tracking method. Our method consists of three components: 1) bottom-up approach (BUA), 2) top-down approach (TDA), and 3) automatic database addition (ADA). An experimental result shows an accuracy and robustness of our method.

We have developed a method for recognizing and tracking known three-dimensional objects in scene images captured from a user's viewpoint. The method uses multiple color histogram matching and condensation tracking. We implemented a prototype 3-D online manual system based on our method.

This paper describes a fast and robust approach to recovering structure and motion from video frames. It first describes a robust recursive factorization method for affine projection. Using the least median of squares (LMedS) criterion, the method estimates the dominant 3D affine motion and discards feature points regarded as outliers. The computational cost of the overall procedure is reduced by combining this robust statistics-based method with a recursive factorization method that can at each frame provide the updated 3D structure of an object at a fixed computational cost by using the principal component analysis. Experiments with synthetic data and with real image sequences demonstrate that the method can be used to estimate the dominant structure and the motion robustly and in real-time, on an off-the-shelf PC. Finally this paper describes preliminary online experiments on selecting feature points, which have the dominant 3D motion, from live video frames by using a PC-cluster system.

The factorization method, which allows us to reconstruct the motion of the camera and shape of the object simultaneously from multiple images, provides high stability in numerical computations and satisfactory results. To apply this method to real-time processing, the recursive factorization method has been proposed. However, the factorization method based on the affine projection has a limitation in reconstruction accuracy, and to achieve accurate reconstruction, the motion should be restricted. To overcome this problem, we present a recursive factorization method for the paraperspective model based on the perspective projection. The present method is far superior to other ones, in that it not only achieves accurate Euclidean reconstruction in a short time but also provides high stability in numerical computations. Moreover, the method produces stable reconstruction in almost all cases even if some images contain errors because all images are treated as uniformly as possible.

The paper describes a fast and robust image registration method that can be used to create a panoramic image/video from video sequences. To estimate alignment parameters for image registration, the method computes pseudo motion vectors that are rough estimates of optical flows at each selected pixel. Using the proposed method, we implemented a software system that can, with a low-cost PC, create and display panoramic images/videos in real time.

We present the intuitive interpretation of affine epipolar geometry for the orthographic, scaled orthographic, and paraperspective projection models in terms of the factorization method for the generalized affine projection (GAP) model proposed by Fujiki and Kurata (1997). Using the GAP model introduced by Mundy and Zisserman (1992), each affine projection model can be resolved into the orthographic projection model by the introduction of virtual image planes, then the affine epipolar geometry can be simply obtained from the estimates of the factorization, method.