Model-based gaze estimation encounters a technical issue where the corneal center cannot be estimated because of loss of glint. Therefore, we propose a glint-free corneal center estimation that uses polarization. This method determines the incident plane at each pixel, including the surface normal from the polarization state of light, and the intersection point of the incident planes is calculated as the corneal center of the eye. Because polarization states contain noise, the random sample consensus (RANSAC) algorithm was used to remove outliers. The distance error of the corneal center and the accuracy of the gaze vector were evaluated by comparison with those of the conventional model-based method. The mean angular error of the proposed method was within the typical error range of model-based methods, and the feasibility of glint-free corneal center estimation was confirmed through experiments.

The increasing prevalence of Alzheimer’s disease (AD) highlights the need for efficient and early detection. However, traditional cognitive assessments, such as the Mini-Mental State Examination (MMSE), which are administered in person, may not capture subtle early stage impairments. Recent studies have suggested that eye movements can serve as a biomarker for AD. In this study, we examined the use of dynamic time warping (DTW) to analyze smooth-pursuit eye movements to detect cognitive decline in 24 elderly participants, of whom 18 had valid eye tracking data for analysis. DTW was chosen because of its ability to capture the temporal alignment between the target and gaze trajectories. DTW demonstrated a stronger correlation with the MMSE scores (R = -0.782 and R 2 = 0.612) than with the conventional Pearson correlation coefficient (R = 0.456 and R 2 = 0.208). These findings highlight the potential of DTW-based analyses to complement existing cognitive screening tools in clinical settings. Future research should expand the participant pool and explore different smooth-pursuit paradigms to improve the predictive accuracy.

Frustrated total internal reflection (FTIR) imaging is widely applied in various touch-sensing systems. However, vision-based touch sensing has structural constraints, and the system size tends to increase. Although a sensing system with reduced thickness has been developed recently using repropagated FTIR (RFTIR), it lacks the property of instant installation anywhere because observation from the side of a transparent medium is required. Therefore, this study proposes an "RFTIRTouch" sensing device to capture RFTIR images from the contact surface. RFTIRTouch detects the touch position on a dual-sided plane using a physics-based estimation and can be retrofitted to existing transparent media with simple calibration. Our evaluation experiments confirm that the touch position can be estimated within an error of approximately 2.1 mm under optimal conditions. Furthermore, several application examples are implemented to demonstrate the advantages of RFTIRTouch, such as its ability to measure dual sides with a single sensor and waterproof the contact surface.

The 3D point-of-gaze has been calculated using the binocular visual axes in the model-based eye gaze estimation. However, the 3D point-of-gaze is often shifted largely when the visual axis includes certain errors. The distance between the estimated point-of-gaze and the visual axis can be used as an error metric; however, horizontal errors cannot be evaluated, and an additional error metric is required to assess the reliability of the estimated point-of-gaze. Therefore, we propose a novel error metric using the correlation between binocular corneal images. We hypothesize that binocular corneal images extracted around the reflection of the estimated point-of-gaze exhibit a high correlation when the 3D point-of-gaze is correctly estimated. The two approaches were implemented using the corneal images extracted directly and generated based on binocular corneal imaging, and these approaches were evaluated through comparative experiments. We confirmed the feasibility of the proposed error metric and the effectiveness of binocular corneal imaging.

The pupil center and glints have been used in conventional eye-gaze estimations, and the optical axis and point-of-gaze are calculated using these as keys. However, the pupil center used as a basis gets shifted when the diameter of the pupil varies, and the variance in illumination condition leads to degradation of accuracy. Therefore, we propose a semi-automated annotation for iris detection and a model-based gaze estimation method that uses an iris center. The iris area was extracted using segmentation trained with annotated data. The optical axis is determined using the iris center, which does not shift, and is therefore implemented instead of the pupil center. We evaluated the robustness of the proposed gaze estimation through an experiment under changing illumination conditions and confirmed the effectiveness of the iris-based estimation.

Conventional eye-tracking methods require NIR-LEDs at the corners and edges of displays as references. However, extensive eyeball rotation results in the loss of reflections. Therefore, we propose imperceptible markers that can be dynamically displayed using liquid crystals. Using the characteristics of polarized light, the imperceptible markers are shown on a screen as references for eye-tracking. Additionally, the marker positions can be changed using the eyeball pose in the previous frame. The point-of-gaze was determined using the imperceptible markers based on model-based eye gaze estimation. The accuracy of the estimated PoG obtained using the imperceptible marker was approximately 1.69°, higher than that obtained using NIR-LEDs. Through experiments, we confirmed the feasibility and effectiveness of relocating imperceptible markers on the screen.

Current vision-based tactile sensors have several limitations, such as their size and measurable surface. Therefore, we propose a novel vision-based tactile sensor based on the re-propagated frustrated total internal reflection (FTIR). The part of the FTIR generated by the contact is re-propagated through the medium, and the FTIR are observed from the side of the medium. We validate the physical principle of observation, including multiple contact images by simulations. In addition, a prototype system is developed to estimate the contact position through observations and regression algorithms. Finally, several experiments were performed to confirm the feasibility of the proposed contact estimation based on the repropagated FTIR.

Several technical issues that affect eye-tracking have arisen concomitantly with the steadily increasing sizes of personal displays recently. One such issue is the loss of the illumination reflection of the near-infrared light-emitting diodes used as reference points around the edge of the display. Another issue is that reference identification is required for practical usage. Therefore, this paper proposes gaze estimation with transparent markers for large display environments to solve these problems. The transparent markers can be distributed on the screen, and a unique ID is assigned to each marker using linear polarization angles. The reference is detected using a polarization camera through the reflection on the cornea. The results of experiments conducted using a 50-inch display indicate that the proposed method can estimate the point-of-gaze to within 2.1 degrees of error. We confirmed that on-screen markers in sizable displays could be effectively used as references instead of illumination sources.

Soft robotic devices, including actuators fabricated from materials with a low modulus of elasticity, such as silicone elastomers, have gained significant interest in recent years. A flexible sensor is a vital component for estimating the conditions of soft actuators, such as shape, and deformation due to contact events. However, it is challenging to develop a flexible sensor with tolerability and versatility for soft actuators. Additionally, when an embedded sensor is employed, the fabrication process becomes complex. Therefore, to have tolerability and to increase versatility, we propose a method for estimating the shape of a soft pneumatic actuator based on vibroacoustic sensing. We employ a data-driven approach by utilizing several machine-learning techniques; hence, the proposed method could be applied to other types of actuators without employing any specific sensor or changing the fabrication process. The convolutional neural network is used as one of the dominant techniques, and huge annotated datasets are required. However, datasets can be obtained automatically using a robotic system, and estimation targets can be changed smoothly by switching datasets. We confirmed the feasibility, and versatility of the proposed method through several evaluation experiments. The bending angle can be estimated in real-time using active vibroacoustic, by emitting sweep signals. The mean error of bending angle estimation was between 5° to 10°. Furthermore, the proposed method is applied to a tensile actuator, and the mean error of the estimated length was approximately 0.21 mm.

Intelligent electric wheelchairs have been developed for personal mobility, and eye-gaze measurement is essential for comprehending the user's attention and for assisting operations. In previous studies, the gaze vector (i.e., the visual or optical axis of the eye) was projected onto the environmental map; hence, an eye tracker was installed on the wheelchair. In addition, hardware calibration, which determines the geometric relationship between the eye tracker and other sensors, such as LiDAR, was performed beforehand. Recently, wearable eye-trackers are expected to employ a daily-use device; therefore, the cooperation between sensors is essential without geometric constraints. Accordingly, we propose a method for estimating focused pedestrians using smooth-pursuit eye movements in the real world. Pedestrians are tracked using a point cloud obtained with 3D LiDAR, and the trajectories of the focused pedestrian are recorded on an environmental map constructed with simultaneous localization and mapping. Several experiments were conducted to evaluate the computational methods for the correlation between eye movements and moving objects, and we confirmed the feasibility and the current issues through these experiments.

Eye tracking, which measures line of sight, is expected to advance as an intuitive and rapid input method for user interfaces, and a cross-ratio based method that calculates the point-of-gaze using homography matrices has attracted attention because it does not require hardware calibration to determine the geometric relationship between an eye camera and a screen. However, this method requires near-infrared (NIR) light-emitting diodes (LEDs) attached to the display in order to detect screen corners. Consequently, LEDs must be installed around the display to estimate the point-of-gaze. Without these requirements, cross-ratio based gaze estimation can be distributed smoothly. Therefore, we propose the use of a polarization camera for detecting the screen area reflected on a corneal surface. The reflection area of display light is easily detected by the polarized image because the light radiated from the display is polarized linearly by the internal polarization filter. With the proposed method, the screen corners can be determined without using NIR LEDs, and the point-of-gaze can be estimated using the detected corners on the corneal surface. We investigated the accuracy of the estimated point-of-gaze based on a cross-ratio method under various illumination and display conditions. Cross-ratio based gaze estimation is expected to be utilized widely in commercial products because the proposed method does not require infrared light sources at display corners.

In corneal imaging methods, it is essential to use a 3D eyeball model for generating an undistorted image. Thus, the relationship between the eye and eye camera is fixed by using a head-mounted device. Remote corneal imaging has several potential applications such as surveillance systems and driver monitoring. Therefore, we integrated a 3D eyeball model with a 3D face model to facilitate remote corneal imaging. We conducted evaluation experiments and confirmed the feasibility of remote corneal imaging. We showed that the center of the eyeball can be estimated based on face tracking, and thus, corneal imaging can function as continuous remote eye tracking.

Eye-trackers are expected to be used in portable daily-use devices. However, it must register object information and define a unified coordinate system in advance for human--computer interaction and quantitative analysis. Therefore, we propose a semantic 3D gaze mapping to collect gaze information from multiple people on the unified map and detect focused objects automatically. The semantic 3D map can be reconstructed using keyframe-based semantic segmentation and structure-from-motion, and the 3D point-of-gaze can also be computed on the map. We confirmed that the fixation time of the focused object can be calculated through an experiment without prior information.

Corneal imaging has much potential for the development of eye-based interactions. However, it can only provide information on the object being focused on. We therefore propose a localization method based on corneal imaging that exploits the reflections of illumination features from the cornea. A virtual corneal image can be generated from an illumination map, and its similarity to the input eye image can be computed. Global and local localizations are then achieved based on this similarity and a particle filter. The x--, y-- coordinates and θ angle of a participant in a room can thus be estimated practically, as demonstrated experimentally.

Eye-based interaction is one of the solutions for achieving intuitive interfaces on surfaces such as a large display, and thus, various eye-tracking methods have been studied. Cross-ratio based gaze estimation, which determines the point-of-gaze on a screen, has been studied actively as a novel eye-tracking method because the method does not require a hardware calibration defining the relationship between a camera and monitor. We expect that the cross-ratio method will be a breakthrough for eye-based interaction under various circumstances such as tabletop devices and digital whiteboards. In eye-tracking, near-infrared light is often emitted, and at least four LEDs are located on display corners for detecting the screen plane in the cross-ratio based method. However, long-time radiation of near-infrared light can make a user fatigued. Therefore, in this study, we attempted to extract the screen area correctly without near-infrared radiation emission. A polarizing filter is included in the display, and thus, visibility of the screen can be controlled by the light's polarization direction of the external polarized light filter. We propose gaze estimation based on the cross-ratio method using a developed polarization camera system, which can capture two polarized images of different angles simultaneously. Further, we confirmed that the point-of-gaze could be estimated using the screen reflection detected by computing the differences between two images without near-infrared emission.

In this paper, we propose a novel eye-tracking method that uses a multispectral camera to simultaneously track the pupil and recognize the iris. Our hybrid approach leverages existing methods, combining them so as to compensate for weaknesses present in each individual method when used alone. Significantly, our method allows for movements of the center of rotation of the eye to be taken into consideration, this having been treated as a static point in most of earlier studies. Additionally, our method allows for the diameter of the pupil to be measured quantitatively using just a single camera. To confirm the effectiveness of our method, we conduct two experiments, in which we estimate the area and shape of the iris, the point-of-gaze, and the size of the pupil. We go on to observe that the effectiveness of our proposed method is increased compared to previous methods, particularly in situations where the eye is moved to the extreme inner corner of its socket.

Recently, the eye-tracker has been developed as a daily-use device. However, when an eye-tracker is used daily, the problem of calibration arises. Even when the calibration for computing the relationship between the scene and eye camera is conducted in advance, the relationship is not maintained in prolonged use. Therefore, we propose a method for conserving the relationship between the scene and eye camera during the execution of an eye-tracking program. The texture information of the corneal surface image is used to estimate the point-of-regard. We confirm the feasibility of the proposed method through preliminary experiments.

Moving to the location designated by the user is the most fundamental task for a mobile robot. Remote control is one of the effective solutions to navigate the robot to the target location, but the user suffers from the burden to continuously concentrate on the remote control. As the result, several users are required in accordance with the number of robots to operate. In this paper, we propose the remote control system that uses touch panel interface, simultaneous localization and mapping (SLAM), and motion planning to achieve autonomy of mobile robots. To navigate the robot in the proposed system, the user only designates the destination and via-points by touching on the map estimated by the SLAM. After receiving the user's input, the Rapidly-exploring Random Tree (RRT) generates the feasible path to the destination using the estimated map. The effectiveness of the proposed system is verified through experiments where multiple mobile robots are remotely controlled.

Unlike conventional portable eye-tracking methods that estimate the position of the mounted camera using 2-D image coordinates, the techniques that are proposed here present richer information about person's gaze when moving over a wide area. They also include visualizing scanpaths when the user with a head-mounted device makes natural head movements. We employ a Visual SLAM technique to estimate the head pose and extract environmental information. When the person's head moves, the proposed method obtains a 3-D point-of-regard. Furthermore, scanpaths can be appropriately overlaid on image sequences to support quantitative analysis. Additionally, a 3-D environment is employed to detect objects of focus and to visualize an attention map.

Natural body gesturing and speech dialogue, is crucial for human-robot interaction (HRI) and human-robot symbiosis. Real interaction is not only with one-to-one communication but also among multiple people. We have therefore developed a system that can adjust gestures and facial expressions based on a speaker's location or situation for multi-party communication. By extending our already developed real-time gesture planning method, we propose a gesture adjustment suitable for human demand through motion parameterization and gaze motion planning, which allows communication through eye-to-eye contact. We implemented the proposed motion planning method on an android Actroid-SIT and we proposed to use a Key-Value Store to connect the components of our systems. The Key-Value Store is a high-speed and lightweight dictionary database with parallelism and scalability. We conducted multi-party HRI experiments for 1,662 subjects in total. In our HRI system, over 60% of subjects started speaking to the Actroid, and the residence time of their communication also became longer. In addition, we confirmed our system gave humans a more sophisticated impression of the Actroid.

Natural body gesture, as well as speech dialog, is crucial for human-robot interaction and human-robot symbiosis. We have already proposed a real-time gesture planning method. In this paper, we afford this method more flexibility by adding motion parameterization function. Especially in multi-person HRI, this function becomes more important because of its adaptation to changes of a speaker's and/or object's locations. We implement our method for multi-person HRI system on the android Actroid-SIT, and conduct two experiments for estimating the precision of gestures and the human impressions about the Actroid. Through these experiments, we confirmed our method gives humans a more sophisticated impressions.

In this paper, we propose a method for semantic motion retrieval in large data sets of human motions to classify body gestures automatically. This method extracts spatio-temporal features from the motions by expressing them in frequency domain. And these features are transformed into the Bag-of-words representation to accelerate the calculation and to emphasize the semantic aspect. The method is inspired by techniques of natural language processing or image processing. We conducted experiments for evaluating the performance of the motion classification using data sets captured by a motion capture system. Through the experiments, we confirmed that our method improves the performance of the motion classification and reduces the computational time drastically.

Androids are potentially required to show human-like behavior, because their appearance resembles humans' physical features. Therefore, we propose a gaze motion planning method. Within this method, we control the convergence of eyes and the ratio of eye angle to head angle, which leads to a more precise estimation of gaze direction. We implemented our method on the android Actroid-SIT and conducted experiments for evaluation of the effects of our method. Through these experiments, we achieved a common guidance for androids when planning more precise gaze motion.

We propose a method for estimating inherent surface color robustly against image noises from two registered images taken under different outdoor illuminations. We formulate the estimation based on maximum likelihood manner while considering both inter-pixel and intra-pixel relationships. We define inter-pixel relationship based on stochastic behavior of image noises and properties of outdoor illumination chromaticity. We rely on the spatial continuity of both surface color and illumination to define intra-pixel relationship. We also propose to maximize the estimation function in two step manner. Experimental results demonstrate the significant improvement of the proposed method in estimation accuracy compared to previous methods.

The portability of an eye tracking system encourages us to develop a technique for estimating 3D point-of-regard. Unlike conventional methods, which estimate the position in the 2D image coordinates of the mounted camera, such a technique can represent richer gaze information of the human moving in the larger area. In this paper, we propose a method for estimating the 3D point-of-regard and a visualization technique of gaze trajectories under natural head movements for the head-mounted device. We employ visual SLAM technique to estimate head configuration and extract environmental information. Even in cases where the head moves dynamically, the proposed method could obtain 3D point-of-regard. Additionally, gaze trajectories are appropriately overlaid on the scene camera image.

In this research, a Universal map, which can be converted to individual maps for heterogeneous mobile robots, is proposed. A Universal map can be generated using our developed measurement robot, and it is composed of a textured 3D environment model. Therefore, every robot can use a Universal map as a common map, and it is utilized for various localization technologies such as view-based and LRF-based methods. In LRF-based localization, accurate localization is achieved using a specific map, which is generated from Universal map. In a view-based approach, localization and navigation are achieved using rendered images. The use of a Universal map enables generation of these maps automatically. The effectiveness of this approach is confirmed through experiments.

Generating natural motion of an articulated object with higher DOF (e.g., humanoid robot and robot hand) is a crucial issue in robotics and computer graphics fields. Use of the motion capture data is one of the solutions, but it requires expensive device and time-consuming measurement. In this paper, we propose a method for generating natural hand motion to play a piano from the inputted music score. The proposed method uses inverse kinematics while considering naturalness of hand poses. We revisit background of the inverse kinematics based on the maximum likelihood estimation and use the prior model of the hand pose to achieve the naturalness. We evaluate the effectiveness of the proposed method using voluntary survey.

We propose a view-based indoor/outdoor navigation method as an extension of the view-sequence navigation. The original view-sequence navigation method uses the template matching method with normalized correlation for localization. Because the matching method is sensitive to local illumination changes, it is only used for indoor environment. In this paper, we propose to adopt the accumulated block matching method to improve robustness against locally changing illumination, in which a template is split into small patches and matched by maximizing the average of the normalized correlations of all the patches.We also propose a localization criterion which helps the robot decide its motion. Our experimental results demonstrate that the proposed methods can be applied to both indoor and outdoor environments.

In this paper, we propose a method to estimate the group attention that is the focus of attention of multiple people. It utilizes the face direction as 3D vectors and estimates the position of group attention defined as the intersection of the multiple vectors. As a result, the position of the group attention can be represented as an arbitrary 3D position unlike other research in which only registered objects can be the focus of attention. As experiments, the group attention at word-chain game is estimated and the feasibility of the method is confirmed. We applied the proposed method to video conferencing, and developed the attention tracking system which acquires the group attention in real-time.