We present EarEOG, a real-time wheelchair control system using around-ear electrooculogram (EOG) signals. Electrodes are placed in standard over-the-ear headphones to improve user comfort. By detecting around-the-ear signals from eye gestures and jaw clenching, EarEOG offers a non-invasive and intuitive approach to low-latency wheelchair control. We describe the methods for signal acquisition, as well as the algorithms used for signal processing and classification. The feasibility, robustness, and low latency of EarEOG were confirmed through two experiments. The algorithm demonstrated a classification accuracy of 94.1% for all motion signals, which further improved to 97.3% when personalized models were applied. To ensure stability, we examined electrode impedance and algorithm accuracy across multiple trials where participants operated simulated wheelchairs while wearing EarEOG. The results indicated that when the electrode impedance was below 1 MΩ, all participants successfully controlled the simulated wheelchair. Furthermore, EarEOG demonstrated low latency, with recognition delays of less than 125 ms.

Within the last 20 years, gaze interaction has become a successful communication solution for numerous people with motor challenges. In this paper, we present two new cases of gaze interactive assistive technology: i) gaze control of an exoskeleton for stroke rehabilitation, and ii) gaze interactive reading support for people with low vision. By applying a Universal Design approach [Mace 1998] both cases are assessed through an ability analysis to identify issues with gaze interaction specific to our applications that need to be further addressed. Finally, we suggest how solutions in our applications may be mainstreamed for a broader user group.

A dry-electrode head-mounted sensor for visually-evoked electroencephalogram (EEG) signals has been introduced to the gamer market, and provides wireless, low-cost tracking of a user’s gaze fixation on target areas in real-time. Unlike traditional EEG sensors, this new device is easy to set up for non-professionals. We conducted a Fitts’ law study (N = 6) and found the mean throughput (TP) to be 0.82 bits/s. The sensor yielded robust performance with error rates below 1%. The overall median activation time (AT) was 2.35 s with a minuscule difference between one or nine concurrent targets. We discuss whether the method might supplement camera-based gaze interaction, for example, in gaze typing or wheelchair control, and note some limitations, such as a slow AT, the difficulty of calibration with thick hair, and the limit of 10 concurrent targets.

Telepresence robots are increasingly used to support remote social interaction. Telerobots allow the user to move a camera and a microphone at a remote location in real time – often with a display of the user’s face at the robot. These robots can increase the quality of life for people with special needs, who are, for instance, bed bound. However, interface accessibility barriers have made them difficult to use for some people. Still, no state-of-the-art literature review has been made of research on telerobots for people with disabilities. We used Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines for a review. Web of Science (WoS), ACM Digital Library, IEEE Xplore, PubMed, and Scopus were searched, and a supplemental by hand examination of reference lists was done. The search includes studies published between 2009 and 2019.A total of 871 articles were included in this review, 42 of which were eligible for the analysis. These articles were further characterized in terms of problems addressed, objectives, types of special needs considered, features of the devices, features of solutions, and the evaluation methods applied. Based on the review, future research directions are being proposed, addressing issues like use-cases; user conditions; universal accessibility; safety; privacy and security; independence and autonomy; evaluation methods; and user training programs. The review provides an overview of existing research, a summary of common research directions, and a summary of issues, which need to be considered in future research.

In this paper, we have measured cognitive load during an interactive eye-tracking task. Eye-typing was chosen as the task, because of its familiarity, ubiquitousness and ease. Experiments with 18 participants, where they memorized and eye-typed easy and difficult sentences over four days, were used to compare the difficulty levels of the tasks using subjective scores and eye-metrics like blink duration, frequency and interval and pupil dilation were explored, in addition to performance measures like typing speed, error rate and attended but not selected rate. Typing performance lowered with increased task difficulty, while blink frequency, duration and interval were higher for the difficult tasks. Pupil dilation indicated the memorization process, but did not demonstrate a difference between easy and difficult tasks.

Head-mounted displays offer full control over lighting conditions. When equipped with eye tracking technology, they are well suited for experiments investigating pupil dilation in response to cognitive tasks, emotional stimuli, and motor task complexity, particularly for studies that would otherwise have required the use of a chinrest, since the eye cameras are fixed with respect to the head. This paper analyses pupil dilations for 13 out of 27 participants completing a Fitts' law task using a virtual reality headset with built-in eye tracking. The largest pupil dilation occurred for the condition subjectively rated as requiring the most physical and mental effort. Fitts' index of difficulty had no significant effect on pupil dilation, suggesting differences in motor task complexity may not affect pupil dilation.

Fatigue detection, monitoring and management is important and needs to be accommodated in the busy lifestyles that many people have these days. It may have an impact on the physical as well as the emotional health of the individuals. Detection of fatigue is the first step towards its management. With eye-tracking software using cameras, and being included in the laptops and smartphones, it now has the potential to become quite ubiquitous.

This paper presents a Fitts' law experiment and a clinical case study performed with a head-mounted display (HMD). The experiment compared gaze, foot, and head pointing. With the equipment setup we used, gaze was slower than the other pointing methods, especially in the lower visual field. Throughputs for gaze and foot pointing were lower than mouse and head pointing and their effective target widths were also higher. A follow-up case study included seven participants with movement disorders. Only two of the participants were able to calibrate for gaze tracking but all seven could use head pointing, although with throughput less than one-third of the non-clinical participants.

Gaze may be a good alternative input modality for people with limited hand mobility. This accessible control based on eye tracking can be implemented into telepresence robots, which are widely used to promote remote social interaction and providing the feeling of presence. This extended abstract introduces a Ph.D. research project, which takes a two-phase approach towards investigating gaze-controlled telepresence robots. A system supporting gaze-controlled telepresence has been implemented. However, our current findings indicate that there were still serious challenges with regard to gaze-based driving. Potential improvements are discussed, and plans for future study are also presented.

Mobile robotic telepresence systems are increasingly used to promote social interaction between geographically dispersed people. People with severe motor disabilities may use eye-gaze to control a telepresence robots. However, use of gaze control for navigation of robots needs to be explored. This paper presents an experimental comparison between gaze-controlled and hand-controlled telepresence robots with a head-mounted display. Participants (n = 16) had similar experience of presence and self-assessment, but gaze control was 31% slower than hand control. Gaze-controlled robots had more collisions and higher deviations from optimal paths. Moreover, with gaze control, participants reported a higher workload, a reduced feeling of dominance, and their situation awareness was significantly degraded. The accuracy of their post-trial reproduction of the maze layout and the trial duration were also significantly lower.

Eye-gaze interaction is a common control mode for people with limited mobility of their hands. Mobile robotic telepresence systems are increasingly used to promote social interaction between geographically dispersed people. We are interested in how gaze interaction can be applied to such robotic systems, in order to provide new opportunities for people with physical challenges. However, few studies have implemented gaze-interaction into a telepresence robot and it is still unclear how gaze-interaction within these robotic systems impacts users and how to improve the systems. This paper introduces our research project, which takes a two-phase approach towards investigating a novel interaction-system we developed. Results of these two studies are discussed and future plans are described.

People who cannot use their hands may use eye-gaze to interact with robots. Emerging virtual reality head-mounted displays (HMD) have built-in eye-tracking sensors. Previous studies suggest that users need substantial practice for gaze steering of wheeled robots with an HMD. In this paper, we propose to apply a VR-based simulator for training of gaze-controlled robot steering. The simulator and preliminary test results are presented.

Gaze interaction with telerobots is a new opportunity for wheelchair users with severe motor disabilities. We present a video showing how head-mounted displays (HMD) with gaze tracking can be used to monitor a robot that carries a 360° video camera and a microphone. Our interface supports autonomous driving via way-points on a map, along with gaze-controlled steering and gaze typing. It is implemented with Unity, which communicates with the Robot Operating System (ROS).

We present a tool that allows reading teachers to record and replay students' voice and gaze behavior during reading. The tool replays scanpaths to reading professionals without prior gaze data experience. On the basis of test experiences with 147 students, we share our initial observations on how teachers make use of the tool to create a dialog with their students.

Gaze tracking in virtual reality (VR) allows for hands-free text entry, but it has not yet been explored. We investigate how the keyboard design, selection method, and motion in the field of view may impact typing performance and user experience. We present two studies of people (n = 32) typing with gaze+dwell and gaze+click inputs in VR. In study 1, the typing keyboard was flat and within-view; in study 2, it was larger-than-view but curved. Both studies included a stationary and a dynamic motion conditions in the user's field of view.

This paper addresses gaze interaction for smart home control, conducted from a wrist-worn unit. First we asked ten people to enact the gaze movements they would propose for e.g. opening a door or adjusting the room temperature. On basis of their suggestions we built and tested different versions of a prototype applying off-screen stroke input. Command prompts were given to twenty participants by text or arrow displays. The success rate achieved by the end of their first encounter with the system was 46% in average; it took them 1.28 seconds to connect with the system and 1.29 seconds to make a correct selection. Their subjective evaluations were positive with regard to the speed of the interaction. We conclude that gaze gesture input seems feasible for fast and brief remote control of smart home technology provided that robustness of tracking is improved.

This paper presents an experimental investigation of gaze-based control modes for unmanned aerial vehicles (UAVs or "drones"). Ten participants performed a simple flying task. We gathered empirical measures, including task completion time, and examined the user experience for difficulty, reliability, and fun. Four control modes were tested, with each mode applying a combination of x-y gaze movement and manual (keyboard) input to control speed (pitch), altitude, rotation (yaw), and drafting (roll). Participants had similar task completion times for all four control modes, but one combination was considered significantly more reliable than the others. We discuss design and performance issues for the gaze-plus-manual split of controls when drones are operated using gaze in conjunction with tablets, near-eye displays (glasses), or monitors.

This paper investigates whether it is feasible to interact with the small screen of a smartphone using eye movements only. Two of the most common gaze-based selection strategies, dwell time selections and gaze gestures are compared in a target selection experiment. Finger-strokes and accelerometer-based interaction, i. e. tilting, are also considered. In an experiment with 11 subjects we found gaze interaction to have a lower performance than touch interaction but comparable to the error rate and completion time of accelerometer (i.e. tilt) interaction. Gaze gestures had a lower error rate and were faster than dwell selections by gaze, especially for small targets, suggesting that this method may be the best option for hands-free gaze control of smartphones.

This paper presents an experiment comparing performance and user experience of gaze and mouse interaction in a minimalistic 3D flying game that only required steering. Mouse interaction provided better performance and participants considered it less physical and mental demanding, less frustrating and less difficult to maneuver. Gaze interaction, however, yielded higher levels of entertainment and engagement. The paper suggests that gaze steering provides a high kinesthetic pleasure both because it is difficult to master and because it presents a unique mapping between fixation and locomotion.

Gaze interaction affords hands-free control of computers. Pointing to and selecting small targets using gaze alone is difficult because of the limited accuracy of gaze pointing. This is the first experimental comparison of gaze-based interface tools for small-target (e.g. <12 × 12 pixels) point-and-select tasks. We conducted two experiments comparing the performance of dwell, magnification and zoom methods in point-and-select tasks with small targets in single- and multiple-target layouts. Both magnification and zoom showed higher hit rates than dwell. Hit rates were higher when using magnification than when using zoom, but total pointing times were shorter using zoom. Furthermore, participants perceived magnification as more fatiguing than zoom. The higher accuracy of magnification makes it preferable when interacting with small targets. Our findings may guide the development of interface tools to facilitate access to mainstream interfaces for people with motor disabilities and other users in need of hands-free interaction.

This paper presents a low-cost gaze tracking system that is based on a webcam mounted close to the user's eye. The performance of the gaze tracker was evaluated in an eye-typing task using two different typing applications. Participants could type between 3.56 and 6.78 words per minute, depending on the typing system used. A pilot study to assess the usability of the system was also carried out in the home of a user with severe motor impairments. The user successfully typed on a wall-projected interface using his eye movements.

This paper examines gaze gestures and their applicability as a generic selection method for gaze-only controlled interfaces. The method explored here is the Single Gaze Gesture (SGG), i.e. gestures consisting of a single point-to-point eye movement. Horizontal and vertical, long and short SGGs were evaluated on two eye tracking devices (Tobii/QuickGlance (QG)). The main findings show that there is a significant difference in selection times between long and short SGGs, between vertical and horizontal selections, as well as between the different tracking systems.

Accessing the smallest targets in mainstream interfaces using gaze alone is difficult, but interface tools that effectively increase the size of selectable objects can help. In this paper, we propose a conceptual framework to organize existing tools and guide the development of new tools. We designed a discrete zoom tool and conducted a proof-of-concept experiment to test the potential of the framework and the tool. Our tool was as fast as and more accurate than the currently available two-step magnification tool. Our framework shows potential to guide the design, development, and testing of zoom tools to facilitate the accessibility of mainstream interfaces for gaze users.

This paper presents StarGazer - a new 3D interface for gaze-based interaction and target selection using continuous pan and zoom. Through StarGazer we address the issues of interacting with graph structured data and applications (i.e. gaze typing systems) using low resolution eye trackers or small-size displays. We show that it is possible to make robust selection even with a large number of selectable items on the screen and noisy gaze trackers. A test with 48 subjects demonstrated that users who have never tried gaze interaction before could rapidly adapt to the navigation principles of StarGazer. We tested three different display sizes (down to PDA-sized displays) and found that large screens are faster to navigate than small displays and that the error rate is higher for the smallest display. Half of the subjects were exposed to severe noise deliberately added on the cursor positions. We found that this had a negative impact on efficiency. However, the user remained in control and the noise did not seem to effect the error rate. Additionally, three subjects tested the effects of temporally adding noise to simulate latency in the gaze tracker. Even with a significant latency (about 200 ms) the subjects were able to type at acceptable rates. In a second test, seven subjects were allowed to adjust the zooming speed themselves. They achieved typing rates of more than eight words per minute without using language modeling. We conclude that the StarGazer application is an intuitive 3D interface for gaze navigation, allowing more selectable objects to be displayed on the screen than the accuracy of the gaze trackers would otherwise permit.

The experiment described in this paper, shows a test environment constructed with two information spaces; one large with 2000 nodes ordered in semi-structured groups in which participants performed search and browse tasks; the other was smaller and designed for precision zooming, where subjects performed target selection simulation tasks. For both tasks, modes of gaze- and mouse-controlled navigation were compared.

The aim of this paper is to examine the learning processes that subjects undertake when they start using gaze as computer input. A 7-day experiment with eight Japanese students was carried out to record novice users' eye movement data during typing of 110 sentences. The experiment revealed that inefficient eye movements was dramatically reduced after only 15–25 sentences of typing, equal to approximately 3–4 h of practice. The performance data fits a general learning model based on the power law of practice. The learning model can be used to estimate further improvements in gaze typing performance. Our experimental results encourage the use of gaze-based interfaces for severely disabled people. This paper provides a taxonomy for gaze actions on dwell time-activated keys and it presents a method by which the learnability of gaze interfaces may be documented.

Low cost eye tracking has received an increased attention due to the rapid developments in tracking hardware (video boards, digital camera and CPU's) [Hansen and Pece 2005; OpenEyes 2005]. We present a gaze typing system based on components that can be bought in most consumer hardware stores around the world. These components are for example cameras and graphics cards that are made in large quantities. This kind of hardware differs from what is often claimed to be "off-the-shelf components", but which in fact is hardware only available from particular vendors.Institutions that supply citizens with communication aids may be reluctant to invest large amounts of money in new equipment that they are unfamiliar with. Recent investiagtions estimate that less than 2000 systems have actually been used by Europeans, even though more than half a million disabled people in Europe could potentially benefit from it. The main group of present users consists of people with motor neuron disease (MND) and amyotrophic lateral sclerosis (ALS). If the price of gaze communication systems can be lowered, it could become a preferred means of control for a large group of people [Jordansen et al. 2005]. Present commercial gaze trackers e.g. [Tobii 2005; LC-Technologies 2004] are easy to use, robust and sufficiently accurate for many screen-based applications but their costs exceed the budget of most people.We use a standard uncalibrated 400$ Sony consumer camera (Sony handycam DCR-HC14E) to obtain the image data. The camera is stationary and placed on a tripod close (variable) to the monitor, but the geometry of the user, monitor and camera varies among sequences. However, the users are sitting about 50 - 60 cm away from a 17" screen. A typical example of the setup is shown in figure 1. We use Sony standard video option for 'night vision' to create an glint with the build-in IR light emitter.Eye tracking based on common components is subject to several unknown factors as various system parameters (i.e. camera parameters and geometry) are unknown. Algorithms that employ robust statistical principles to accommodate uncertainties in image data as well as in gaze estimates in the typing process are therefore needed. We propose to use the RANSAC algorithm [Fischler and Bolles 1981] for both robust maximum likelihood estimation of iris observations [Hansen and Pece 2005] as well as for handling outliers in the calibration procedure [Morimoto et al. 2000].Our low-resolution gaze tracker can be calibrated in less than 3 minutes by looking at 9 predefined positions on the screen. The users sit on a standard office chair without headrests or other physical constraints. Under these conditions we have succeeded in tracking the gaze of people, obtaining accuracies about 160 pixels on screen. This is still less than accuracies claimed by the best current off-the-shelf eye trackers systems (i.e. 30-60 pixels). However comparing these eye trackers wouldn't be correct as they are based on different hardware and image data.Low-cost gaze trackers do not need to be as accurate and robust as the commercial systems, if they are used together with applications designed to tolerate noisy inputs.We use the GazeTalk [COGAIN 2005] typing communication system components and have through proper design of the typing interface, reduced the need for high accuracy. We have observed typing speeds in the range of 3 - 5 words per minute for untrained subjects using large on-screen buttons and a new noise tolerant dwell-time principle. We modify the traditional dwell-time activation to one that maintains a full distribution of all hypothetical button selections and then activate one button when the evidence become high enough.

The complex interplay between gaze tracker accuracy and interface design is the focus of this paper. Two slightly different variants of GazeTalk, a hierarchical typing interface, were contrasted with a novel interface, Dasher, in which text entry is done by continuous navigation. All of the interfaces were tested with a good and a deliberate bad calibration of the tracker. The purpose was to investigate, if performance indices normally used for evaluation of typing systems, such as characters per minute (CPM) and error-rate, could differentiate between the conditions, and thus guide an iterative system development of both trackers and interfaces. Gaze typing with one version of the static, hierarchical menu systems was slightly faster than the others. Error measures, in terms of rate of backspacing, were also significantly different for the systems, while the deliberate bad tracker calibrations did not have any measurable effect. Learning effects were evident under all conditions. Power-law-of-practice learning models suggested that Dasher might be more efficient than GazeTalk in the long run.

What if eye trackers could be downloaded and used immediately with standard cameras connected to a computer, without the need for an expert to setup the system? This has already the case for head trackers, so why not for eye trackers?Using components off-the-shelf (COTS) for camera-based eye tracking tasks has many advantages, but it certainly introduces several new problems as less assumptions on the system can be made. As a consequence of using COTS the price for eye tracking devices can be reduced while increasing the accessibility of these systems. Eye tracking based on COTS holds potential for a large number of possible applications such as in the games industry and eye typing [Majaranta and Räihä 2002]. Different cameras may be purchased depending on the need and the amount of money the user is willing to spend on the camera. In this framework it is not possible to use IR light sources and other novel engineered devices as they cannot be bought in a common hardware store. Very little control over the cameras and the geometry of the setup can be expected. The methods employed for eye tracking should therefore be able to handle changes in light conditions and image defocusing and scale changes [Hansen and Pece 2003]. On the same token pan-and-tilt cameras cannot be used, thus forcing such systems to be passive. Figure 1 shows a possible setup of a COTS-based eye tracker. When designing systems for the general public, it is unrealistic to assume that people are able to do camera calibration and make accurate setups of camera, monitor and user. Since little is known about the setup, would this then require a vast amount of calibration points needed for gaze estimation? That is, how many calibration points are really needed? Obviously the more calibration points are used the better the chances are to be able to infer the mapping from the image to gaze direction. It would even be possible to sample the entire function space provided sufficiently many calibration points are given. From the point of view of the users, a low number of calibration points is preferred as calibration may be considered as a tedious procedure. Systems that require many calibration points for every session are therefore not likely to succeed. It is also important to know the accuracy in gaze determination when using COTS to determine their applicability for various tasks.

This paper investigates the usability of gaze-typing systems for disabled people in a broad perspective that takes into account the usage scenarios and the particular users that these systems benefit. Design goals for a gaze-typing system are identified: productivity above 25 words per minute, robust tracking, high availability, and support of multimodal input. A detailed investigation of the efficiency and user satisfaction with a Danish and a Japanese gaze-typing system compares it to head- and mouse (hand) - typing. We found gaze typing to be more erroneous than the other two modalities. Gaze typing was just as fast as head typing, and both were slower than mouse (hand-) typing. Possibilities for design improvements are discussed.

We propose a non-intrusive eye tracking system intended for the use of everyday gaze typing using web cameras. We argue that high precision in gaze tracking is not needed for on-screen typing due to natural language redundancy. This facilitates the use of low-cost video components for advanced multi-modal interactions based on video tracking systems. Robust methods are needed to track the eyes using web cameras due to the poor image quality. A realtime tracking scheme using a mean-shift color tracker and an Active Appearance Model of the eye is proposed. It is possible from this model to infer the state of the eye such as eye corners and the pupil location under scale and rotational changes.

This is an overview of the recent progress leading towards a full subject-centered paradigm in human-computer interaction.At this new phase in the evolution of computer technologies it will be possible to take into account not just characteristics of average human beings, but create systems sensitive to the actual states of attention and intentions of interacting persons.We discuss some of these methods concentrating on the eye-tracking and brain imaging.The development is based on the use of eye movement data for a control of output devices, for gaze-contingent image processing and for disambiguating verbal as well as nonverbal information.