Gaze-based selection, via techniques such as gaze dwell, is one of the most common hands-free interaction performed by users in eXtended Reality (XR) environments. However, selecting a small constant dwell threshold to activate a target might lead to miss-interactions, also known as the Midas Touch problem, while a large threshold leads to eye fatigue. Prior research has proposed methodologies to adapt dwell thresholds based on the probability of the user activating a certain target considering past interactions or predicting intent based on gaze features. However, utilizing past inputs or gaze features leads to a heavily biased system towards individual strategy or physiology and cannot be generalized to other XR scenarios or users. In this work, we propose a novel context-aware system that leverages visual features of the task environment and user behavioral features such as the frequency of interactions, gaze speed variance, and head rotation velocity to adapt dwell thresholds across three distinct levels. We conducted a data collection experiment with twenty participants performing gaze dwell interactions in a general User Interface (UI) navigation task, and a visual search task. We trained a hierarchical machine learning model to predict and adapt dwell thresholds into three levels based on the induced cognitive load. We evaluated our system by utilizing standard machine learning metrics and by conducting a user study (n=17) based on quantitative and qualitative measures. Our system achieves a classification accuracy of \(70.72\%\) on the first level and \(85.43\%\) on the second. In addition, the system significantly reduces task completion time in less complex tasks and improves error rates in more cognitive intensive scenes.

As Extended Reality (XR) media experiences become a commodity, there is a unique opportunity to deploy XR for environmental awareness. Interaction challenges of Augmented Reality (AR) still exist, focused on limited gesture and head tracking affordances. AR technologies should also be seamlessly integrated with sensor data, analytics and ultimately status prediction, to be visualized in an AR experience, rather than merely superimposing visuals onto the real world. This paper presents an innovative, work-in-progress, multimodal AR experience integrating interactive narration, gestures, hands recognition and voice commands, while a citizen is wearing a head-worn AR display, promoting environmental awareness, health, and wellness. By combining AR technologies and a sensor network, GoNature AR provides citizen awareness of real-time, multimodal, air and noise pollution data.

Audio Augmented Reality (AAR) is a novel and unexplored area of AR representing the augmentation of reality with auditory as well as visual content. AAR’s interaction affordances as well as accurate real-world registration of visual as well as sound elements are challenging issues, especially in noisy, bright and busy environments outdoors. This paper presents a novel, mobile AAR experience that is deployed in a city environment while walking past six archaeological excavation sites in the city of Chania, Crete, Greece. The proposed AAR experience utilizes cutting-edge gamification techniques, non-linear storytelling, precise AR visualization and spatial audio, offering innovative AAR interaction while exploring the city’s archaeological sites and history, outdoors.

As digitization has transformed media and Augmented Reality (AR) is evolving from a research area to a commodity, museums are creating interactive AR experiences to digitally enhance their collection and increase audience engagement. Head-worn AR experiences, though, face interaction challenges as they are often employed in busy spaces and are in need of intuitive multimodal interfaces for users on the move. This paper presents an innovative, work-in-progress, multimodal AR experience integrating non-obtrusive dialogue, music, and sound as well as gesture and gaze-based interaction, while a user is wearing a head-worn AR display. Users are motivated to explore and interact with digital cultural artefacts superimposed onto the real-world museum setting and physical artefacts, while moving around in a museum setting. We initially analyze interactive AR experiences to identify specific user requirements related to head-worn AR experiences. We deploy these requirements for the design of interactive, multimodal AR in a museum setting.

This paper presents an innovative, near-contact, person-to-3D character, real-time dance training framework for leaders developed in Augmented Reality (AR) for interactive entertainment. Rather than mimicking pre-recorded dance animations as in previous work, the user is trained to lead a 3D partner while dancing, employing mid-air gestures. Real and virtual hands are joined and dance movements are initiated to which the 3D character responds to, through Unity's Inverse Kinematics solvers, in a non-predetermined manner following Latin dance rules. We evaluate task workload and usability of our AR dance training system compared with the Virtual Reality (VR) equivalent. Perceived training performance, accuracy of actions and fun were lower in AR, evoking increased training effort in AR compared to playfully training in VR. Design recommendations include increasing AR's Field of View, enhancing high quality graphics in AR, removing obtrusive cabling in VR and improving finger tracking in VR.

In this paper, we present a desktop-based CNC machining training system developed as a serious game for CNC manufacturing motivating young digital-ready users to train in the machining sector. The proposed serious game trains users to write G-code and set up virtual machines for completing Milling and Turning tasks, contrary to previous work mainly focusing on machining simulations. G-code is a simple and usable low-level command set for manufacturing complex objects without prior knowledge of Computer-Aided Manufacturing (CAM) systems. The proposed CNC machining serious game can be used as a standalone introductory training application before being involved with CNC manufacturing. It contains CNC machining and G-code tutorials explaining processes without the need for a book or the assistance from a trainer. The user can select to train in turning and milling machines. The system adapts to each user's progression and performance, assigning missions to a user when mistakes are continuous, while hastening the progress of an experienced user to more advanced missions. The user can check the correctness of the code they have written and receive feedback in relation to potential mistakes.

The ”ADDventurous Rhythmical Planet” is a multimodal 3D serious game that exploits the benefits of music and rhythm to help children diagnosed with Attention Deficit Hyperactivity Disorder (ADHD) to confront important psychosocial challenges. Music is shown to significantly help children maintain focused attention as well as offer a timed structure around their actions, involving melody and tempo. The advantages of music are exploited in the game, using a tin drum. This drum is used by the player to produce rhythm and it is connected to the computer via a Makey Makey board. This board sends each drum knock as a mouse click signal to the computer. Using the drum, the player tries to reproduce a given beat, and subsequently synchronize with another player, so that they cooperatively produce an expected beat. This creates a connection between the real and the virtual world of the game. In order for the hero of the game to progress in the game, the player uses the tin drum to reproduce a number of rhythms. After completing the single-user mode, the player moves on to the multi-user mode. In this mode, the two players have to hold hands and use the tin drum to collaboratively reproduce the requested beat. For the input management and comparison of rhythms, a special algorithm was implemented. The algorithm takes the beat created by the player and compares it to the expected beat. The player completes the game level if the rhythm played is similar enough to the rhythm heard. If not, the player tries again to reproduce the expected rhythm. The player receives a visual representation of the beat played, showing the correct knocks and the wrong ones. The pilot evaluation presented confirms the effectiveness of the game and demonstrates the benefits of rhythm in order to enhance social skills and collaboration between children.

At full resolution, path tracing cannot be deployed in real-time based on current graphics hardware due to slow convergence times and noisy outputs, despite recent advances in denoisers. In this work, we develop a perceptual sandbox emulating a foveated path tracer to determine the eccentricity angle thresholds that enable imperceptible foveated path tracing. In a foveated path tracer the number of rays fired can be decreased, and thus performance can be increased. For this study, due to current hardware limitations prohibiting real-time path-tracing for multiple samples-per-pixel, we pre-render image buffers and emulate foveated rendering as a post-process by selectively blending the pre-rendered content, driven by an eye tracker capturing eye motion. We then perform three experiments to estimate conservative thresholds of eccentricity boundaries for which image manipulations are imperceptible. Contrary to our expectation of a single threshold across the three experiments, our results indicated three different average thresholds, one for each experiment. We hypothesise that this is due to the dissimilarity of the methodologies, i.e., A-B testing vs sequential presentation vs custom adjustment of eccentricities affecting the perceptibility of peripheral blur among others. We estimate, for the first time for path tracing, specific thresholds of eccentricity that limit any perceptual repercussions whilst maintaining high performance. We perform an analysis to determine potential computational complexity reductions due to foveation in path tracing. Our analysis shows a significant boost in path-tracing performance (≥ 2x − 3x) using our foveated rendering method as a result of the reduction in the primary rays.

Input methods for interaction in smartphone-based virtual and mixed reality (VR/MR) are currently based on uncomfortable head tracking controlling a pointer on the screen. User fixations are a fast and natural input method for VR/MR interaction. Previously, eye tracking in mobile VR suffered from low accuracy, long processing time, and the need for hardware add-ons such as anti-reflective lens coating and infrared emitters. We present an innovative mobile VR eye tracking methodology utilizing only the eye images from the front-facing (selfie) camera through the headset’s lens, without any modifications. Our system first enhances the low-contrast, poorly lit eye images by applying a pipeline of customised low-level image enhancements suppressing obtrusive lens reflections. We then propose an iris region-of-interest detection algorithm that is run only once. This increases the iris tracking speed by reducing the iris search space in mobile devices. We iteratively fit a customised geometric model to the iris to refine its coordinates. We display a thin bezel of light at the top edge of the screen for constant illumination. A confidence metric calculates the probability of successful iris detection. Calibration and linear gaze mapping between the estimated iris centroid and physical pixels on the screen results in low latency, real-time iris tracking. A formal study confirmed that our system’s accuracy is similar to eye trackers in commercial VR headsets in the central part of the headset’s field-of-view. In a VR game, gaze-driven user completion time was as fast as with head-tracked interaction, without the need for consecutive head motions. In a VR panorama viewer, users could successfully switch between panoramas using gaze.

Abstract Virtual and augmented reality (VR/AR) are expected to revolutionise entertainment, healthcare, communication and the manufacturing industries among many others. Near‐eye displays are an enabling vessel for VR/AR applications, which have to tackle many challenges related to ergonomics, comfort, visual quality and natural interaction. These challenges are related to the core elements of these near‐eye display hardware and tracking technologies. In this state‐of‐the‐art report, we investigate the background theory of perception and vision as well as the latest advancements in display engineering and tracking technologies. We begin our discussion by describing the basics of light and image formation. Later, we recount principles of visual perception by relating to the human visual system. We provide two structured overviews on state‐of‐the‐art near‐eye display and tracking technologies involved in such near‐eye displays. We conclude by outlining unresolved research questions to inspire the next generation of researchers.

Comfortable, high-quality 3D stereo viewing is becoming a requirement for interactive applications today. Previous research shows that manipulating disparity can alleviate some of the discomfort caused by 3D stereo, but it is best to do this locally, around the object the user is gazing at. The main challenge is thus to develop a gaze predictor in the demanding context of real-time, heavily task-oriented applications such as games. Our key observation is that player actions are highly correlated with the present state of a game, encoded by game variables. Based on this, we train a classifier to learn these correlations using an eye-tracker which provides the ground-truth object being looked at. The classifier is used at runtime to predict object category - and thus gaze - during game play, based on the current state of game variables. We use this prediction to propose a dynamic disparity manipulation method, which provides rich and comfortable depth. We evaluate the quality of our gaze predictor numerically and experimentally, showing that it predicts gaze more accurately than previous approaches. A subjective rating study demonstrates that our localized disparity manipulation is preferred over previous methods.

Abstract Attention‐based Level‐Of–Detail (LOD) managers downgrade the quality of areas that are expected to go unnoticed by an observer to economize on computational resources. The perceptibility of lowered visual fidelity is determined by the accuracy of the attention model that assigns quality levels. Most previous attention based LOD managers do not take into account saliency provoked by context, failing to provide consistently accurate attention predictions. In this work, we extend a recent high level saliency model with four additional components yielding more accurate predictions: an object‐intrinsic factor accounting for canonical form of objects, an object‐context factor for contextual isolation of objects, a feature uniqueness term that accounts for the number of salient features in an image, and a temporal context that generates recurring fixations for objects inconsistent with the context. We conduct a perceptual experiment to acquire the weighting factors to initialize our model. We design C‐LOD, a LOD manager that maintains a constant frame rate on mobile devices by dynamically re‐adjusting material quality on secondary visual features of non‐attended objects. In a proof of concept study we establish that by incorporating C‐LOD, complex effects such as parallax occlusion mapping usually omitted in mobile devices can now be employed, without overloading GPU capability and, at the same time, conserving battery power.

Successfully predicting visual attention can significantly improve many aspects of computer graphics: scene design, interactivity and rendering. Most previous attention models are mainly based on low-level image features, and fail to take into account high-level factors such as scene context, topology, or task. Low-level saliency has previously been combined with task maps, but only for predetermined tasks. Thus, the application of these methods to graphics (e.g., for selective rendering) has not achieved its full potential. In this article, we present the first automated high-level saliency predictor incorporating two hypotheses from perception and cognitive science that can be adapted to different tasks. The first states that a scene is comprised of objects expected to be found in a specific context as well objects out of context which are salient (scene schemata) while the other claims that viewer’s attention is captured by isolated objects (singletons). We propose a new model of attention by extending Eckstein’s Differential Weighting Model. We conducted a formal eye-tracking experiment which confirmed that object saliency guides attention to specific objects in a game scene and determined appropriate parameters for a model. We present a GPU-based system architecture that estimates the probabilities of objects to be attended in real- time. We embedded this tool in a game level editor to automatically adjust game level difficulty based on object saliency, offering a novel way to facilitate game design. We perform a study confirming that game level completion time depends on object topology as predicted by our system.

The advent of affordable display technology and seamless integration of real-world scenes with computer graphics fuels our continuing ability to create and display stunning realistic imagery. With the arrival of new technology, algorithms and display methods comes the realization that gains can be made by tailoring output to the intended audience - humans. Human beings have amazingly complex perceptual systems, which have the ability to quickly capture and process vast amounts of complex data. With all its capability however, the Human Visual System (HVS) has some surprising nuances and limitations that can be exploited to the benefit of numerous graphics applications. This short course will provide insight into those aspects of HVS and other perceptual systems that can serve as both a guide and yard-stick to further the development and evaluation of computer graphics imagery and presentations. The literature on perception provides a rich source of knowledge that can be applied to the realm of computer graphics for immediate and direct benefit, generating images that not only exhibit higher quality, but use less time and resources to process. In addition, knowledge of the HVS serves as a guide on how best to present the images to fulfill the application at hand.

No abstract available.

An individual's prior experience will influence how new visual information in a scene is perceived and remembered. Accuracy of memory performance per se is an imperfect reflection of the cognitive activity (awareness states) that underlies performance in memory tasks. The aim of this research is to investigate the effect of varied visual fidelity of training environments on the transfer of training to the real world after exposure to immersive simulations representing a real-world scene. A between groups experiment was carried out to explore the effect of rendering quality on measurements of location-based recognition memory for objects and associated states of awareness. The immersive simulation consisted of one room that was either rendered flat-shaded or using radiosity rendering. The simulation was displayed on a stereo head-tracked head mounted display. Post exposure to the synthetic simulation, participants completed a memory recognition task conducted in a real-world scene by physically arranging objects in their physical form in a real-world room. Participants also reported one of four states of awareness following object recognition. They were given several options of awareness states that reflected the level of visual mental imagery involved during retrieval, the familiarity of the recollection and related guesses. The scene incorporated objects that “fitted” into the specific context of the real-world scene, referred to as consistent objects, and objects that were not related to the specific context of the real-world scene, referred to as inconsistent objects. A follow-up study was conducted a week after the initial test. Interestingly, results revealed a higher proportion of correct object recognition associated with mental imagery when participants were exposed to low-fidelity flat-shaded training scenes rather than the radiosity rendered ones. Memory psychology indicates that awareness states based on visual imagery require stronger attentional processing in the first instance than those based on familiarity. A tentative claim would, therefore, be that those immersive environments that are distinctive because of their variation from “real,” such as flat-shaded environments, recruit stronger attentional resources. This additional attentional processing may bring about a change in participants' subjective experiences of “remembering” when they later transfer the training from that environment into a real-world situation.

In a virtual environment (VE), efficient techniques are often needed to economize on rendering computation without compromising the information transmitted. The reported experiments devise a functional fidelity metric by exploiting research on memory schemata. According to the proposed measure, similar information would be transmitted across synthetic and real-world scenes depicting a specific schema. This would ultimately indicate which areas in a VE could be rendered in lower quality without affecting information uptake. We examine whether computationally more expensive scenes of greater visual fidelity affect memory performance after exposure to immersive VEs, or whether they are merely more aesthetically pleasing than their diminished visual quality counterparts. Results indicate that memory schemata function in VEs similar to real-world environments. “High-level” visual cognition related to late visual processing is unaffected by ubiquitous graphics manipulations such as polygon count and depth of shadow rendering; “normal” cognition operates as long as the scenes look acceptably realistic. However, when the overall realism of the scene is greatly reduced, such as in wireframe, then visual cognition becomes abnormal. Effects that distinguish schema-consistent from schema-inconsistent objects change because the whole scene now looks incongruent. We have shown that this effect is not due to a failure of basic recognition.

Computer graphics algorithms have for long dealt with simulation of physics: simulation of the geometry of a real-world space, simulation of the light propagation in a real environment and simulation of motor actions with appropriate tracking. Perception principles have subsequently been incorporated into rendering algorithms in order to save rendering computation and produce photo realistic images from a human rather than a machine point of view. Spatial memory tasks are often incorporated in benchmarking processes when assessing fidelity of a VE simulation, since spatial awareness is crucial for human performance efficiency of any task that entails awareness of space. This paper is exploring the exploitation of visual perception principles towards efficient selective rendering techniques. In order to economize on rendering computation, selective rendering guides a high level of detail to appropriate regions of a synthetic scene. Such decisions are based on predictive attention modeling, gaze or even cognitive information. Experimental studies presented will show that it is possible to produce scenes from a human rather than a physics point of view.

This paper expands on the presentation of a methodology that provides a technology-enhanced exhibition of a cultural artefact through the use of a safe hybrid 2D/3D multimodal interface. Such tangible interactions are based on the integration of a 3DOF orientation tracker and information sensors with a 'Kromstaf' rapid prototype replica to provide tactile feedback. The multimodal interface allows the user to manipulate the object via physical gestures which, during evaluation, establish a profound level of virtual object presence and user satisfaction. If a user cannot manipulate the virtual object effectively many application specific tasks cannot be performed. This paper assesses the usability of the multimodal interface by comparing it with two input devices--the Magellan SpaceMouse, and a 'black box', which contains the same electronics as the multimodal interface but without the tactile feedback offered by the 'Kromstaf' replica. A complete human-centred usability evaluation was conducted utilizing task based measures in the form of memory recall investigations after exposure to the interface in conjunction with perceived presence and user satisfaction assessments. Fifty-four participants across three conditions (Kromstaf, space mouse and black box) took part in the evaluation.

Accuracy of memory performance per se is an imperfect reflection of the cognitive activity (awareness states) that underlies performance in memory tasks. The aim of this research is to investigate the effect of varied visual and interaction fidelity of immersive virtual environments on memory awareness states. A between groups experiment was carried out to explore the effect of rendering quality on location-based recognition memory for objects and associated states of awareness. The experimental space, consisting of two interconnected rooms, was rendered either flat-shaded or using radiosity rendering. The computer graphics simulations were displayed on a stereo head-tracked Head Mounted Display. Participants completed a recognition memory task after exposure to the experimental space and reported one of four states of awareness following object recognition. These reflected the level of visual mental imagery involved during retrieval, the familiarity of the recollection, and also included guesses. Experimental results revealed variations in the distribution of participants' awareness states across conditions while memory performance failed to reveal any. Interestingly, results revealed a higher proportion of recollections associated with mental imagery in the flat-shaded condition. These findings comply with similar effects revealed in two earlier studies summarized here, which demonstrated that the less "naturalistic" interaction interface or interface of low interaction fidelity provoked a higher proportion of recognitions based on visual mental images.

This sketch presents a new method for assessing 'functional realism' across a range of applications. The basic premise is that an individual's prior experience will influence how he or she perceives, comprehends and remembers new information in a scene. 36 participants across three conditions of varied rendering quality of the same space were exposed to the computer graphics environment and completed an object-based memory recognition task. The preliminary experiments presented here could have significant implications while identifying areas of an interactive computer graphics scene that require higher quality of rendering as well as areas where lower fidelity could be adequate, based on this premise.

The research community is challenged to investigate the factors that make virtual reality technologies effective and productive. Realising the goals of virtual reality systems and harnessing them to successful applications can be accomplished only by employing robust metrics and formal experimentation. Subsequently, the quality of interfaces and platforms implemented could be assessed.