The recent proliferation of immersive technology has led to the rapid adoption of consumer-ready hardware for Augmented Reality (AR) and Virtual Reality (VR). While this increase has resulted in a variety of platforms that can offer a richer interactive experience, the advances in technology bring more variability in display types, interaction sensors and use cases. This provides a spectrum of device-specific interaction possibilities, with each offering a tailor-made solution for delivering immersive experiences to users, but often with an inherent lack of standardisation across devices and applications. To address this, a systematic review and an evaluation of explicit, task-based interaction methods in immersive environments are presented in this paper. A corpus of papers published between 2013 and 2020 is reviewed to thoroughly explore state-of-the-art user studies, which investigate input methods and their implementation for immersive interaction tasks (pointing, selection, translation, rotation, scale, viewport, menu-based and abstract). Focus is given to how input methods have been applied within the spectrum of immersive technology (AR, VR, XR). This is achieved by categorising findings based on display type, input method, study type, use case and task. Results illustrate key trends surrounding the benefits and limitations of each interaction technique and highlight the gaps in current research. The review provides a foundation for understanding the current and future directions for interaction studies in immersive environments, which, at this pivotal point in XR technology adoption, provides routes forward for achieving more valuable, intuitive and natural interactive experiences.

Voice input holds significant potential to support people with physical impairments in producing creative visual design outputs, although it is unclear whether well-established interaction methods used for manipulating graphical assets within mainstream creative applications (typically operated via a mouse, keyboard, or touch input) also present benefits for speech interaction. We present three new voice controlled approaches utilizing interface snapping techniques for manipulating a graphical object's dimensions: NoSnap, UserSnap, and AutoSnap. A user evaluation with people who have physical impairments (N=25) found that each method enabled participants to successfully control a graphical object's size across a series of design tasks, although the automated snapping approach utilized within AutoSnap was found to be more efficient, accurate, and usable. Subjective feedback from participants also highlighted a strong preference for AutoSnap over the other techniques in terms of efficiency and ease of use.

Navigation of source code typically requires extensive use of a traditional mouse and keyboard which can present significant barriers for developers with physical impairments. We present research exploring how commonly used code navigation approaches (e.g. locating references to user-defined identifiers, jumping to function definitions, conducting a search for specific syntax, etc.) can be optimized for multimodal voice interaction. An exploratory study was initially conducted with five developers who have physical impairments to elicit insights around their experiences in navigating code within existing voice-controlled development environments. Findings from this study informed the design of a code editor integrating different navigation features tailored for multimodal speech input. A user evaluation with 14 developers with physical impairments was conducted with results demonstrating that all participants were able to successfully complete a series of standard navigation tasks. Participants also highlighted that the code navigation techniques were intuitive to use and provided a high-level of usability.

People with physical impairments can experience challenges when using traditional input devices such as a mouse and keyboard for creative visual design work. Speech interaction is an alternative input method that can facilitate a more inclusive approach for supporting creative work. However, there has been a lack of work to date investigating how to position and move creative assets (e.g. images) around a digital workspace via speech interaction techniques. We present three multimodal speech interaction approaches to support the positioning of graphical objects around a design canvas via speech commands: Speed-based Control, Location Guides, and Positional Guides. A user evaluation with non-disabled participants (N=30) found that the Location Guides approach was significantly more efficient, accurate, and usable for the positioning of images when compared with the other methods. A follow-up study with physically impaired users (N=6) demonstrated they were able to effectively position images around a design canvas using the Location Guides technique with participants also rating this approach as exhibiting a high level of usability.

Natural and intuitive interaction such as freehand grasping of virtual objects is still a significant challenge in VR due to the dexterous versatility of the human grasping actions. Currently, the design considerations for creating freehand grasping interactions in VR are drawn from the body of historical knowledge presented for real object grasping. While this may be suitable for some applications, recent work has shown that users grasp virtual objects differently than they grasp real objects, presenting an absence of knowledge on how users intuitively grasp virtual objects. To begin to address this, we present an elicitation study where participants (N =39) grasped 16 virtual objects categorised by shape in a mixed docking task exploring placement and rotation. We report on a Wizard of OZ methodology, extract Grasp Type and Grasp Dimension and present grasping patterns in VR. Our results are of value to be taken forward into a framework of recommendations for grasping interactions, as well as parameterizing grasp types for developing intuitive grasp models and thus begin to bridge the gap in understanding natural grasping patters for VR object interactions.

Object categorisation methods have been historically used in literature for understanding and collecting real objects together into meaningful groups and can be used to define human interaction patterns (i. e grasping). When investigating grasping patterns for Virtual Reality (VR), researchers used Zingg’s methodology which categorises objects based on shape and form. However, this methodology is limited and does not take into consideration other object attributes that might influence grasping interaction in VR. To address this, our work presents a study into three categorisation methods for virtual objects. We employ Zingg’s object categorisation as a benchmark against existing real and virtual object interaction work and introduce two new categorisation methods that focus on virtual object equilibrium and virtual object component parts. We evaluate these categorisation methods using a dataset of 1872 grasps from a VR docking task on 16 virtual representations of real objects and report findings on grasp patterns. We report on findings for each virtual object categorisation method showing differences in terms of grasp classes, grasp type and aperture. We conclude by detailing recommendations and future ideas on how these categorisation methods can be taken forward to inform a richer understanding of grasping in VR.

People with physical impairments who are unable to use traditional input devices (i.e. mouse and keyboard) are often excluded from technical professions (e.g. web development). Alternative input methods such as eye gaze tracking and speech recognition have become more readily available in recent years with both being explored independently to support people with physical impairments in coding activities. This paper describes a novel multimodal application ("Voiceye") that combines voice input, gaze interaction, and mechanical switches as an alternative approach for writing code. The system was evaluated with non-disabled participants who have coding experience (N=29) to assess the feasibility of the application in writing HTML and CSS code. Results found that Voiceye was perceived positively and enabled successful completion of coding tasks. A follow-up study with disabled participants (N=5) demonstrated that this method of multimodal interaction can support people with physical impairments in writing and editing code.

Influence of interaction fidelity and rendering quality on perceived user experience have been largely explored in Virtual Reality (VR). However, differences in interaction choices triggered by these rendering cues have not yet been explored. We present a study analysing the effect of thermal visual cues and contextual information on 50 participants' approach to grasp and move a virtual mug. This study comprises 3 different temperature cues (baseline empty, hot and cold) and 4 contextual representations; all embedded in a VR scenario. We evaluate 2 different hand representations (abstract and human) to assess grasp metrics. Results show temperature cues influenced grasp location, with the mug handle being predominantly grasped with a smaller grasp aperture for the hot condition, while the body and top were preferred for baseline and cold conditions.

We present a new application ("Sakura") that enables people with physical impairments to produce creative visual design work using a multimodal gaze approach. The system integrates multiple features tailored for gaze interaction including the selection of design artefacts via a novel grid approach, control methods for manipulating canvas objects, creative typography, a new color selection approach, and a customizable guide technique facilitating the alignment of design elements. A user evaluation (N=24) found that non-disabled users were able to utilize the application to complete common design activities and that they rated the system positively in terms of usability. A follow-up study with physically impaired participants (N=6) demonstrated they were able to control the system when working towards a website design, rating the application as having a good level of usability. Our research highlights new directions in making creative activities more accessible for people with physical impairments.

This paper presents an evaluation of ultrasound mid-air haptics as a supplementary feedback cue for grasping and lifting virtual objects in Virtual Reality (VR). We present a user study with 27 participants and evaluate 6 different object sizes ranging from 40 mm to 100 mm. We compare three supplementary feedback cues in VR; mid-air haptics, visual feedback (glow effect) and no supplementary feedback. We report on precision metrics (time to completion, grasp aperture and grasp accuracy) and interaction metrics (post-test questionnaire, observations and feedback) to understand general trends and preferences. The results showed an overall preference for visual cues for bigger objects () while ultrasound mid-air haptics were preferred for small virtual targets ().

This paper presents an assessment of the variability in freehand grasping of virtual objects in an exocentric mixed reality environment. We report on an experiment covering 480 grasp based motions in the transition phase of interaction to determine the level of variation to the grasp aperture. Controlled laboratory conditions were used where 30 right-handed participants were instructed to grasp and move an object (cube or sphere) using a single medium wrap grasp from a starting location (A) to a target location (B) in a controlled manner. We present a comprehensive statistical analysis of the results showing the variation in grasp change during this phase of interaction. In conclusion we detail recommendations for freehand virtual object interaction design notably that considerations should be given to the change in grasp aperture over the transition phase of interaction.

The video game market has become an established and ever-growing global industry. The health of the video and computer games industry, together with the variety of genres and technologies available, means that video game concepts and programmes are being applied in numerous different disciplines. One of these is the field known as serious games. The main goal of this paper is to collect all the relevant articles published during the last decade and create a trend analysis about the use of certain artificial intelligence algorithms related to decision making and learning in the field of serious games. A categorization framework was designed and outlined to classify the 129 papers that met the inclusion criteria. The authors made use of this categorization framework for drawing some conclusions regarding the actual use of intelligent serious games. The authors consider that over recent years enough knowledge has been gathered to create new intelligent serious games to consider not only the final aim but also the technologies and techniques used to provide players with a nearly real experience. However, researchers may need to improve their testing methodology for developed serious games, so as to ensure they meet their final purposes.

This article presents an analysis into the accuracy and problems of freehand grasping in exocentric Mixed Reality (MR). We report on two experiments (1710 grasps) which quantify the influence different virtual object shape, size and position has on the most common physical grasp, a medium wrap. We propose two methods for grasp measurement, namely, the Grasp Aperture (GAp) and Grasp Displacement (GDisp). Controlled laboratory conditions are used where 30 right-handed participants attempt to recreate a medium wrap grasp. We present a comprehensive statistical analysis of the results giving pairwise comparisons of all conditions under test. The results illustrate that user Grasp Aperture varies less than expected in comparison to the variation of virtual object size, with common aperture sizes found. Regarding the position of the virtual object, depth estimation is often mismatched due to under judgement of the z position and x, y displacement has common patterns. Results from this work can be applied to aid in the development of freehand grasping and considered as the first study into accuracy of freehand grasping in MR, provide a starting point for future interaction design.

This paper presents a first study into the use of dual view visual feedback in an exocentric MR environment for assisting freehand grasping of virtual objects. Recent work has highlighted problems associated with user errors in freehand grasping, via an analysis of virtual object type, location and size. Our work presents an extension to this, where 30 participants are recruited for two experiments (one assessing object size and the second object position), giving 1710 grasps in total. We report on results following the same protocol of the aforementioned study using a dual view visual feedback method. Results show that dual view visual feedback significantly increases user z placement accuracy and improves grasp placement in the x and y axes, however completion time is significantly higher. No improvement was found in user grasp aperture using dual view visual feedback for changes in object size and position.