Accessibility barriers associated with Digital Audio Workstations (DAWs) for users who are Blind and Visually Impaired (BVI) are well established. However, there has been a lack of research that examines how musical hardware devices are used as assistive tools and their role in improving DAW workflows. Nineteen participants from the BVI community were interviewed to detail their workflows, highlighting methods, benefits, and challenges of integrating musical hardware devices within their DAW. Results found that sixteen participants are using at least one hardware device (e.g., Komplete Kontrol) in their workflows as an accessibility tool for DAW features (e.g., FX parameters). Participants’ choice of hardware devices was influenced by multiple factors, such as technical specification, accessibility features, usability, and cost. Finally, we highlight suggestions for future research and development areas with a particular emphasis on proposing ways of implementing vibrotactile feedback that will enhance DAWs’ accessibility.

Augmented and virtual reality experiences present significant barriers for disabled people, making it challenging to fully engage with immersive platforms. Whilst researchers have started to explore potential solutions addressing these accessibility issues, we currently lack a comprehensive understanding of research areas requiring further investigation to support the development of inclusive AR/VR systems. To address current gaps in knowledge, we led a series of multidisciplinary sandpits with relevant stakeholders (i.e., academic researchers, industry specialists, people with lived experience of disability, assistive technologists, and representatives from disability organisations, charities, and special needs educational institutions) to collaboratively explore research challenges, opportunities, and solutions. Based on insights shared by participants, we present a research agenda identifying key areas where further work is required in relation to specific forms of disability (i.e., across the spectrum of physical, visual, cognitive, and hearing impairments), including wider considerations associated with the development of more accessible immersive platforms.

In this special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the journal papers from the 23rd IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2024), which will be held as a hybrid conference between October 21 and 25, 2024 in the Greater Seattle Area, USA. ISMAR continues the over twenty-year long tradition of IWAR, ISMR, and ISAR, and is the premier conference for Mixed and Augmented Reality in the world.

This paper focuses on the design, development and preliminary evaluation of cloud-based Internet of Things (IoT) system that utilises wearable sensors to remotely monitor the physiological patterns of young person with Emotional Behavioral Disorder (EBD). We report on an exploratory study with caregivers to understand the challenges exhibited by young people with EBD. This informed the design of cloud-based IoT system that retrieves real-time physiological parameters such as Heart Rate, Body Temperature through non-invasive wearable sensors, which enables caregivers to monitor the activity of young person remotely and provide enhanced support based on the collected parameters. To validate system's feasibility, an initial pilot study within a healthcare environment involving children with EBD was conducted. The study confirms the system's viability and identifies important areas for future improvements. By utilizing IoT technology, our work aims to enhance the healthcare sector's ability to support individuals with EBD and improve their overall well-being.

Abstract A number of approaches to online time warping have been proposed based on plotting an alignment path backwards from a selected end‐point. When continually updating a time warp to align a time series with a live data source, such as a stream of motion capture data, the use of backwards plotting makes it hard to maintain a monotonic constraint. To solve this problem, a number of time warping approaches based on forward plotting, referred to as FP‐DTW, are proposed and evaluated by applying them to human motions. We demonstrate that forward plotting for temporal alignment is a viable solution when backwards plotting is otherwise not possible.

Abstract The ability to measure similarity and alignment of motions is a key tool in motion retrieval and motion editing. Similarity metrics based on distance functions are often utilized when measuring similarity of human motions, however, metrics based on correlation can also potentially useful for measuring similarity and alignment. This paper evaluates the use of correlation as a method of measuring the alignment and similarity of human motion and compares them against more established distance‐based metrics. Three correlation methods and five methods of parameterising rotation are evaluated. The results show that parameterization based on displacement vectors and Kendall Tau rank correlation are optimal for measuring the alignment between two motions. If measuring similarity of motions, however, an approach based on distance metrics for angular or positional distance should be used.

The recent proliferation of immersive technology has led to the rapid adoption of consumer-ready hardware for Augmented Reality (AR) and Virtual Reality (VR). While this increase has resulted in a variety of platforms that can offer a richer interactive experience, the advances in technology bring more variability in display types, interaction sensors and use cases. This provides a spectrum of device-specific interaction possibilities, with each offering a tailor-made solution for delivering immersive experiences to users, but often with an inherent lack of standardisation across devices and applications. To address this, a systematic review and an evaluation of explicit, task-based interaction methods in immersive environments are presented in this paper. A corpus of papers published between 2013 and 2020 is reviewed to thoroughly explore state-of-the-art user studies, which investigate input methods and their implementation for immersive interaction tasks (pointing, selection, translation, rotation, scale, viewport, menu-based and abstract). Focus is given to how input methods have been applied within the spectrum of immersive technology (AR, VR, XR). This is achieved by categorising findings based on display type, input method, study type, use case and task. Results illustrate key trends surrounding the benefits and limitations of each interaction technique and highlight the gaps in current research. The review provides a foundation for understanding the current and future directions for interaction studies in immersive environments, which, at this pivotal point in XR technology adoption, provides routes forward for achieving more valuable, intuitive and natural interactive experiences.

Voice input holds significant potential to support people with physical impairments in producing creative visual design outputs, although it is unclear whether well-established interaction methods used for manipulating graphical assets within mainstream creative applications (typically operated via a mouse, keyboard, or touch input) also present benefits for speech interaction. We present three new voice controlled approaches utilizing interface snapping techniques for manipulating a graphical object's dimensions: NoSnap, UserSnap, and AutoSnap. A user evaluation with people who have physical impairments (N=25) found that each method enabled participants to successfully control a graphical object's size across a series of design tasks, although the automated snapping approach utilized within AutoSnap was found to be more efficient, accurate, and usable. Subjective feedback from participants also highlighted a strong preference for AutoSnap over the other techniques in terms of efficiency and ease of use.

Navigation of source code typically requires extensive use of a traditional mouse and keyboard which can present significant barriers for developers with physical impairments. We present research exploring how commonly used code navigation approaches (e.g. locating references to user-defined identifiers, jumping to function definitions, conducting a search for specific syntax, etc.) can be optimized for multimodal voice interaction. An exploratory study was initially conducted with five developers who have physical impairments to elicit insights around their experiences in navigating code within existing voice-controlled development environments. Findings from this study informed the design of a code editor integrating different navigation features tailored for multimodal speech input. A user evaluation with 14 developers with physical impairments was conducted with results demonstrating that all participants were able to successfully complete a series of standard navigation tasks. Participants also highlighted that the code navigation techniques were intuitive to use and provided a high-level of usability.

People with physical impairments can experience challenges when using traditional input devices such as a mouse and keyboard for creative visual design work. Speech interaction is an alternative input method that can facilitate a more inclusive approach for supporting creative work. However, there has been a lack of work to date investigating how to position and move creative assets (e.g. images) around a digital workspace via speech interaction techniques. We present three multimodal speech interaction approaches to support the positioning of graphical objects around a design canvas via speech commands: Speed-based Control, Location Guides, and Positional Guides. A user evaluation with non-disabled participants (N=30) found that the Location Guides approach was significantly more efficient, accurate, and usable for the positioning of images when compared with the other methods. A follow-up study with physically impaired users (N=6) demonstrated they were able to effectively position images around a design canvas using the Location Guides technique with participants also rating this approach as exhibiting a high level of usability.

Natural and intuitive interaction such as freehand grasping of virtual objects is still a significant challenge in VR due to the dexterous versatility of the human grasping actions. Currently, the design considerations for creating freehand grasping interactions in VR are drawn from the body of historical knowledge presented for real object grasping. While this may be suitable for some applications, recent work has shown that users grasp virtual objects differently than they grasp real objects, presenting an absence of knowledge on how users intuitively grasp virtual objects. To begin to address this, we present an elicitation study where participants (N =39) grasped 16 virtual objects categorised by shape in a mixed docking task exploring placement and rotation. We report on a Wizard of OZ methodology, extract Grasp Type and Grasp Dimension and present grasping patterns in VR. Our results are of value to be taken forward into a framework of recommendations for grasping interactions, as well as parameterizing grasp types for developing intuitive grasp models and thus begin to bridge the gap in understanding natural grasping patters for VR object interactions.

Object categorisation methods have been historically used in literature for understanding and collecting real objects together into meaningful groups and can be used to define human interaction patterns (i. e grasping). When investigating grasping patterns for Virtual Reality (VR), researchers used Zingg’s methodology which categorises objects based on shape and form. However, this methodology is limited and does not take into consideration other object attributes that might influence grasping interaction in VR. To address this, our work presents a study into three categorisation methods for virtual objects. We employ Zingg’s object categorisation as a benchmark against existing real and virtual object interaction work and introduce two new categorisation methods that focus on virtual object equilibrium and virtual object component parts. We evaluate these categorisation methods using a dataset of 1872 grasps from a VR docking task on 16 virtual representations of real objects and report findings on grasp patterns. We report on findings for each virtual object categorisation method showing differences in terms of grasp classes, grasp type and aperture. We conclude by detailing recommendations and future ideas on how these categorisation methods can be taken forward to inform a richer understanding of grasping in VR.

People with physical impairments who are unable to use traditional input devices (i.e. mouse and keyboard) are often excluded from technical professions (e.g. web development). Alternative input methods such as eye gaze tracking and speech recognition have become more readily available in recent years with both being explored independently to support people with physical impairments in coding activities. This paper describes a novel multimodal application ("Voiceye") that combines voice input, gaze interaction, and mechanical switches as an alternative approach for writing code. The system was evaluated with non-disabled participants who have coding experience (N=29) to assess the feasibility of the application in writing HTML and CSS code. Results found that Voiceye was perceived positively and enabled successful completion of coding tasks. A follow-up study with disabled participants (N=5) demonstrated that this method of multimodal interaction can support people with physical impairments in writing and editing code.

Influence of interaction fidelity and rendering quality on perceived user experience have been largely explored in Virtual Reality (VR). However, differences in interaction choices triggered by these rendering cues have not yet been explored. We present a study analysing the effect of thermal visual cues and contextual information on 50 participants' approach to grasp and move a virtual mug. This study comprises 3 different temperature cues (baseline empty, hot and cold) and 4 contextual representations; all embedded in a VR scenario. We evaluate 2 different hand representations (abstract and human) to assess grasp metrics. Results show temperature cues influenced grasp location, with the mug handle being predominantly grasped with a smaller grasp aperture for the hot condition, while the body and top were preferred for baseline and cold conditions.

We present a new application ("Sakura") that enables people with physical impairments to produce creative visual design work using a multimodal gaze approach. The system integrates multiple features tailored for gaze interaction including the selection of design artefacts via a novel grid approach, control methods for manipulating canvas objects, creative typography, a new color selection approach, and a customizable guide technique facilitating the alignment of design elements. A user evaluation (N=24) found that non-disabled users were able to utilize the application to complete common design activities and that they rated the system positively in terms of usability. A follow-up study with physically impaired participants (N=6) demonstrated they were able to control the system when working towards a website design, rating the application as having a good level of usability. Our research highlights new directions in making creative activities more accessible for people with physical impairments.

Many tasks in computer vision are often calibrated and evaluated relative to human perception. In this paper, we propose to directly approximate the perceptual function performed by human observers completing a visual detection task. Specifically, we present a novel methodology for learning to detect image transformations visible to human observers through approximating perceptual thresholds. To do this, we carry out a subjective two-alternative forced-choice study to estimate perceptual thresholds of human observers detecting local exposure shifts in images. We then leverage transformation equivariant representation learning to overcome issues of limited perceptual data. This representation is then used to train a dense convolutional classifier capable of detecting local suprathreshold exposure shifts - a distortion common to image composites. In this context, our model can approximate perceptual thresholds with an average error of 0.1148 exposure stops between empirical and predicted thresholds. It can also be trained to detect a range of different local transformations.

This paper presents an evaluation of ultrasound mid-air haptics as a supplementary feedback cue for grasping and lifting virtual objects in Virtual Reality (VR). We present a user study with 27 participants and evaluate 6 different object sizes ranging from 40 mm to 100 mm. We compare three supplementary feedback cues in VR; mid-air haptics, visual feedback (glow effect) and no supplementary feedback. We report on precision metrics (time to completion, grasp aperture and grasp accuracy) and interaction metrics (post-test questionnaire, observations and feedback) to understand general trends and preferences. The results showed an overall preference for visual cues for bigger objects () while ultrasound mid-air haptics were preferred for small virtual targets ().

This paper presents an assessment of the variability in freehand grasping of virtual objects in an exocentric mixed reality environment. We report on an experiment covering 480 grasp based motions in the transition phase of interaction to determine the level of variation to the grasp aperture. Controlled laboratory conditions were used where 30 right-handed participants were instructed to grasp and move an object (cube or sphere) using a single medium wrap grasp from a starting location (A) to a target location (B) in a controlled manner. We present a comprehensive statistical analysis of the results showing the variation in grasp change during this phase of interaction. In conclusion we detail recommendations for freehand virtual object interaction design notably that considerations should be given to the change in grasp aperture over the transition phase of interaction.

This article presents an analysis into the accuracy and problems of freehand grasping in exocentric Mixed Reality (MR). We report on two experiments (1710 grasps) which quantify the influence different virtual object shape, size and position has on the most common physical grasp, a medium wrap. We propose two methods for grasp measurement, namely, the Grasp Aperture (GAp) and Grasp Displacement (GDisp). Controlled laboratory conditions are used where 30 right-handed participants attempt to recreate a medium wrap grasp. We present a comprehensive statistical analysis of the results giving pairwise comparisons of all conditions under test. The results illustrate that user Grasp Aperture varies less than expected in comparison to the variation of virtual object size, with common aperture sizes found. Regarding the position of the virtual object, depth estimation is often mismatched due to under judgement of the z position and x, y displacement has common patterns. Results from this work can be applied to aid in the development of freehand grasping and considered as the first study into accuracy of freehand grasping in MR, provide a starting point for future interaction design.

This paper presents a first study into the use of dual view visual feedback in an exocentric MR environment for assisting freehand grasping of virtual objects. Recent work has highlighted problems associated with user errors in freehand grasping, via an analysis of virtual object type, location and size. Our work presents an extension to this, where 30 participants are recruited for two experiments (one assessing object size and the second object position), giving 1710 grasps in total. We report on results following the same protocol of the aforementioned study using a dual view visual feedback method. Results show that dual view visual feedback significantly increases user z placement accuracy and improves grasp placement in the x and y axes, however completion time is significantly higher. No improvement was found in user grasp aperture using dual view visual feedback for changes in object size and position.

In this letter, we present a new 3D statistical method for surface detection which provides improvements over competitive methods both in terms of noise suppression and detection of complete surfaces. The methods are applied to both synthetically created image volumes, and MRI data. Accuracy against a ground truth is assessed using the quantitative figure of merit performance measure, with the statistical methods outperforming both a 3D implementation of the gradient Canny operator and a 3D optimal steerable filter method. The results also confirm how 3D surface detection methods avoid missing surface information by successfully locating complete boundaries irrespective of the object orientation and plane of image capture. We conclude that the statistical 3D methods are capable of producing more accurate surface maps in textured images, that reflect the 3D boundary information, improving on current 2D and 3D standards.

When human actors interact with virtual objects the result is often not convincing to a third party viewer, due to incongruities between the actor and object positions. In this study we aim to quantify the magnitude and impact of the errors that occur in a bimanual interaction, that is when an actor attempts to move a virtual object by holding it between both hands. A three stage framework is presented which firstly captures the magnitude of these interaction errors, then quantifies their effect on the relevant third party audience, and thirdly assesses methods to mitigate the impact of the errors. Findings from this work show that the degree of error was dependent on the size of the virtual object and also on the axis of the hand placement with respect to the axis of the interactive motion. In addition, actor hand placement outside and away from the object surface was found to affect the visual plausibility considerably more than when the actor's hands were within the object boundaries. Finally, a method for automatic adaptation of the object size to match the distance between the actor's hands gave a significant improvement in the viewers' assessment of the scene plausibility.

This paper presents a method for measuring the magnitude and impact of errors in mixed reality interactions. We define the errors as measurements of hand placement accuracy and consistency within bimanual movement of an interactive virtual object. First, a study is presented which illustrates the amount of variability between the hands and the mean distance of the hands from the surfaces of a common virtual object. The results allow a discussion of the most significant factors which should be considered in the frame of developing realistic mixed reality interaction systems. The degree of error was found to be independent of interaction speed, whilst the size of virtual object and the position of the hands are significant. Second, a further study illustrates how perceptible these errors are to a third person viewer of the interaction (e.g. an audience member). We found that interaction errors arising from the overestimation of an object surface affected the visual credibility for the viewer considerably more than an underestimation of the object. This work is presented within the application of a real-time Interactive Virtual Television Studio, which offers convincing realtime interaction for live TV production. We believe the results and methodology presented here could also be applied for designing, implementing and assessing interaction quality in many other Mixed Reality applications.

This demonstration is a live example of the experiment presented in [1], namely a method of assessing the visual credibility of a scene where a real person interacts with a virtual object in realtime. Inconsistencies created by actor's incorrect estimation of the virtual object are measured through a series of videos, each containing a defined visual error and rated against interaction credibility on a scale of 1–5 by conference delegates.

This work presents an objective performance analysis of statistical tests for edge detection which are suitable for textured or cluttered images. The tests are subdivided into two-sample parametric and non-parametric tests and are applied using a dual-region based edge detector which analyses local image texture difference. Through a series of experimental tests objective results are presented across a comprehensive dataset of images using a Pixel Correspondence Metric (PCM). The results show that statistical tests can in many cases, outperform the Canny edge detection method giving robust edge detection, accurate edge localisation and improved edge connectivity throughout. A visual comparison of the tests is also presented using representative images taken from typical textured histological data sets. The results conclude that the non-parametric Chi Square (χ2) and Kolmogorov Smirnov (KS) statistical tests are the most robust edge detection tests where image statistical properties cannot be assumed a priori or where intensity changes in the image are nonuniform and that the parametric Difference of Boxes (DoB) test and the Student’s t-test are the most suitable for intensity based edges. Conclusions and recommendations are finally presented contrasting the tests and giving guidelines for their practical use while finally confirming which situations improved edge detection can be expected.