Immersive technologies offer advantages for the visualization of and interaction with complex setups within manufacturing maintenance processes. The present work catalogs different applications of AR/VR in manufacturing maintenance practices as an extended version of a workshop paper presented at the International Workshop on eXtended Reality for Industrial and Occupational Supports (XRIOS). Through a scoping review in three computing and engineering digital libraries, we outline the key attributes of immersive solutions ( N p = 115 ) for industrial maintenance, categorizing functional prototypes with ten parameters related to interaction, visualization, and research methods. Moreover, we conducted a workshop with three manufacturing experts discussing the future of maintenance interfaces. By bringing forth their recommendations and insights, we targeted a key training challenge in maintenance. We designed and implemented a situated visualization prototype for a safety-critical procedure with real-time, in-depth, spatially relevant instructions. We compared the effects of 2D labels and 3D ghosts in a VR-simulated AR environment. In a preliminary between-subjects evaluation study ( N u = 24 ), we measured usability, workload, simulator sickness, completion time, and delayed recall. Although we did not find statistically significant differences between conditions, 3D ghosts showed slightly lower perceived workload and discomfort levels, along with shorter completion times. On the other hand, 2D labels produced higher usability. Overall, we contribute by a mapping out the state of the art and discovering knowledge gaps within immersive maintenance, presenting a design and preliminary user study that adheres to our recommendations.

Eye-tracking offers new ways to augment our interaction possibilities in extended reality. This paper investigates how gaze can assist pen users in translating shape points within graphical models. By leveraging gaze, we can support the usual design activities with an option where objects can be selected and repositioned through eye movements, with the pen serving as a confirmation tool. This can reduce manual effort and enhance efficiency and ergonomics. To evaluate its effectiveness, we compare four interaction techniques: two pen-based baselines (direct and ray-based) and two gaze-supported methods (gaze for selection and/or object dragging), using a probability based selection scheme. In a user study, 16 participants carried out a shape point translation task and their performance, effort, and user experience were measured. The results highlight the performance trade-offs of each technique—while the gaze-based dragging method introduced marginally more errors, it significantly reduced task time. Our findings offer comparative insights into the strength and limitations of gaze-and pen-based interaction methods, supporting the design of future multimodal 3D design tools.

A common challenge for people on the autism spectrum with intellectual disability, is indoor navigation and related daily activities, as found in previous research. In this paper we report on co-design of assistive augmented reality applications, where the goal is to help people on the autism spectrum to gain more independence in their daily lives. This study is based on initially two full-day workshops with staff only, followed by ten individual workshops with the end-users and their staff at day centers, using a mix of methods and prototypes. The results show a clear potential of augmented reality as assistive technology for indoor navigation, depending on individual capability and/or complexity of environments, as well as for other activities. We also found that new barriers may arise, which are discussed for future research.

Abstract Online learning has become an everyday form of learning for many students across different disciplines, including STEM subjects in the setting of higher education. Studying in these settings requires students to self-regulate their learning to a higher degree as compared to campus-based education. A vital aspect of self-regulated learning is the application of goal-setting strategies. Universities act to support students’ goal-setting through the achievement of course learning outcomes, which work both as a promise and metric of academic achievement. However, a lack of clear integration between course activities and course learning outcomes leaves a dissonance between students’ study efforts and the course progress. This demo study presents a student-centered learning analytics dashboard aimed at assisting students in their achievement of course learning goals in the setting of STEM higher education. The dashboard was designed using a design science methodological approach. Thirty-seven students have contributed to its development and evaluation during different stages of the design process, including the conceptual iterative design and prototyping. The preliminary results show that students found the tool to be easy to use and useful for the achievement of the course goals.

Abstract The adoption of Extended Reality (XR) technologies for supporting learning processes is an increasingly popular research topic for a wide variety of domains, including medical education. Currently, within this community, the metrics applied to quantify the potential impact these technologies have on procedural knowledge acquisition are inconsistent. This paper proposes a practical definition of standard metrics for the learning goals in the application of XR to surgical training. Their value in the context of previous research in neurosurgical training is also discussed. Objective metrics of performance include: spatial accuracy and precision, time-to-task completion, number of attempts. The objective definition of what the learner’s aims are enables the creation of comparable XR systems that track progress during training. The first impact is to provide a community-wide metric of progress that allows for consistent measurements. Furthermore, a measurable target opens the possibility for automated performance assessments with constructive feedback.

Aligning virtual and real objects in Augmented Reality (AR) is essential for the user experience. Without alignment, the user loses suspension of disbelief and the sense of depth, distance, and size. Occlusion is a key feature to be aligned. Virtual content should be partially or fully occluded if real world objects are in its line-of-sight. The challenge for simulating occlusion is to construct the geometric model of the environment. Earlier studies have aimed to create realistic occlusions, yet most have either required depth-sensing hardware or a static predefined environment. This paper proposes and evaluates an alternative model-based method for dynamic outdoor AR of virtual buildings rendered on non depth-sensing smartphones. It uses geospatial data to construct the geometric model of real buildings surrounding the virtual building. The method removes the target regions from the virtual building using masks constructed from real buildings. While the method is not pixel-perfect, meaning that the simulated occlusion is not fully realistic, results from the user study indicate that it fulfilled its goal. A majority of the participants expressed that their experience and depth perception improved with the method activated. The result from this study has applications to mobile AR since the majority of smartphones are not equipped with depth sensors. Using geospatial data for simulating occlusions is a sufficiently effective solution until depth-sensing AR devices are more widely available.

Most Human-Computer Interaction (HCI) researchers are accustomed to the process of formal ethics review for their evaluation or field trial protocol. Although this process varies by country, the underlying principles are universal. While this process is often a formality, for field research or lab-based studies with vulnerable users, formal ethics requirements can be challenging to navigate -- a common occurrence in the social sciences; yet, in many cases, foreign to HCI researchers. Nevertheless, with the increase in new areas of research such as mobile technologies for marginalized populations or assistive technologies, this is a current reality. In this paper we present our experiences and challenges in conducting several studies that evaluate interactive systems in difficult settings, from the perspective of the ethics process. Based on these, we draft recommendations for mitigating the effect of such challenges to the ethical conduct of research. We then issue a call for interaction researchers, together with policy makers, to refine existing ethics guidelines and protocols in order to more accurately capture the particularities of such field-based evaluations, qualitative studies, challenging lab-based evaluations, and ethnographic observations.

We present a case study analyzing and discussing the effects of introducing the requirement of public outreach of original student work into the project-based learning of Advanced Graphics and Interaction (AGI) at KTH Royal Institute of Technology. We propose Expo-Based Learning as Project-Based Learning augmented with the constructively aligned goal of achieving public outreach beyond the course. We promote this outreach through three challenges: 1) large public presentations; 2) multidisciplinary collaboration; and 3) professional portfolio building. We demonstrate that the introduction of these challenges, especially the public presentations, had lasting positive impact in the intended technical learning outcomes of AGI with the added benefit of learning team work, presentation skills, timeliness, accountability, self-motivation, technical expertise, and professionalism.

We introduce a new problem domain for activity recognition: the analysis of children's social and communicative behaviors based on video and audio data. We specifically target interactions between children aged 1-2 years and an adult. Such interactions arise naturally in the diagnosis and treatment of developmental disorders such as autism. We introduce a new publicly-available dataset containing over 160 sessions of a 3-5 minute child-adult interaction. In each session, the adult examiner followed a semi-structured play interaction protocol which was designed to elicit a broad range of social behaviors. We identify the key technical challenges in analyzing these behaviors, and describe methods for decoding the interactions. We present experimental results that demonstrate the potential of the dataset to drive interesting research questions, and show preliminary results for multi-modal activity recognition.

We present the evaluation of BrailleTouch, an accessible keyboard for blind users on touchscreen smartphones. Based on the standard Perkins Brailler, BrailleTouch implements a six-key chorded braille soft keyboard. Eleven blind participants typed for 165 twenty-minute sessions on three mobile devices: 1) BrailleTouch on a smartphone; 2) a soft braille keyboard on a touchscreen tablet; and 3) a commercial braille keyboard with physical keys. Expert blind users averaged 23.2 words per minute (wpm) on the BrailleTouch smartphone. The fastest participant, a touchscreen novice, achieved 32.1 wpm during his first session. Overall, participants were able to transfer their existing braille typing skills to a touchscreen device within an hour of practice. We report the speed for braille text entry on three mobile devices, an in depth error analysis, and the lessons learned for the design and evaluation of accessible and eyes-free soft keyboards.

Previously, we presented Viz-A-Vis, a VIsualiZation of Activity through computer VISion [17]. Viz-A-Vis visualizes behavior as aggregate motion over observation space. In this paper, we present two complementary user studies of Viz-A-Vis measuring its performance and discovery affordances. First, we present a controlled user study aimed at comparatively measuring behavioral analysis preference and performance for observation and search tasks. Second, we describe a study with architects measuring discovery affordances and potential impacts on their work practices. We conclude: 1) Viz-A-Vis significantly reduced search time; and 2) it increased the number and quality of insightful discoveries.

Texting is the essence of mobile communication and connectivity, as evidenced by today's teenagers, tomorrow's workforce. Fifty-four percent of American teens contact each other daily by texting, as compared to face-to-face (33%) and talking on the phone (30%) according to the Pew Research Center's Internet & American Life Project, 2010. Arguably, today's technologies support mobile text input poorly, primarily due to the size constraints of mobile devices. This is the case for everyone, but it is particularly relevant to the visually impaired. According to the World Health Organization, 284 million people are visually impaired worldwide. In order to connect these users to the global mobile community, we need to design effective and efficient methods for eyes-free text input on mobile devices. Furthermore, everyone would benefit from effective mobile texting for safety and speed. This design brief presents BrailleTouch, our working prototype solution for eyes-free mobile text input.

We present a longitudinal investigation of Tableau Machine, an intelligent entity that interprets and reflects the lives of occupants in the home. We created Tableau Machine (TM) to explore the parts of home life that are unrelated to accomplishing tasks. Task support for "smart homes" has inspired many researchers in the community. We consider design for experience, an orthogonal dimension to task-centric home life. TM produces abstract visualizations on a large LCD every few minutes, driven by a set of four overhead cameras that capture a sense of the social life of a domestic space. The openness and ambiguity of TM allow for a cycle of co-interpretation with householders. We report on three longitudinal deployments of TM for a period of six weeks. Participant families engaged with TM at the outset to understand how their behaviors were influencing the machine, and, while TM remained puzzling, householders interacted richly with TM and its images. We extract some key design implications for an experience-focused smart home.

In the established procedural model of information visualization, the first operation is to transform raw data into data tables [1]. The transforms typically include abstractions that aggregate and segment relevant data and are usually defined by a human, user or programmer. The theme of this paper is that for video, data transforms should be supported by low level computer vision. High level reasoning still resides in the human analyst, while part of the low level perception is handled by the computer. To illustrate this approach, we present Viz-A-Vis, an overhead video capture and access system for activity analysis in natural settings over variable periods of time. Overhead video provides rich opportunities for long-term behavioral and occupancy analysis, but it poses considerable challenges. We present initial steps addressing two challenges. First, overhead video generates overwhelmingly large volumes of video impractical to analyze manually. Second, automatic video analysis remains an open problem for computer vision.