As consumer-focused Augmented Reality (AR) becomes more commonplace, it is important to investigate interaction techniques that support users in their environment. Using everyday objects as interaction devices is compelling due to the availability of everyday objects and the capacity to provide a variety of passive haptic sensations, however, they typically lack active feedback. To study the addition of active feedback with everyday objects, a within-subjects study was conducted with n = 28 participants, utilising a flowerpot with vibro-tactile feedback to perform dial interactions, with four conditions: haptic only, audio only, both, and no additional feedback. Task completion time, accuracy, immersion, and preference were measured. No significant differences between conditions were found for task completion time or accuracy. Immersion and user preference measures only showed weak differences between haptic and audio feedback, suggesting that there may be little advantage to implement active haptic feedback for AR experiences using everyday objects.

Regardless of the style, most western dance is taught in a similar manner. A student comes to a studio, the instructor demonstrates a series of movements that the student attempts to replicate - often in front of a mirror - and the instructor provides corrective feedback. Unfortunately, this approach means that beginners are only able to practice and receive corrective feedback when the instructor is available. This paper reports on a study that assessed whether a Mixed Reality (MR) mirror displaying a virtual instructor overlaid with visual feedback can be used to teach a beginner a simple dance routine, replacing the traditional instructor and mirror method, and making beginners dance training more accessible. Three visual feedback modes to indicate how to achieve a correct pose in a dance sequence were designed, based on findings from a literature review, input from expert interviews and an online survey. These feedback modes, titled Spheres, Rubber Bands, and Arrows, were implemented and used as randomized conditions in a user study where participants learned three simple dance sequences. The user study showed that participants performed best with the Arrows feedback mode, though the preference rank for this feedback mode was the lowest. In contrast, the participants’ most preferred feedback mode was Spheres, though participants performed poorest with this mode. These findings suggest that user preference and performance in MR mirror dance training systems need to be balanced to create a system that is effective and enjoyable.

Due to the level of immersion and differences in the user interface, there can be a very large discrepancy in the user experience between users in immersive systems and non-immersive systems when playing games together. To investigate the impact of the cross-reality experience, which refers to the asymmetric use of eXtended Reality, we aim to understand the different affordances and experiences in an asymmetric setup, where one participant uses a desktop setup with a mouse and keyboard, and one uses a virtual reality (VR) headset and controller in two different task modes, Competition or Collaboration. In our research, a pair of participants played a game in real-time, using either the VR setup or the desktop setup. In Competition mode, the two participants were asked to defeat each other. In Collaboration mode, the pair of participants played as a team and were asked to defeat a pair of AI enemies. Our results show the VR group reported a better gaming experience and perceptual responses compared to the desktop group regardless of game mode, but that the desktop group showed superior gaming performance compared to the VR group in Competition mode.

Underfoot haptics, a largely unexplored area, offers rich tactile information close to that of hand-based interactions. Haptic feedback gives a sense of physicality to virtual environments making for a more realistic and immersive experience. Augmented Virtuality offers the ability to render virtual materials on a physical object, or haptic proxy, without the user being aware of the object’s physical appearance while seeing their own body. In this research, we investigate how the visual appearance of physical objects can be altered virtually to impact the tactile perception of the object. An Augmented Virtuality system was developed to explore this, and two tactile perception experiments, consisting of 18 participants, were conducted. Specifically, we explore whether changing the visual appearance of materials affects a person’s underfoot tactile perception and which tactile perception is most affected by the change through a within-subjects experiment. The study shows that a change in visual appearance significantly impacted the tactile perception of roughness.

In this paper, we present an elicitation study which explores how people use common household objects as props to control virtual objects in augmented reality first-person perspective games. 24 participants were asked to select items from a range of common household objects to use as controllers for three different types of virtual object: a sword, shield, and crossbow. Participants completed short gameplay tasks using their selected items and rated the AR experience using the Augmented Reality Immersion (ARI) questionnaire. Results found no strong consensus linking any specific household object to any virtual object across our test group and, in addition, those who chose the most commonly selected object for each task did not have significantly higher scores on the ARI questionnaire compared to those who did not. A short post-experiment interview indicated a few key factors that were important to participants when selecting their household object, such as shape, size, grip feel and weight distribution. Based on our findings we recommend that developers provide the ability for users to choose which household objects to use as props based on the user's own preferences, and that they design intuitive ways for users to interact with virtual objects.

Location-based activities can challenge citizens to explore their neighbourhood in new playful ways. This paper presents a classification of such playful activities based on the literature and experience gained (1) in co-creation sessions and (2) gameplay for neighbourhood exploration with citizens in the Hague. The location-based game designed for this purpose encouraged neighbourhood exploration and social interaction that went beyond the everyday normal. Results showed that citizens are specifically interested in activities that allow them to discover new information about, and places in, their neighbourhood when these are related to their own life. The results of this paper provide new knowledge on game design to inform designers on which type of interactions and activities foster neighbourhood exploration and social interaction.

Abstract COVID-19 has urged researchers to explore new options for distributed participatory design, as physical meetings and workshops became unfeasible. This situation posed new challenges but also opportunities, in particular with respect to engagement and inclusion. This paper focuses on distributed PD with Irish teenagers to support place-making during this period: to build relationships with each other and the community. In a two-week online summer school, teenagers explored a concern or highlighted a unique aspect of their local community and designed digital artworks in response. Activities and materials were designed to support reflection, empowerment, inclusiveness, emergence, and playfulness for participatory place-making. Analysis of the summer school provides insights and guidance on the design of online PD for engaging experiences, especially in the context of place-making.

Social interaction is part of the fabric of society and is essential to challenge many types of social barriers. Location-based games (LBGs) provide a means to foster such interaction in local communities. The design of such games is currently based primarily on designer experience and on the literature on game design in general, and not on an understanding of user requirements. This article explores the preferences and desires of adolescents in neighborhoods of Rotterdam South to socially interact with others and engage with their own neighborhood via LBGs. Adolescents are informants in the exploration of gaming activities for social interaction, which, when subjected to expert review with the mechanics-dynamics-aesthetics framework, produce gameplay requirements for the desired purpose: social interaction in public space. Such requirements provide researchers and game designers insights on the game dynamics best suited to foster location-based social interaction.

Validation of serious games tends to focus on evaluating their design as a whole. While this helps to assess whether a particular combination of game mechanics is successful, it provides little insight into how individual mechanics contribute or detract from a serious game's purpose or a player's game experience. This study analyses the effect of game mechanics commonly used in casual games for engagement, measured as a combination of player behaviour and reported game experience. Secondly, it examines the role of a serious game's purpose on those same measures. An experimental study was conducted with 204 participants playing several versions of a serious game to explore these points. The results show that adding additional game mechanics to a core gameplay loop did not lead to participants playing more or longer, nor did it improve their game experience. Players who were aware of the game's purpose, however, perceived the game as more beneficial, scored their game experience higher, and progressed further. The results show that game mechanics on their own do not necessarily improve engagement, while the effect of perceived value deserves further study.

Location-based systems (LBS) can be of benefit to obtain information when and where it is needed. What is not captured with LBS are points of interest (dynamic hotspots) that are not fixed in one location, such as people. Furthermore, using 2D interface elements makes it difficult to communicate precise spatial information to users. The current study aims to assess the viability of dynamic hotspots and augmented content in a mobile hotspot policing application. Mock cases were constructed to mimic real field policing work. The system, comprised of a smartphone and smartwatch, gave static and dynamic hotspot information, which was enriched with the use of augmented elements viewed through the smartphone-camera. Interviews, observations and usability measures were used to gain insight for augmented reality (AR) design considerations in the domain of field-policing. The detection of dynamic hotspots was seen as desirable. While static and dynamic hotspot information was seen as valuable, AR information, as it was utilized in the current study, has too many drawbacks to be used in the field. Alternative modalities, insights into possible future iterations and technical limitations of AR for the emergency response domain are discussed.

Over the last years, significant and highly visible results have been achieved using Artifical Intelligence (AI). Based on those, AI is the reigning world champion in games such as checkers, chess, Jeopardy, Go or Quake 3. However, there are still several disciplines in which humans remain leading, mainly because AI lacks qualities like experience, creativity or leadership. Exactly those qualities make humans great in innovation and design. Instead of focusing on improving AI, I advocate to use AI to augment humans and human intelligence. In such a way, humancentred intelligent systems as well as intelligent user interfaces can be created that either amplify existing or create new human skills and capabilities. Such augmented humans and augmented intelligence will allow humans to exceed their capabilities, go beyond current human capabilities and offer new experiences. Athletes could, e.g., receive real-time feedback on how to optimally use their resources. Police agents could, e.g., based on their current location and environment receive notifications on urgent tasks to be performed. Over the last years, I have explored the design of augmented humans and intelligence in the domains of sports, health and safety & security. In my talk, I present my recent research and discuss resulting design recommendations for augmented humans and intelligence. I conclude with an agenda for future research on augmented humans and intelligence.

In clinical practice, objective and quantitative assessment of motor dysfunction is required for monitoring disease progression over time and evaluating response to therapeutic interventions. Thereby, clinicians typically want their patients to make movements to their full physical potential. Augmented reality (AR) games using 3D hand and body tracking that engage patients, could motivate them to perform repetitive tasks to the limit of their physical capabilities in a safe environment. This paper reports on different AR games developed for objective upper extremity motor dysfunction assessment of Parkinson's Disease (PD) patients and stroke patients. Quantitative and qualitative evaluations of various user studies involving 23 PD patients, 22 stroke patients and 39 healthy persons are discussed to make design recommendations for designing engaging AR games for objective assessment of upper extremity motor dysfunction.

In recent years, many promising developments have taken place around augmented, virtual and mixed-reality technology. One could wonder whether these technologies can be combined to yield a mix of video games and sports, involving physical activities previously deemed impossible: creating a superhuman sport. This work investigates how mixed-reality can be used to create a fun, intuitive and engaging superhuman sport. To this end, the game League of Lasers was developed, in which two teams compete in a mix between football and Pong, using the physical movement of the players as main means to interact within mixed-reality. An evaluation of League of Lasers with a user study with 32 participants showed that League of Lasers is perceived as a fun and immersive skill-based game.

Superhuman sports (SHS) is a field where technological augmentations of human abilities and environment are combined to play a new and exciting sport. SHS make use of artificial senses or virtual reality to create a new experience that involves physical fitness and skills. Star Tag aims to combine these aspects with an engaging audience experience. This augmented reality game uses the Microsoft HoloLens, making it possible to move through a mixed reality space effectively. Star Tag is a competitive multiplayer game where players need to conquer all planets from their opponent to win the game. Players need to move around in a physical space from virtual planet to virtual planet in order to navigate the game-space and reach the planets. The audience is involved with the game via their phones, through which they can support the players. Through playtesting and conducting a survey, the results show that Star Tag is a superhuman sport that motivates people to be physically active.

The project “Factory-in-a-day” aims at reducing the installation time of a new hybrid robot-human production line, from weeks or months that current industrial systems now take, down to one day. The ability to rapidly install (and reconfigure) production lines where robots work alongside humans will strongly reduce operating cost and open a range of new opportunities for industry. In this paper, we explore a method of collaborative fabrication planning with the help of Augmented Reality as part of the concept Augmented Fabrication. In order to plan a new production line, two co-located workers at the factory wear a Microsoft Hololens head-mounted display and thus share a common visual context on the planed position of the robots and the production machines. They are assisted by an external remote expert connected via the Internet who is virtually co-located. We developed three different visualizations of the state of the local collaboration and plan to compare them in a user study.

Advances in technology offer new opportunities for a better understanding of how different disorders affect motor function. In this paper, we explore the potential of an augmented reality (AR) game implemented using free hand and body tracking to develop a uniform, cost-effective and objective methods for evaluation of upper extremity motor dysfunction in different patient groups. We conducted a study with 20 patients (10 Parkinson's Disease patients and 10 stroke patients) who performed hand/arm movement tasks in four different conditions in AR and one condition in real world. Despite usability issues mainly due to non-robust hand tracking, the patients were moderately engaged while playing the AR game. Our findings show that moving virtual objects was less targeted, took more time and was associated with larger trunk displacement and a lower variability of elbow angle and upper arm angle than moving real objects. No significant correlations were observed between characteristics of movements in AR and movements in the real world. Still, our findings suggest that the AR game may be suitable for assessing the hand and arm function of mildly affected patients if usability can be further improved.

In the early stages of 3D design, sketches are used to quickly conceptualize ideas and gain insight into problems and possible solutions. Computer-aided design tools are widely used for 3D modeling and design, but their required precision and 2D mouse and screen-based interface inhibit the flow of ideas. A study was conducted to explore the efficiency of hand tracking and virtual reality (VR) for 3D object manipulations in conceptual design. Based on existing research on conceptual design and hand gestures, an intuitive hand-based interaction model is proposed. An experiment on basic 3D manipulation shows that participants using a simple VR and hand-tracking interface prototype have similar performance to those using a traditional mouse and screen interface. For the improvement of gestural conceptual design interfaces, the relevant issues are identified.

Recent research showed that augmented reality (AR) technology is a promising approach to support distributed teams in crime scene investigation. This paper reports on the development and evaluation of a handheld AR system to support team situational awareness and collaboration among co-located and remote forensic investigators. The AR system runs on a smartphone strapped to the wrists of the local investigators and on a laptop for the remote investigator. The AR system has been evaluated with three experienced forensic investigators and was carried out in two rounds: (1) with one local and one remote investigator and (2) with two local and one remote investigator. The evaluation focused on the usability of the AR system and its effect on situational awareness as well as collaboration quality. Compared to earlier experiments using head-mounted devices (HMDs), the findings suggest that the handheld AR system addresses limitations of current HMD based AR system, but the necessary division of attention between smartphone AR system and real environment impacts the situational awareness.

For the clinical community, there is great need for objective, quantitative and valid measures of the factors contributing to motor dysfunction. Currently, there are no standard protocols to assess motor dysfunction in various patient groups, where each medical discipline uses subjectively scored clinical tests, qualitative video analysis, or marker-based motion capturing. We investigate the potential of Augmented Reality (AR) combined with serious gaming and marker-less tracking of the hand to facilitate efficient, cost-effective and patient-friendly methods for evaluation of upper extremity motor dysfunction in different patient groups. First, the design process of the game and the system architecture of the AR framework are described. To provide unhindered assessment of motor dysfunction, patients should operate with the system in a natural way and be able to understand their actions in the virtual AR world. To test this in our system, we conducted a usability study with five healthy people (aged between 57-63) on three different modalities of visual feedback for natural hand interaction with AR objects. These modalities are: no augmented hand, partial augmented hand (tip of index finger and tip of thumb) and a full augmented hand model. The results of the study show that a virtual representation of the fingertips or hand improves the usability of natural hand interaction.

Augmented Reality (AR) is a technology that allows users to view and interact in real time with virtual images seamlessly superimposed over the real world. AR systems can be used to create unique collaborative experiences. For example, co-located users can see shared 3D virtual objects that they interact with, or a user can annotate the live video view of a remote worker, enabling them to collaborate at a distance. The overall goal is to augment the face-to-face collaborative experience, or to enable remote people to feel that they are virtually co-located. In this special issue on collaboration in augmented reality, we begin with the visions of science fiction authors of future technologies that might significantly improve collaboration, then introduce research articles which describe progress towards these visions, finally we outline a research agenda discussing the work still to be done.

For operational units in the security domain that work together in teams, it is important to quickly and adequately exchange context-related information to ensure well-working collaboration. Currently, most information exchange is based on oral communication. This paper reports on different scenarios from the security domain in which augmented reality (AR) techniques are used to support such information exchange. The scenarios have been designed with a User Centred Design approach, in order to make the scenarios as realistic as possible. To support these scenarios, an AR system has been developed and evaluated in two rounds. In the first round, the usability and feasibility of the AR support has been evaluated with experts from different operational units in the security domain. The second evaluation round then focussed on the effect of AR on collaboration and situational awareness within the expert teams. With regard to the usability and feasibility of AR, the evaluation shows that the scenarios are well defined and the AR system can successfully support information exchange in teams operating in the security domain. The second evaluation round showed that AR can especially improve the situational awareness of remote colleagues not physically present at a scene.

One of the key challenges of augmented reality (AR) interfaces is to design effective hand-based interaction supported by computer vision. Hand-based interaction requires free-hands tracking to support user interaction in AR for which this article presents a novel approach. This approach makes it possible to compare different types of hand-based interaction in AR for navigating using a spatial user interface. Quantitative and qualitative analyses of a study with 25 subjects indicate that tangible interaction is the preferred type of interaction with which to determine the position of the user interface in AR and to physically point to a preferred option for navigation in augmented reality.

For operational units in the security domain that work together in teams it is important to quickly and adequately exchange context-related information. This extended abstract investigates the potential of augmented reality (AR) techniques to facilitate information exchange and situational awareness of teams from the security domain. First, different scenarios from the security domain that have been elicited using an end-user oriented design approach are described. Second, a usability study is briefly presented based on an experiment with experts from operational security units. The results of the study show that the scenarios are well-defined and the AR environment can successfully support information exchange in teams operating in the security domain.

In this half-day workshop we will explore how Augmented Reality (AR) and Mediated Reality (MR) can be used to develop radically new types of collaborative experiences that overcome some of the limitations of current conferencing systems. In combination, AR and MR technologies could be used to merge the shared perceived realities of different users as well as enriching their own individual experience in a collaborative task. The goal of the workshop is to bring together researchers who are interested developing collaborative systems using AR and MR technologies. They will build a picture of current and prior research on collaboration in AR and MR as well as set up a common research agenda for work going forward. Topics of the workshop will address open research issues and include but are not restricted to the following: •Case studies on using MR/AR for collaboration •Tools for building collaborative MR/AR systems •Effects of MR/AR on trust, presence, and coordination •Interaction models for collaboration in MR/AR •Tools for collaboration in MR/AR •Collaboration awareness in MR/AR.

Workspace awareness support is mandatory for group support systems. It allows users not only to follow actions of others, but to understand and respond to any changes others make to the workspace. In this paper, we present a novel approach for asynchronous awareness support by offering different filters to retrieve relevant awareness information and visualizing the evolution of the shared artifact. We illustrate our approach with a tool for collaborative non-linear storytelling in which users can jointly create a story graph of interconnected audio files. Such a story graph is an example for a non-linear story. We describe the development of a prototype that visualizes how the collaborative story has evolved over time. We evaluate our approach for asynchronous awareness support in an experiment with 40 participants exploring story graphs of different complexity. The evaluation results show that our visualization approach helps group members to assess who has modified the shared story, how it was modified, what exactly has been modified, and when it has been modified.

This paper presents a novel mediated reality system designed to support collaboration between crime scene investigators during a first analysis on a crime scene, remotely supported by expert colleagues. Requirements elicited from interviews and interactive sessions with practicing crime scene investigators, provided the design criteria for a head mounted display capable of real-time map-making for spatial collaboration. The resulting system is evaluated in a staged crime scene experiment in which lay investigators collaboratively solve a spatial problem with remote experts. The results show that our novel approach to remote spatial interaction with the physical scene enables investigators to tackle current issues on site in collaboration with experts at a distance.

This paper addresses the challenges of providing customized collaboration support to teams of experts. Current groupware systems only provide support for workflow-based collaboration, avoiding new forms of collaboration such as emergent collaboration. Therefore this paper proposes a elastic collaboration approach and its implementation in a groupware system.

In this paper, we present an innovative mediated reality-oriented, real-time software system designed to support multimodal collaboration among remote CSI experts and forensic investigators at the crime scene. Our prototype integrates state-of-the art technologies for stereo navigation, 3D digital mapping and adaptive hand gesture-based user interface for natural interaction. The multimodal interface accepts mouse inputs, audio and hand gestures possibly while interacting with a purposively designed physical object. The evaluation done by a panel of international CSI practitioners [5] using an adapted Burkhardt et al. method [1], shows that our prototype system genuinely fits into the common practice of forensic investigation while clearly boosting the quality of collaboration. At the moment, the system is under consideration for routinely being adopted as part of special procedures by The Forensic Institute in The Hague, The Netherlands [6].

The International Workshop on Semantic Models for Adaptive Interactive Systems (SEMAIS 2012) aims to identify emerging trends in interactive system design using semantic models.

The International Workshop on Semantic Models for Adaptive Interactive Systems (SEMAIS 2011) aims to identify emerging trends in interactive system design and execution using semantic models.

Knowledge work in modern organizations is increasingly performed collaboratively by distributed teams. Such collaboration involves dynamically changing situations. Making the collaboration environment fit the needs of these situations is supported by system-initiated adaptation based on the user and

The International Workshop on Semantic Models for Adaptive Interactive Systems (SEMAIS 2010) aims to identify emerging trends in interactive system design using semantic models.

Collaborative work is characterized by frequently changing situations and corresponding demands for tool support and interaction behavior provided by the collaboration environment. Current approaches to address these changing demands include manual tailoring by end-users and automatic adaptation of single user tools or for individual users. Few systems use context as a basis for adapting collaborative work environments, mostly focusing on document recommendation and awareness provision. In this paper, we present, firstly, a generic four layer framework for modeling and exploiting context. Secondly, a generic adaptation process translating user activity into state, deriving context for a given focus, and executing adaptation rules on this context. Thirdly, a collaboration domain model for describing collaboration environments and collaborative situations. Fourthly, examples of exploiting our approach to support context-based adaptation in four typical collaboration situations: co-location, co-access, co-recommendation, and co-dependency.

Web-based collaborative systems support a variety of complex scenarios. Not only the interaction among one user and a computer has to be modeled but also the interaction among the collaborating users as well. As a result, the user interfaces of many web-based collaborative systems are quite complex, but hardly use approved user interface concepts for the design of interactive systems. Thereby, web-based collaborative systems aggravate the interaction of the users with the system and also with each other. In this article, we describe how the adaptability and usability of such systems can particularly be improved by supporting direct manipulation techniques for navigation as well as tailoring. The new functionality for tailoring and navigation is complemented by new forms of visualizing synchronous awareness information and supporting communication in web-based systems. We show this exemplarily by retrofitting the web-based collaborative system CURE while highlighting the concepts that can be easily transferred to other web-based collaborative systems.

Fostering interaction between end-users and developers is one of the most important issues when developing groupware. Insufficient interaction leads to groupware systems that do not fulfill the group's requirements and thus to low acceptance. Furthermore, as group processes change dynamically the requirements are not static as well. Groupware system development and use, therefore, have to address users' changing needs. Current design methodologies insufficiently focus on this aspect. Therefore, we propose the Oregon Software Development Process (OSDP) that fosters end-user participation throughout the whole groupware life cycle, structures the interaction between end-users and developers, and emphasizes the use of a shared language between users and developers. We illustrate the application of the process with experiences made in an interdisciplinary development project of a collaborative learning platform.

Software engineering education is most often complemented by a software engineering project where a team of students has to develop a large software system. At a distance teaching university such projects challenge the students in communication, coordination, and collaboration, because team members work in different places, many miles away from each other. We present an ECLIPSE-based unified platform that leverages available tools and solutions and discuss the problems involved. Besides using plug-ins that support the students during implementation, our platform integrates a collaborative distant education environment and a software project management system that eases the students' collaboration in the software engineering project