Virtual Reality (VR) depends on haptic feedback to create immersive experiences. Traditional passive proxies align physical props with their virtual counterparts but remain limited in scalability and expressiveness, or require bulky actuators to support reconfiguration. We introduce User-reconfigured Haptics, an approach that utilizes implicit user actions to reconfigure haptic interfaces to extend the gamut of VR haptic experiences. Modular 3D-printed cells are assembled into dynamic interfaces that express diverse haptic properties such as softness and weight. By masking physical reconfigurations with visual (re)mapping, user actions unnoticeably change haptic properties, resulting in user-driven, dynamic haptic experiences. User studies show that our design can provide distinguishable haptic experiences and is perceived as realistic and enjoyable in a VR task. We further showcase four applications: a fishing rod that changes weight and flexibility, a dynamic desktop of pressable buttons, a glove with adjustable squeezing, and a crossbow with variable pulling resistance.

The rapid advancement of the automotive industry toward automated and semi-automated vehicles has rendered traditional methods of vehicle interaction, such as touch-based and voice command systems, inadequate for a widening range of non-driving related tasks, such as referencing objects outside of the vehicle. Consequently, research has shifted toward gestural input (e.g., hand, gaze, and head pose gestures) as a more suitable mode of interaction during driving. However, due to the dynamic nature of driving and individual variation, there are significant differences in drivers’ gestural input performance. While, in theory, this inherent variability could be moderated by substantial data-driven machine learning models, prevalent methodologies lean toward constrained, single-instance trained models for object referencing. These models show a limited capacity to continuously adapt to the divergent behaviors of individual drivers and the variety of driving scenarios. To address this, we proposed, in our previous work, an incremental learning approach, adapting to changing user behavior. Although this method proved superior over state-of-the-art approaches, it still lacked in performance and had some computational resource limitations, mainly the small-sized memory needed to store a subset of old data for incremental learning. Therefore, in this work, we propose an alternative solution using reinforcement learning that can be used to overcome memory storage limitations, achieving a zero data storage approach for incremental learning without performance loss. Our newly enhanced solution has been added to our previous framework at https://github.com/amrgomaaelhady/IcRegress .

In the tourism industry, Extended Reality (XR) technologies are evolving beyond simple marketing tools into sophisticated platforms that deliver immersive experiences to visitors. While these technologies offer significant potential for enhancing tourism experiences, current research primarily examines the tourist perspective, with limited attention to the professional guides who facilitate these interactions. This study investigates how expert tourist guides adopt and integrate XR technology into their professional practice through a novel process for Expert Adoption of XR technology that theoretically grounds the participatory development process in TAM's acceptance mechanisms. Through a co-design process involving experienced guides, we developed an XR application aimed at complementing, rather than replacing, the guide-tourist relationship. Utilizing the Technology Acceptance Model (TAM) for qualitative research, we identified specific concerns and considerations of tourist guides regarding XR adoption and outlined strategies for addressing these challenges during both development and implementation phases. Our findings reveal task-related, individual, and organizational factors for successful XR integration, as well as misconceptions that must be addressed. We suggest that initial scepticism toward XR can be mitigated by ensuring a positive first encounter with technology. Additionally, recognizing stakeholder preferences for specific XR features and prioritizing the most accepted options can enhance adoption. All relevant components of the task, in our case guided tour, must be considered during development, as neglecting any component can impact the overall acceptance. Furthermore, city guides often feel personally responsible for technological failures, highlighting the need for reliable and stable XR applications. These insights provide a methodology applicable to other domains where expert adoption is essential.

Plant breeders use field trials across locations and years to identify superior plant varieties with traits such as disease resistance and higher yield. However, comparing breeding candidates across locations and years is challenging and resource demanding. To address this, we developed an integrated system that combines data acquisition through a field robot with an immersive virtual reality (VR) interface for remote assessments. The robot autonomously collects images, spectral data and 3D scans of canola breeding trials. Our VR application, developed through a user-centered approach, offers photo-realistic 3D visualizations, enabling breeders to compare candidates across locations and growth stages—capabilities unavailable in field assessments. In a user study, five breeders conducted visual trait scoring in VR to evaluate how well the system supported typical field trial tasks. The results demonstrated consistent scoring patterns among raters. Feedback from breeders indicated that the ability to compare candidates across locations and growth stages enhanced their decision making in trait assessment. This work highlights the potential of combining robotics and VR to transform data-intensive processes in agriculture.

To enhance interactions in VR, many techniques introduce offsets between the virtual and real-world position of users’ hands. Nevertheless, such hand redirection (HR) techniques are only effective as long as they go unnoticed by users—not disrupting the VR experience. While several studies consider how much unnoticeable redirection can be applied, these focus on mid-air floating hands that are disconnected from users’ bodies. Increasingly, VR avatars are embodied as being directly connected with the user’s body, which provide more visual cue anchoring, and may therefore reduce the unnoticeable redirection threshold. In this work, we studied more complete avatars and their effect on the sense of embodiment and the detectability of HR. We found that higher avatar completeness increases embodiment, and we provide evidence for the absence of practically relevant effects on the detectability of HR.

In pursuit of hand redirection techniques that are ever more tailored to human perception, we propose the first algorithm for hand redirection in virtual reality that makes use of saccades, i.e., fast ballistic eye movements that are accompanied by the perceptual phenomenon of change blindness. Our technique combines the previously proposed approaches of gradual hand warping and blink-suppressed hand redirection with the novel approach of saccadic redirection in one unified yet simple algorithm. We compare three variants of the proposed Saccadic & Blink-Suppressed Hand Redirection (SBHR) technique with the conventional approach to redirection in a psychophysical study (N = 25). Our results highlight the great potential of our proposed technique for comfortable redirection by showing that SBHR allows for significantly greater magnitudes of unnoticeable redirection while being perceived as significantly less intrusive and less noticeable than commonly employed techniques that only use gradual hand warping.

Distinguishing target from non-target fixations during visual search is a fundamental building block to understand users’ intended actions and to build effective assistance systems. While prior research indicated the feasibility of classifying target vs. non-target fixations based on eye tracking and electroencephalography (EEG) data, these studies were conducted with explicitly instructed search trajectories, abstract visual stimuli, and disregarded any scene context. This is in stark contrast with the fact that human visual search is largely driven by scene characteristics and raises questions regarding generalizability to more realistic scenarios. To close this gap, we, for the first time, investigate the classification of target vs. non-target fixations during free visual search in realistic scenes. In particular, we conducted a 36-participants user study using a large variety of 140 realistic visual search scenes in two highly relevant application scenarios: searching for icons on desktop backgrounds and finding tools in a cluttered workshop. Our approach based on gaze and EEG features outperforms the previous state-of-the-art approach based on a combination of fixation duration and saccade-related potentials. We perform extensive evaluations to assess the generalizability of our approach across scene types. Our approach significantly advances the ability to distinguish between target and non-target fixations in realistic scenarios, achieving 83.6% accuracy in cross-user evaluations. This substantially outperforms previous methods based on saccade-related potentials, which reached only 56.9% accuracy.

Robot-assisted surgical systems have demonstrated significant potential in enhancing surgical precision and minimizing human errors. However, existing systems cannot accommodate individual surgeons’ unique preferences and requirements. Additionally, they primarily focus on general surgeries (e.g., laparoscopy) and are unsuitable for highly precise microsurgeries, such as ophthalmic procedures. Thus, we propose an image-guided approach for surgeon-centered autonomous agents that can adapt to the individual surgeon’s skill level and preferred surgical techniques during ophthalmic cataract surgery. Our approach trains reinforcement and imitation learning agents simultaneously using curriculum learning approaches guided by image data to perform all tasks of the incision phase of cataract surgery. By integrating the surgeon’s actions and preferences into the training process, our approach enables the robot to implicitly learn and adapt to the individual surgeon’s unique techniques through surgeon-in-the-loop demonstrations. This results in a more intuitive and personalized surgical experience for the surgeon while ensuring consistent performance for the autonomous robotic apprentice. We define and evaluate the effectiveness of our approach in a simulated environment using our proposed metrics and highlight the trade-off between a generic agent and a surgeon-centered adapted agent. Finally, our approach has the potential to extend to other ophthalmic and microsurgical procedures, opening the door to a new generation of surgeon-in-the-loop autonomous surgical robots. We provide an open-source simulation framework for future development and reproducibility at https://github.com/amrgomaaelhady/CataractAdaptSurgRobot.

Enabling multi-fingered robots to choose an appropriate grasp on an object from natural language instructions poses great difficulties for such systems. The diversity, imprecision, and limited information contained in the language make this task particularly challenging. However, speech serves humans as a natural communication interface that can aid robots in adapting to the environment more easily. Therefore, providing robots with relevant data about the objects they interact with is essential for them to understand how to carry out object manipulation tasks. By leveraging Named Entity Recognition (NER) to automatically extract semantic data, our work introduces a novel approach to text-based grasp predictions. Our methodology involves a multistage learning approach using a semantic information extractor that provides significant features to a grasp prediction model. To assess the effectiveness of our approach, we conducted experiments on an existing corpus and two corpora generated by ChatGPT. Our results demonstrate superior performance compared to similar grasp prediction models while overcoming limitations in the literature. Additionally, we open-source our training data for reproducibility and future research advancement.

The rapid advancement of the automotive industry towards automated and semi-automated vehicles has rendered traditional methods of vehicle interaction, such as touch-based and voice command systems, inadequate for a widening range of non-driving related tasks, such as referencing objects outside of the vehicle. Consequently, research has shifted toward gestural input (e.g., hand, gaze, and head pose gestures) as a more suitable mode of interaction during driving. However, due to the dynamic nature of driving and individual variation, there are significant differences in drivers’ gestural input performance. While, in theory, this inherent variability could be moderated by substantial data-driven machine learning models, prevalent methodologies lean towards constrained, single-instance trained models for object referencing. These models show a limited capacity to continuously adapt to the divergent behaviors of individual drivers and the variety of driving scenarios. To address this, we propose IcRegress, a novel regression-based incremental learning approach that adapts to changing behavior and the unique characteristics of drivers engaged in the dual task of driving and referencing objects. We suggest a more personalized and adaptable solution for multimodal gestural interfaces, employing continuous lifelong learning to enhance driver experience, safety, and convenience. Our approach was evaluated using an outside-the-vehicle object referencing use case, highlighting the superiority of the incremental learning models adapted over a single trained model across various driver traits such as handedness, driving experience, and numerous driving conditions. Finally, to facilitate reproducibility, ease deployment, and promote further research, we offer our approach as an open-source framework at https://github.com/amrgomaaelhady/IcRegress.

Dynamic hand gesture recognition is crucial for human-machine interfaces in the automotive domain. However, creating a diverse and comprehensive dataset of hand gestures can be challenging and time-consuming, especially in dynamic dual-task situations like driving. To address these challenges, we propose using synthetic gesture datasets generated by virtual 3D models as an alternative. Our framework synthesizes realistic hand gestures using a combination of 3D models and animation software, particularly utilizing Unreal Engine. This approach enables the creation of diverse and customizable gesture datasets, reducing the risk of overfitting and improving the model’s generalizability. Specifically, our framework generates natural-looking dynamic hand gestures with multiple variants, including gesture speed, performance, and hand shape. Moreover, we simulate various camera locations, such as above the driver and behind the wheel, and different camera types, such as RGB, infrared, and depth cameras, without incurring additional time and cost to obtain these cameras. Our experiments demonstrate that our proposed framework, SynthoGestures (available at https://github.com/amrgomaaelhady/SynthoGestures), can augment or replace existing real-hand datasets with additional enhancement in gesture recognition accuracy. Our tool for generating synthetic static and dynamic hand gestures saves time and effort in creating large datasets, facilitating the faster development of gesture recognition systems for automotive applications.

One of the advantages of formalizing domain knowledge in OWL ontologies is that one can use reasoning systems to infer implicit information automatically. However, it is not always straightforward to understand why certain entailments are inferred, and others are not. The popular ontology editor Protégé offers two explanation services to deal with this issue: justifications for OWL 2 DL ontologies, and proofs generated by the reasoner ELK for lightweight OWL 2 EL ontologies. Since justifications are often insufficient for explaining inferences, there is thus only little tool support for more comprehensive explanations in expressive ontology languages, and there is no tool support at all to explain why something was not derived. In this paper, we present Evee, a Java library and a collection of plug-ins for Protégé that offers advanced explanation services for both inferred and missing entailments. Evee explains inferred entailments using proofs in description logics up to ALCH. Missing entailments can be explained using counterexamples and abduction. We evaluated the effectiveness and the interface design of our plug-ins with description logic experts, ontology engineers, and students in two user studies. In these experiments, we were able to not only validate the tool but also gather feedback and insights to improve the existing designs.

Many illusion and interaction techniques in Virtual Reality (VR) rely on Hand Redirection (HR), which has proved to be effective as long as the introduced offsets between the position of the real and virtual hand do not noticeably disturb the user experience. Yet calibrating HR offsets is a tedious and time-consuming process involving psychophysical experimentation, and the resulting thresholds are known to be affected by many variables—limiting HR’s practical utility. As a result, there is a clear need for alternative methods that allow tailoring HR to the perceptual boundaries of individual users. We conducted an experiment with 18 participants combining movement, eye gaze and EEG data to detect HR offsets Below, At, and Above individuals’ detection thresholds. Our results suggest that we can distinguish HR At and Above from no HR. Our exploration provides a promising new direction with potentially strong implications for the broad field of VR illusions.

Teaching medical practical and soft skills in clinical routines is increasingly difficult, and manikin or actor-based simulations have gained popularity in the last decades. These simulations, however, hardly scale with the demand, are commonly insufficient to train crucial clinical competencies, and cannot portray complex visual and dynamic symptomatologies as required in, for example, brain death examinations. In this paper, we explore the requirements and challenges of integrating a large-scale high-throughput VR setup into a real medical curriculum and describe our approaches and implementation. Therefore we extend and evaluate an interactive virtual reality-based simulation for training brain death diagnostics in a virtual intensive care environment, featuring a fully reactive simulated patient. To enable the required scalability we integrated the simulation into a dedicated hardware and software framework, enabling 12 simultaneous VR trainings which are controlled by a centralized server system. Using this setup we continuously collected feedback on the application's usability and realism from hundreds of students to gain first insights into the applicability of large-scale VR-based learning systems in real course designs. After integrating this feedback, we conducted a controlled curricular study in which we compared the virtual brain death simulation with the classical manikin-based training approach. Our results indicate that the immersive learning experience is perceived to be more realistic and engaging and is overall preferred by the students while also providing the same learning effect as the alternatives.

In gamification users show increased motivation and engagement towards tasks. So far, this effect has mostly been empirically tested based on the visual depiction of game elements, while research on the use and addition of auditory aspects is sparse. In this work we investigate the effect of different modalities of game elements (auditory, visual, audiovisual) on user experience, perception and performance in a gamified image-tagging task. We approached this via an online validation survey to find suitable sound effects (N=50), a main quantitative study (N=109) and a qualitative semi-structured interview (N=9). Our results show that while visual gamification increases performance, auditory and audiovisual gamification had no effect. However, they were shown to have an influence on the user's flow state. Our qualitative follow-up study shed light on underlying reasons and revealed each modality has its own drawbacks and advantages and that combining both visual and auditory aspects was preferred by participants.

Entertainment media, including video games, utilize background music (BGM) to enhance ambiance and gameplay, with a growing trend of players replacing in-game audio with personal music for potentially improved gameplay. Yet, research has not systematically investigated the connection between liked or disliked BGM and player behavior and experience. With our work, we make this step by letting users play the popular and well-researched game Tetris with BGM of their choice, and go one step further by adding music to the game that they specifically dislike, as this direction has been neglected thus far, but could prove interesting to target certain player behavior or experience. In our lab-study (N=31) we found that liked BGM increased engagement with the game and enhanced player experience, while disliked music reduced immersion. These findings suggest that music preferences should be taken into greater account in game design, e.g. by offering more options for BGM customization.

Autonomous vehicles (AVs; SAE levels 4 and 5) face substantial challenges regarding acceptance and UX. Novel human-machine interfaces (HMIs) providing transparent system information could account for those and facilitate adoption. However, since the availability of AVs for early concept studies is limited, context-based interface prototyping is required. This paper demonstrates the prototype and wizard-of-oz-based on-road evaluation of a futuristic windshield HMI concept that visualizes real-time object detections via augmented reality (AR). In a mixed-methods within-subjects study (N = 30), participants assessed three early-stage concept variants to explore whether object detection visualization can counteract the aforementioned challenges. The findings confirm that transparent system feedback can increase understandability, perceived usefulness, and hedonic UX, but the amount and the timing of the provided information are crucial. The applied prototyping method proved suitable for investigating HMI concepts with real-time AR on urban roads. Based on a critical discussion, the paper concludes with design and prototyping recommendations.

As more and more virtual reality (VR) headsets support eye tracking, recent techniques started to use eye blinks to induce unnoticeable manipulations to the virtual environment, e.g., to redirect users’ actions. However, to exploit their full potential, more control over users’ blinking behavior in VR is required. To this end, we propose a set of reflex-based blink triggers that are suited specifically for VR. In accordance with blink-based techniques for redirection, we formulate (i) effectiveness, (ii) efficiency, (iii) reliability, and (iv) unobtrusiveness as central requirements for successful triggers. We implement the soft- and hardware-based methods and compare the four most promising approaches in a user study. Our results highlight the pros and cons of the tested triggers, and show those based on the menace, corneal, and dazzle reflexes to perform best. From these results, we derive recommendations that help choosing suitable blink triggers for VR applications.

Experiencing virtual environments is often limited to abstract interactions with objects. Physical proxies allow users to feel virtual objects, but are often inaccessible. We present the VoxelHap toolkit which enables users to construct highly functional proxy objects using Voxels and Plates. Voxels are blocks with special functionalities that form the core of each physical proxy. Plates increase a proxy’s haptic resolution, such as its shape, texture or weight. Beyond providing physical capabilities to realize haptic sensations, VoxelHap utilizes VR illusion techniques to expand its haptic resolution. We evaluated the capabilities of the VoxelHap toolkit through the construction of a range of fully functional proxies across a variety of use cases and applications. In two experiments with 24 participants, we investigate a subset of the constructed proxies, studying how they compare to a traditional VR controller. First, we investigated VoxelHap’s combined haptic feedback and second, the trade-offs of using ShapePlates. Our findings show that VoxelHap’s proxies outperform traditional controllers and were favored by participants.

Rendering haptic feedback for interactions with virtual objects is an essential part of effective virtual reality experiences. In this work, we explore providing haptic feedback for rotational manipulations, e.g., through knobs. We propose the use of a Pseudo-Haptic technique alongside a physical proxy knob to simulate various physical resistances. In a psychophysical experiment with 20 participants, we found that designers can introduce unnoticeable offsets between real and virtual rotations of the knob, and we report the corresponding detection thresholds. Based on these, we present the Pseudo-Haptic Resistance technique to convey physical resistance while applying only unnoticeable pseudo-haptic manipulation. Additionally, we provide a first model of how C/D gains correspond to physical resistance perceived during object rotation, and outline how our results can be translated to other rotational manipulations. Finally, we present two example use cases that demonstrate the versatility and power of our approach.

Hand redirection is effective so long as the introduced offsets are not noticeably disruptive to users. In this work we investigate the use of physiological and interaction data to detect movement discrepancies between a user's real and virtual hand, pushing towards a novel approach to identify discrepancies which are too large and therefore can be noticed. We ran a study with 22 participants, collecting EEG, ECG, EDA, RSP, and interaction data. Our results suggest that EEG and interaction data can be reliably used to detect visuo-motor discrepancies, whereas ECG and RSP seem to suffer from inconsistencies. Our findings also show that participants quickly adapt to large discrepancies, and that they constantly attempt to establish a stable mental model of their environment. Together, these findings suggest that there is no absolute threshold for possible non-detectable discrepancies; instead, it depends primarily on participants' most recent experience with this kind of interaction.

On the way towards novel hand redirection (HR) techniques that make use of change blindness, the next step is to take advantage of saccades for hiding body warping. A prerequisite for saccadic HR algorithms, however, is to know how much the user’s virtual hand can unnoticeably be offset during saccadic suppression. We contribute this knowledge by conducting a psychophysical experiment, which lays the ground for upcoming HR techniques by exploring the perceptual detection thresholds (DTs) of hand offset injected during saccades. Our findings highlight the pivotal role of saccade direction for unnoticeable hand jumps, and reveal that most offset goes unnoticed when the saccade and hand move in opposite directions. Based on the gathered perceptual data, we derived a model that considers the angle between saccade and hand offset direction to predict the DTs of saccadic hand jumps.

We propose a novel open-source software toolkit to support researchers in the domains of human-computer interaction (HCI) and virtual reality (VR) in conducting psychophysical experiments. Our toolkit is designed to work with the widely-used Unity engine and is implemented in C# and Python. With the toolkit, researchers can easily set up, run, and analyze experiments to find perceptual detection thresholds using the adaptive weighted up/down method, also known as the staircase procedure. Besides being straightforward to integrate in Unity projects, the toolkit automatically stores experiment results, features a live plotter that visualizes answers in real time, and offers scripts that help researchers analyze the gathered data using statistical tests.

Visuo-haptic illusions are a method to expand proxy-based interactions in VR by introducing unnoticeable discrepancies between the virtual and real world. Yet how different design variables affect the illusions with proxies is still unclear. To unpack a subset of variables, we conducted two user studies with 48 participants to explore the impact of (1) different grasping types and movement trajectories, and (2) different grasping types and object masses on the discrepancy which may be introduced. Our Bayes analysis suggests that grasping types and object masses (≤ 500 g) did not noticeably affect the discrepancy, but for movement trajectory, results were inconclusive. Further, we identified a significant difference between (un)restricted movement trajectories. Our data shows considerable differences in participants’ proprioceptive accuracy, which seem to correlate with their prior VR experience. Finally, we illustrate the impact of our key findings on the visuo-haptic illusion design process by showcasing a new design workflow.

Silent speech Brain-Computer Interfaces (BCIs) try to decode imagined speech from brain activity. Those BCIs require a tremendous amount of training data usually collected during mentally and physically exhausting sessions in which participants silently repeat words presented on a screen for several hours. Within this work we present an approach to overcome those exhausting sessions by training a silent speech classifier on data recorded while speaking certain words and transferring this classifier to EEG data recorded during silent repetition of the same words. This approach does not only allow for a less mentally and physically exhausting training procedure but also for a more productive one as the overt speech output can be used for interaction while the classifier for silent speech is trained simultaneously. We evaluated our approach in a study in which 15 participants navigated a virtual robot on a screen in a game like scenario through a maze once with 5 overtly spoken and once with the same 5 silently spoken command words. In an offline analysis we trained a classifier on overt speech data and let it predict silent speech data. Our classification results do not only show successful results for the transfer (61.78%) significantly above chance level but also comparable results to a standard silents speech classifier (71.48%) trained and tested on the same data. These results illustrate the potential of the method to replace the currently tedious training procedures for silent speech BCIs with a more comfortable, engaging and productive approach by a transfer from overt to silent speech.

Optical see-through head-mounted displays (OST HMDs) are becoming increasingly popular as they get better and smaller. One application area is interaction with virtual content, which is more intuitive when using physical objects as tangibles. Since it is not possible to use a matching replica for each virtual object, it is necessary to identify physical objects that can represent several different virtual objects. As a first step, we investigated to what extent a physical object can differ in size from its virtual counterpart.Since the perception of content in optical see-through Augmented Reality (OST AR) is strongly influenced by the ambient lighting, the illumination intensity was considered in our study. We investigated three indoor lighting conditions and their effects on the perception of seven different size variations between the physical object and its virtual overlay.The results of the study show that there is a decrease in usability and presence with increasing illuminance. However, this cannot be avoided when applications are run under realistic interior lighting conditions. Furthermore, the results demonstrate that the size ranges in which a physical object can deviate from its virtual counterpart without having a strong negative impact on usability, presence and performance increase with increasing environmental illumination. Therefore, it is possible to interact with even smaller and even larger physical props to manipulate the associated virtual content under brighter lighting conditions.

The perception of gamification elements differs across users, which is why personalizing gamified systems is important. Past research showed that the Hexad user types model is particularly suitable for this purpose by demonstrating correlations between user types and gamification elements. However, previous studies were mostly survey-based, i.e. relied on participants' rating of gamification elements based on e.g. textual descriptions or storyboards. Thus, the question whether personalization based on Hexad user types provides benefits in implemented gameful systems was neglected. We contribute to this by investigating the effects of (contra-) tailoring the set of gamification elements to a user's Hexad type on performance and user experience, assessed with survey and physiological measures, in an image tagging context. In a lab study (N=29), we found that gamification increases performance and affects psychophysiological measures of arousal. Moreover, we demonstrate that personalization increases enjoyment, positively-valenced affective experiences and participants' absorption in the task at hand.

In the context of game live-streams, we focus on whether audience influence options can be extended beyond the game, e.g., to the streamers' environment (e.g., controlling their room lighting), the streamers themselves (e.g., activating vibration feedback on their forearms) and the streamers' hardware and peripheral devices (e.g., switching key bindings). An online study (n=81) showed that viewers have mixed feelings and that the context matters, as only a few game genres are assessed as suitable for such an experience. In a lab-study six streamers experienced the idea while playing a horror game. They saw appeal in extending the influence space, but also again highlighted that context matters and customization options are necessary. The studies hint that although this interactivity is from a technical perspective independent of the content streamed, in practice it is not, as the perception is moderated by e.g., the game and why the streamer is playing it.

Abstract Although privacy settings are important not only for data privacy, but also to prevent hacking attacks like social engineering that depend on leaked private data, most users do not care about them. Research has tried to help users in setting their privacy settings by using some settings that have already been adapted by the user or individual factors like personality to predict the remaining settings. But in some cases, neither is available. However, the user might have already done privacy settings in another domain, for example, she already adapted the privacy settings on the smartphone, but not on her social network account. In this article, we investigate with the example of four domains (social network posts, location sharing, smartphone app permission settings and data of an intelligent retail store), whether and how precise privacy settings of a domain can be predicted across domains. We performed an exploratory study to examine which privacy settings of the aforementioned domains could be useful, and validated our findings in a validation study. Our results indicate that such an approach works with a prediction precision about 15%–20% better than random and a prediction without input coefficients. We identified clusters of domains that allow model transfer between their members, and discuss which kind of privacy settings (general or context-based) leads to a better prediction accuracy. Based on the results, we would like to conduct user studies to find out whether the prediction precision is perceived by users as a significant improvement over a “one-size-fits-all” solution, where every user is given the same privacy settings.

Abstract Producing and consuming live-streamed content is a growing trend attracting many people today. While the actual content that is streamed is diverse, one especially popular context is games. Streamers of gaming content broadcast how they play digital or analog games, attracting several thousand viewers at once. Previous scientific work has revealed that different motivations drive people to become viewers, which apparently impacts how they interact with the offered features and which streamers’ behaviors they appreciate. In this paper, we wanted to understand whether viewers’ motivations can be formulated as viewer types and systematically measured. We present an exploratory factor analysis (followed by a validation study) with which we developed a 25-item questionnaire assessing five different viewer types. In addition, we analyzed the predictive validity of the viewer types for existing and potential live stream features. We were able to show that a relationship between the assessed viewer type and preferences for streamers’ behaviors and features in a stream exists, which can guide fellow researchers and streamers to understand viewers better and potentially provide more suitable experiences.

Rashad Albo Jamara, Nico Herbig, Antonio Krüger, Josef van Genabith. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Providing haptic feedback when manipulating virtual objects is an essential part of immersive virtual reality experiences; however, it is challenging to replicate all of an object's properties and characteristics. We propose the use of visuo-haptic illusions alongside physical proxies to enhance the scope of proxy-based interactions with virtual objects. In this work, we focus on two manipulation techniques, linear translation and stretching across different distances, and investigate how much discrepancy between the physical proxy and the virtual object may be introduced without participants noticing. In a study with 24 participants, we found that manipulation technique and travel distance significantly affect the detection thresholds, and that visuo-haptic illusions impact performance and accuracy. We show that this technique can be used to enable functional proxy objects that act as stand-ins for multiple virtual objects, illustrating the technique through a showcase VR-DJ application.

Compared to fully manual translation, postediting (PE) machine translation (MT) output can save time and reduce errors.Automatic word-level quality estimation (QE) aims to predict the correctness of words in MT output and holds great promise to aid PE by flagging problematic output.Quality of QE is crucial, as incorrect QE might lead to translators missing errors or wasting time on already correct MT output.Achieving accurate automatic word-level QE is very hard, and it is currently not known (i) at what quality threshold QE is actually beginning to be useful for human PE, and (ii), how to best present wordlevel QE information to translators.In particular, should word-level QE visualization indicate uncertainty of the QE model or not?In this paper, we address both research questions with real and simulated word-level QE, visualizations, and user studies, where time, subjective ratings, and quality of the final translations are assessed.Results show that current wordlevel QE models are not yet good enough to support PE. Instead, quality levels of ≥ 80% F1 are required.For helpful quality levels, a visualization reflecting the uncertainty of the QE model is preferred.Our analysis further shows that speed gains achieved through QE are not merely a result of blindly trusting the QE system, but that the quality of the final translations also improves.The threshold results from the paper establish a quality goal for future wordlevel QE research.

In highly automated driving (HAD), it is still an open question how machines can safely hand over control to humans, and if an advance notice with additional explanations can be beneficial in critical situations. Conceptually, use of formal methods from AI – description logic (DL) and automated planning – in order to more reliably predict when a handover is necessary, and to increase the advance notice for handovers by planning ahead at runtime, can provide a technological support for explanations using natural language generation. However, in this work we address only the user’s perspective with two contributions: First, we evaluate our concept in a driving simulator study (N=23) and find that an advance notice and spoken explanations were preferred over classical handover methods. Second, we propose a framework and an example test scenario specific to handovers that is based on the results of our study.

Optical see-through AR headsets are becoming increasingly attractive for many applications. Interaction with the virtual content is usually achieved via hand gestures or with controllers. A more seamless interaction between the real and virtual world can be achieved by using tangible objects to manipulate the virtual content. Instead of interacting with detailed physical replicas, working with abstractions allows a single physical object to represent a variety of virtual objects. These abstractions would differ from their virtual representations in shape, size, texture and material. This paper investigates for the first time in optical see-through AR whether size variations are possible without major losses in performance, usability and immersion. The conducted study shows that size can be varied within a limited range without significantly affecting task completion times as well as feelings of disturbance and presence. Stronger size deviations are possible for physical objects smaller than the virtual object than for larger physical objects.

Steady-State-Visually-Evoked-Potential (SSVEP) Brain-Computer Interfaces (BCIs) make use of flickering stimuli to determine the target a user is looking at and select commands accordingly. Those types of BCI can be operated with little to no training, achieve high classification accuracies and are robust in application. A drawback of this approach is the reduced user comfort due to the constant flickering of the stimuli which can be annoying and tiring to look at. Existing studies addressing this issue try to make use of motion to disguise the oscillating patterns. However, this makes them look abstract and restricts the design of those applications as those patterns do not blend in to conventional user interfaces. In this work we introduce the concept of spinning icons to evoke SSVEPs. The icons are rotating in a certain frequency around their vertical axis and are supposed to appear more natural and be less stressing for the human eye. Furthermore this concept is not bound to any kind of abstract motion based pattern but rather supposed to work with any type of icon or image. The newly designed stimuli were evaluated in an application-oriented scenario and compared to standard and state-of-the-art movement-based SSVEP stimuli regarding the classification accuracy and experienced visual fatigue. The results show that the newly created stimuli performed equally well and partially even better in terms of classification accuracy and were rated throughout better concerning visual fatigue by the study participants. This work therefore lays the foundation for more comfortable SSVEP-BCIs which can be used with basically every icon or UI element spinning around their vertical axis.

One challenge in highly automated driving is the safe transfer of control (ToC). A safe ToC requires estimating the take-over time depending on the driver’s state in different environmental conditions, to adapt the timing and design of the ToC request. We introduce environmental complexity as one factor that affects the ToC time. In a driving simulator experiment (N=12), the participants drove in five scenes having different environmental complexities (i.e. density and height of the background objects) with and without a secondary task. The results revealed that the ToC time is proportional to the environmental complexity. Thus, in the same driving task and the same traffic, an increasing environmental complexity yields higher ToC times in both conditions, with and without a secondary task. Our model of environmental complexity is a first step towards measuring the complexity of the real world, for a better prediction of ToC times in highly automated driving.

In autonomous mobility-on-demand (AMoD) systems, passengers will solely interact with autonomous vehicles via digital user interfaces (UIs). Hence, UIs are crucial for acceptance and user experience (UX). As a foundation for deriving empirically grounded design guidelines, we investigate two approaches for mobile interaction: chatbots and ‘classic’ graphical UIs (GUIs). We evaluated prototypes of both in expert studies (nGUI = 6; nChatbot = 5) and a between-subjects simulator user study (n = 34). The latter enabled participants to experience a complete AMoD journey. While both concepts receive good acceptance and positive UX evaluations, the GUI results in significantly higher attractiveness and user satisfaction ratings. A significant interaction effect reveals a higher intention to use the chatbot in scenarios with a change of plans, but the GUI in ‘happy path’ scenarios. Interview data and emotion curves support this effect. Balancing the concepts‘ advantages and disadvantages, we provide design recommendations and propose to use GUI-based mobile applications with integrated (text-based) conversational elements for future human-AMoD interaction.

Modern, globalized lifestyles make physical separation of family members an inevitable part of life. Due to changes in routines and social roles, remaining connected to distant loved ones is challenging and adversely affects psychological well-being. To support relatedness, we created FamilyFlower, an artificial flower aimed at fostering a connection between remote households by bringing awareness of everyday activities and offering basic expressivity. In its environment, each device detects human presence, movement, sound, and touch. In the remote household, the paired prototype responds by respectively opening the flower bud, actuating the stem, altering the seeds’ color, and dispensing a fragrance. In a 2-week study of deployment between 2 family members, we collected initial impressions of our prototype, and identified key aspects our system could support in future research. From our results, we see that FamilyFlower supports an awareness of ongoing activities in a distant household and built a feeling of connectedness.

Effective haptic feedback in virtual reality (VR) is an essential element for creating convincing immersive experiences. To design such feedback, state-of-the-art VR setups provide APIs for programmatically generating controller vibration patterns. While tools for designing vibrotactile feedback keep evolving, they often require expert knowledge and rarely support direct manipulation methods for mapping feedback to user interactions within the VR environment. To address these challenges, we contribute a novel concept called Weirding Haptics, that supports fast-prototyping by leveraging the user’s voice to design such feedback while manipulating virtual objects in-situ. Through a pilot study (N = 9) focusing on how tactile experiences are vocalized during object manipulation, we identify spatio-temporal mappings and supporting features needed to produce intended vocalizations. To study our concept, we built a VR design tool informed by the results of the pilot study. This tool enables users to design tactile experiences using their voice while manipulating objects, provides a set of modifiers for fine-tuning the created experiences in VR, and allows to rapidly compare various experiences by feeling them. Results from a validation study (N = 8) show that novice hapticians can vocalize experiences and refine their designs with the fine-tuning modifiers to match their intentions. We conclude our work by discussing uncovered design implications for direct manipulation and vocalization of vibrotactile feedback in immersive virtual environments.

Tactile feedback of an object’s surface enables us to discern its material properties and affordances. This understanding is used in digital fabrication processes by creating objects with high-resolution surface variations to influence a user’s tactile perception. As the design of such surface haptics commonly relies on knowledge from real-life experiences, it is unclear how to adapt this information for digital design methods. In this work, we investigate replicating the haptics of real materials. Using an existing process for capturing an object’s microgeometry, we digitize and reproduce the stable surface information of a set of 15 fabric samples. In a psychophysical experiment, we evaluate the tactile qualities of our set of original samples and their replicas. From our results, we see that direct reproduction of surface variations is able to influence different psychophysical dimensions of the tactile perception of surface textures. While the fabrication process did not preserve all properties, our approach underlines that replication of surface microgeometries benefits fabrication methods in terms of haptic perception by covering a large range of tactile variations. Moreover, by changing the surface structure of a single fabricated material, its material perception can be influenced. We conclude by proposing strategies for capturing and reproducing digitized textures to better resemble the perceived haptics of the originals.

Many interaction techniques in virtual reality break with the 1-to-1 mapping from real to virtual space. Instead, specialized techniques for 3D interaction and haptic retargeting leverage hand redirection, offsetting the virtual hand rendering from the real hand position. To achieve unnoticeable hand redirection, however, the utilization of change blindness phenomena has not been systematically explored. Inspired by recent advances in the domain of redirected walking, we present the first hand redirection technique that makes use of blink-induced visual suppression and corresponding change blindness. We introduce Blink-Suppressed Hand Redirection (BSHR) to study the feasibility and detectability of hand redirection based on blink suppression. Our technique is based on Cheng et al.'s (2017) [9] body warping algorithm and instantaneously shifts the virtual hand when the user's vision is suppressed during a blink. Additionally, it can be configured to continuously increment hand offsets when the user's eyes are opened, limited to an extent below detection thresholds. In a psychophysical experiment, we verify that unnoticeable blink-suppressed hand redirection is possible even in worst -case scenarios, and derive the corresponding conservative detection thresholds (CDTs). Moreover, our results show that the range of unnoticeable redirection can be increased by combining continuous warping and blink-suppressed instantaneous shifts. As an additional contribution, we derive the CDTs for Cheng et al.'s (2017) [9] redirection technique that does not leverage blinks.

To provide immersive haptic experiences, proxy-based haptic feedback systems for virtual reality (VR) face two central challenges: (1) similarity, and (2) colocation. While to solve challenge (1), physical proxy objects need to be sufficiently similar to their virtual counterparts in terms of haptic properties, for challenge (2), proxies and virtual counterparts need to be sufficiently colocated to allow for seamless interactions. To solve these challenges, past research introduced, among others, two successful techniques: (a) Dynamic Passive Haptic Feedback (DPHF), a hardware-based technique that leverages actuated props adapting their physical state during the VR experience, and (b) Haptic Retargeting, a software-based technique leveraging hand redirection to bridge spatial offsets between real and virtual objects. Both concepts have, up to now, not ever been studied in combination. This paper proposes to combine both techniques and reports on the results of a perceptual and a psychophysical experiment situated in a proof-of-concept scenario focused on the perception of virtual weight distribution. We show that users in VR overestimate weight shifts and that, when DPHF and HR are combined, significantly greater shifts can be rendered, compared to using only a weight-shifting prop or unnoticeable hand redirection. Moreover, we find the combination of DPHF and HR to let significantly larger spatial dislocations of proxy and virtual counterpart go unnoticed by users. Our investigation is the first to show the value of combining DPHF and HR in practice, validating that their combination can better solve the challenges of similarity and colocation than the individual techniques can do alone.

Abstract Personalizing gameful applications is essential to account for interpersonal differences in the perception of gameful design elements. Considering that an increasing number of people lead sedentary lifestyles, using personalized gameful applications to encourage physical activity is a particularly relevant domain. In this article, we investigate behavior change intentions and Hexad user types as factors to personalize gameful fitness applications. We first explored the potential of these two factors by analyzing differences in the perceived persuasiveness of gameful design elements using a storyboards-based online study ( $$N=178$$ N=178 ). Our results show several significant effects regarding both factors and thus support the usefulness of them in explaining perceptual differences. Based on these findings, we implemented “Endless Universe,” a personalized gameful application encouraging physical activity on a treadmill. We used the system in a laboratory study ( $$N=20$$ N=20 ) to study actual effects of personalization on the users’ performance, enjoyment and affective experiences. While we did not find effects on the immediate performance of users, positive effects on user experience-related measures were found. The results of this study support the relevance of behavior change intentions and Hexad user types for personalizing gameful fitness systems further.

Autonomous vehicles (AVs; SAE levels 4 and 5) develop rapidly, whereas appropriate methods for interface design and development for such driverless vehicles are still in their infancy. This paper presents a simple approach for context-based prototyping and evaluation of human-machine interfaces for (shared) AVs in public transportation. It demonstrates how to set up a lightweight immersive video-based AV simulator using real-world video and audio footage captured in urban traffic. In two user studies (n1 = 9; n2 = 31) we investigate presence perception and simulator sickness to provide initial evidence for the suitability of this cost-effective method. Furthermore, with the intent to increase presence perception and technology acceptance, we combine the AV simulator with a human actor imitating a passenger that gets on and off a shared AV ride.

Nico Herbig, Tim Düwel, Santanu Pal, Kalliopi Meladaki, Mahsa Monshizadeh, Antonio Krüger, Josef van Genabith. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

This work presents a baseline for remotely monitoring patients with Alzheimer's disease (AD), on an everyday's life basis, using distributed tangible tabletops. Our proposed system aims to help individuals with AD live at home independently of physical presence of caregivers, nurses, and/or family members, since these latter can assist the patients with AD and monitor their activity through a virtually duplicated (dual reality) environment. This monitoring happens in real time, and monitors can encourage and remind patients with AD to take actions, while respecting their privacy. The interaction is achieved by moving physical (tangible) and/or virtual (digital) objects on the tabletop's surface, or by displaying messages and reminders on its surface. A first prototype is described and illustrated, as well as propositions to validate and extend the system.

In automatic post-editing (APE) it makes sense to condition post-editing (pe) decisions on both the source (src) and the machine translated text (mt) as input.This has led to multi-encoder based neural APE approaches.A research challenge now is the search for architectures that best support the capture, preparation and provision of src and mt information and its integration with pe decisions.In this paper we present an efficient multi-encoder based APE model, called transference.Unlike previous approaches, it (i) uses a transformer encoder block for src, (ii) followed by a decoder block, but without masking for self-attention on mt, which effectively acts as second encoder combining src → mt, and (iii) feeds this representation into a final decoder block generating pe.Our model outperforms the best performing systems by 1 BLEU point on the WMT 2016, 2017, and 2018 English-German APE shared tasks (PBSMT and NMT).Furthermore, the results of our model on the WMT 2019 APE task using NMT data shows performance at the level of the state-of-the-art.The inference time of our model is similar to the vanilla transformer-based NMT system although our model deals with two separate encoders.We further investigate the importance of our newly introduced second encoder and find that decreasing the number of layers hurts performance, while reducing the number of layers of the decoder does not matter much.

In this paper, we analyze a wide range of physiological, behavioral, performance, and subjective measures to estimate cognitive load (CL) during e-learning. To the best of our knowledge, the analyzed sensor measures comprise the most diverse set of features from a variety of modalities that have to date been investigated in the e-learning domain. Our focus lies on predicting the subjectively reported CL and difficulty as well as intrinsic content difficulty based on the explored features. A study with 21 participants, who learned through videos and quizzes in a Moodle environment, shows that classifying intrinsic content difficulty works better for quizzes than for videos, where participants actively solve problems instead of passively consuming videos. Regression analysis for predicting the subjectively reported level of CL and difficulty also works with very low error within content topics. Among the explored feature modalities, eye-based features yield the best results, followed by heart-based and then skin-based measures. Furthermore, combining multiple modalities results in better performance compared to using a single modality. The presented results can guide researchers and developers of cognition-aware e-learning environments by suggesting modalities and features that work particularly well for estimating difficulty and CL.

In many professional domains, relevant processes are documented as abstract process models, such as event-driven process chains (EPCs). EPCs are traditionally visualized as 2D graphs and their size varies with the complexity of the process. While process modeling experts are used to interpreting complex 2D EPCs, in certain scenarios such as, for example, professional training or education, also novice users inexperienced in interpreting 2D EPC data are facing the challenge of learning and understanding complex process models. To communicate process knowledge in an effective yet motivating and interesting way, we propose a novel virtual reality (VR) interface for non-expert users. Our proposed system turns the exploration of arbitrarily complex EPCs into an interactive and multi-sensory VR experience. It automatically generates a virtual 3D environment from a process model and lets users explore processes through a combination of natural walking and teleportation. Our immersive interface leverages basic gamification in the form of a logical walkthrough mode to motivate users to interact with the virtual process. The generated user experience is entirely novel in the field of immersive data exploration and supported by a combination of visual, auditory, vibrotactile and passive haptic feedback. In a user study with N=27 novice users, we evaluate the effect of our proposed system on process model understandability and user experience, while comparing it to a traditional 2D interface on a tablet device. The results indicate a tradeoff between efficiency and user interest as assessed by the UEQ novelty subscale, while no significant decrease in model understanding performance was found using the proposed VR interface. Our investigation highlights the potential of multi-sensory VR for less time-critical professional application domains, such as employee training, communication, education, and related scenarios focusing on user interest.

While the use of ad blockers prevents negative impacts of advertising on user experience, it poses a serious threat to the business model of commercial web services and freely available content on the web. As an alternative, we investigate the user enjoyment and the advertising effectiveness of playfully deactivating online ads. We created eight game concepts, performed a pre-study assessing the users' perception of them (N=50) and implemented three well-perceived ones. In a lab study (N=72), we found that these game concepts are more enjoyable than deactivating ads without game elements. Additionally, one game concept was even preferred over using an ad blocker. Notably, playfully deactivating ads was shown to have a positive impact on users' brand and product memory, enhancing the advertising effectiveness. Thus, our results indicate that playfully deactivating ads is a promising way of bridging the gap between user enjoyment and effective advertising.

Experiencing materials in virtual reality (VR) is enhanced by combining visual and haptic feedback. While VR easily allows changes to visual appearances, modifying haptic impressions remains challenging. Existing passive haptic techniques require access to a large set of tangible proxies. To reduce the number of physical representations, we look towards fabrication to create more versatile counterparts. In a user study, 3D-printed hairs with length varying in steps of 2.5 mm were used to influence the feeling of roughness and hardness. By overlaying fabricated hair with visual textures, the resolution of the user's haptic perception increased. As changing haptic sensations are able to elicit perceptual switches, our approach can extend a limited set of textures to a much broader set of material impressions. Our results give insights into the effectiveness of 3D-printed hair for enhancing texture perception in VR.

Current advances in machine translation increase the need for translators to switch from traditional translation to post-editing (PE) of machine-translated text, a process that saves time and improves quality. This affects the design of translation interfaces, as the task changes from mainly generating text to correcting errors within otherwise helpful translation proposals. Our results of an elicitation study with professional translators indicate that a combination of pen, touch, and speech could well support common PE tasks, and received high subjective ratings by our participants. Therefore, we argue that future translation environment research should focus more strongly on these modalities in addition to mouse- and keyboard-based approaches. On the other hand, eye tracking and gesture modalities seem less important. An additional interview regarding interface design revealed that most translators would also see value in automatically receiving additional resources when a high cognitive load is detected during PE.

This paper investigates a simple form of customization: giving users the choice to enable or disable gamification. We present a study (N=77) in the context of image tagging, in which a gamification approach was shown to be effective in previous work. In our case, some participants could enable or disable gamification after they had experienced the task with and without it. Other participants had no choice and did the task with or without game elements. The results indicate that those who are not attracted by the elements can be motivated to tag more through this choice. In contrast, those that like the elements are not affected by it. This suggests that systems should provide the option to disable gamification in the absence of more sophisticated tailoring.

Standard controllers for virtual reality (VR) lack sophisticated means to convey a realistic, kinesthetic impression of size, resistance or inertia. We present the concept and implementation of Drag:on, an ungrounded shape-changing VR controller that provides dynamic passive haptic feedback based on drag, i.e. air resistance, and weight shift. Drag:on leverages the airflow occurring at the controller during interaction. By dynamically adjusting its surface area, the controller changes the drag and rotational inertia felt by the user. In a user study, we found that Drag:on can provide distinguishable levels of haptic feedback. Our prototype increases the haptic realism in VR compared to standard controllers and when rotated or swung improves the perception of virtual resistance. By this, Drag:on provides haptic feedback suitable for rendering different virtual mechanical resistances, virtual gas streams, and virtual objects differing in scale, material and fill state.

The idea to enhance the learning experience of e-learning platforms by incorporating measures about the user's cognitive state, e.g. the cognitive load, boredom, or attention, has been proposed several times and shows promising results. However, the works have mostly dealt with conceptual implications and technical possibilities to detect the state, without considering the user acceptance. This paper therefore investigates whether users would actually be willing to provide access to sensor data such as heart or skin measurements for the sake of making e-learning systems cognition-aware. The results of an online survey with 50 participants show that people would provide access to behavioral data like keyboard input without major concerns; however, other sensors are considered more sensitive and would require strong learning experience improvements to make disclosure worthwhile. Participants also appear less concerned about sensors that are integrated into consumer devices than about less widespread ones. Furthermore, we report the general opinions regarding cognition-aware e-learning and discuss ideas on how best to adapt to the cognitive state.

Modern heuristic optimization systems leverage the parallel processing power of Graphics Processing Units (GPUs). Many states are maintained and evaluated in parallel to improve runtime by orders of magnitudes in comparison to purely CPUbased approaches. A well known example is the parallel Monte Carlo tree search, which is often used in combination with more advanced machine-learning methods these days. However, all approaches require different optimization states in memory to update or manipulate variables and observe their behavior over time. Large real-world problems often require a large number of states that are typically limited by the amount of available memory. This is particularly challenging in cases in which older states (that are not currently being evaluated) are still required for backtracking purposes. In this paper, we propose a new general high-level approach to track and reconstruct states in the scope of heuristic optimization systems on GPUs. Our method has a considerably lower memory consumption compared to traditional approaches and scales well with the complexity of the optimization problem.

Virtual reality (VR) interaction techniques like haptic retargeting offset the user's rendered virtual hand from the real hand location to redirect the user's physical hand movement. This paper explores the order of magnitude of hand redirection that can be applied without the user noticing it. By deriving lower-bound estimates of detection thresholds, we quantify the range of unnoticeable redirection for the three basic redirection dimensions, horizontal, vertical and gain-based hand warping. In a two-alternative forced choice (2AFC) experiment, we individually explore these three hand warping dimensions each in three different scenarios: a very conservative scenario without any distraction and two conservative but more realistic scenarios that distract users from the redirection. Additionally, we combine the results of all scenarios to derive robust recommendations for each redirection technique. Our results indicate that within a certain range, desktop-scale VR hand redirection can go unnoticed by the user, but that this range is narrow. The findings show that the virtual hand can be unnoticeably displaced horizontally or vertically by up to 4.5° in either direction, respectively. This allows for a range of ca. 9°, in which users cannot reliably detect applied redirection. For our gain-based hand redirection technique, we found that gain factors between g = 0.88 and g = 1.07 can go unnoticed, which corresponds to a user grasping up to 13.75% further or up to 6.18% less far than in virtual space. Our findings are of value for the development of VR applications that aim to redirect users in an undetectable manner, such as for haptic retargeting.

Gamification, the use of game elements in non-game contexts, has been successfully used to motivate people to reach their goals more efficiently or turn unpleasant tasks into fun ones. However, most gamified systems are conceptualized for a younger audience and do not account for age-specific changes in the motivation to play or the perception of game elements. To inform the design of gamified applications targeting elderly people (aged 75+), we investigated their gaming experiences, what affects them positively while playing and their attitudes towards the most commonly used game elements. We report findings from semi-structured interviews and a storyboard-based game element assessment (N=18, mean age=84.61), indicating that the main motivation to play is socializing, that participants avoid competition and prefer collaboration and care-taking as well as that badges and points are considered meaningless and provide a level of visibility that puts participants under pressure.

While shoppers increasingly value sustainable products, considering sustainability can be difficult and time-consuming while shopping. In an expert workshop with 22 stakeholders, we gathered requirements for an assistance system supporting customers in identifying the sustainability of products at the point of sale. We integrated the resulting demands in a first mockup prototype, which was tested and discussed with a focus group. From the workshop and the focus group discussion, we deduced a set of ten guidelines for sustainability-oriented assistance systems. These guidelines were transferred into a prototypical mobile application, which allows customers to specify their personal understanding of multiple dimensions of sustainability. According to this profile, they receive easily understandable ratings for scanned products while shopping. A user study in a real supermarket strengthens the deduced guidelines and indicates that such a system can support customers to make more sustainable product choices.

In contrast to conventional retail stores, online shopping comes with many advantages, like unrestricted opening hours and is more focused on functionality. However, these pros often come at a cost of complex search and limited product visualization. Virtual Reality (VR) has the potential to create novel shopping experiences that combine the advantages of e-commerce sites and conventional stores. In this work, we propose a VR shop concept where product placement is not organized in shelves but through spatial placement in appropriate locations in an apartment environment. We thus investigated how the spatial arrangement of products in a non-retail environment affects the user, and how the actual shopping task can be supported in VR. In order to answer these questions, we designed two product selection and manipulation techniques (grabbing and pointing) and two VR shopping cart concepts (a realistic basket and an abstract one) and evaluated them in a user study. The results indicate that product interaction using pointing in combination with the abstract cart concept performs best with regard to error rate, user experience and workload. Overall, the proposed apartment metaphor provides excellent customer satisfaction, as well as a particularly high level of immersion and user experience, and it opens up new possibilities for VR shopping experiences that go far beyond mimicking real shop environments in VR.

In recent years, Virtual Reality (VR) and 3D User Interfaces (3DUI) have seen a drastic increase in popularity, especially in terms of consumer-ready hardware and software. While the technology for input as well as output devices is market ready, only a few solutions for text input exist, and empirical knowledge about performance and user preferences is lacking. In this paper, we study text entry in VR by selecting characters on a virtual keyboard. We discuss the design space for assessing selection-based text entry in VR. Then, we implement six methods that span different parts of the design space and evaluate their performance and user preferences. Our results show that pointing using tracked hand-held controllers outperforms all other methods. Other methods such as head pointing can be viable alternatives depending on available resources. We summarize our findings by formulating guidelines for choosing optimal virtual keyboard text entry methods in VR.

In this paper we present Slackliner, an interactive slackline training assistant which features a life-size projection, skeleton tracking and real-time feedback. As in other sports, proper training leads to a faster buildup of skill and lessens the risk for injuries. We chose a set of exercises from slackline literature and implemented an interactive trainer which guides the user through the exercises and gives feedback if the exercise was executed correctly.

Live-streaming of video games is a recent phenomenon. One driving factor is the direct communication between the streamer and the audience. Currently, besides the platform-integrated options such as text chats, streamers often use external sources to let their community better articulate their opinions. In this paper we present a case study with our tool Helpstone, a live-streaming tool for the card game Hearthstone. Helpstone provides several new communication channels that allow for a better viewer-streamer interaction. We evaluated the tool within a live-streaming session with 23 viewers using Helpstone, and interviewed the streamer. The results indicate that not every implemented interactivity option is relevant. However, in general, new communication channels appear to be valuable and novel influence options are appreciated.

The financial crisis and austerity politics in Europe has had a devastating impact on public services, social security and vulnerable populations. Greek civil society responded quickly by establishing solidarity structures aimed at helping vulnerable citizens to meet their basic needs and empower them to co-create an anti-austerity movement. While digital technology and social media played an important role in the initiation of the movement, it has a negligible role in the movement's on-going practices. Through embedded work with several solidarity structures in Greece, we have begun to understand the "solidarity economy" (SE) as an experiment in direct democracy and self-organization. Working with a range of solidarity structures we are developing a vision for a "Solidarity HCI" committed to designing to support personal, social and institutional transformation through processes of agonistic pluralism and contestation, where the aims and objectives of the SE are continuously re-formulated and put into practice.

The small display size of smartwatches creates a challenge for touch input, which is still the interaction technique of choice. Researchers and producers have started to investigate alternative interaction techniques. Apple and Samsung, for example, introduced digital versions of classic watch components such as the digital crown and the rotatable bezel. However, it remains an open question how well these components behave in terms of user interaction. Based on a self-built smartwatch prototype, we compare current interaction paradigms (touch input, rotatable bezel and digital crown) for one-dimensional tasks, i.e. scrolling in a list, two-dimensional tasks, i.e. navigation on a digital map, and a complex navigation/zoom task. To check for ecological validity of our results, we conducted an additional study focusing on interaction with currently available off-the-shelf devices using our considered interaction paradigms. Following our results, we present guidelines on which interaction techniques to use for the respective tasks.

In this paper, we present a novel real-time hand gesture recognition system based on surface electromyography. We employ a user-independent approach based on a support vector machine utilizing ten features extracted from the raw electromyographic data obtained from the Myo armband by Thalmic Labs. Through an improved synchronization approach, we simplified the application process of the sensing armband. We report the results of a user study with 14 participants using an extended set consisting of 40 gestures. Considering the set of five hand gestures currently supported off-the-shelf by the Myo armband, we outperform their approach with an overall accuracy of 95% compared to 68% with the original algorithm on the same dataset.

Today, ambitioned amateur athletes often do not have access to professional coaching but still invest great effort in becoming faster runners. Apart from a pure increase in the quantitative training load, a change of the running technique, e.g. transitioning from heel striking to fore- or midfoot running, can be highly effective and usually prevents knee-related injuries.

Gaze is a powerful measure of people's attracted attention and reveals where we are looking within our current FOV. Hence gaze-based interfaces are gaining in importance. However, gaze estimation usually requires extensive hardware and depends on a calibration that has to be renewed regularly. We present EyeMirror, a mobile device for calibration-free gaze approximation on surfaces (e.g., displays). It consists of a head-mounted camera, connected to a wearable mini-computer, capturing the environment reflected on the human cornea. The corneal images are analyzed using natural feature tracking for gaze estimation on surfaces. In two lab studies we compared variations of EyeMirror against established methods for gaze estimation in a display scenario, and investigated the effect of display content (i.e. number of features). EyeMirror achieved 4.03° gaze estimation error, while we found no significant effect of display content.

Today, ambitious amateur athletes often do not have access to professional coaching but still invest great effort in becoming faster runners. Apart from a pure increase in the quantitative training load, a change of the running technique, e.g. transitioning from heel striking to mid-/forefoot running, can be highly effective and usually prevents knee-related injuries.

Rock climbing involves complex movements and therefore requires guidance when acquiring a new technique. The classic approach is mimicking the movements of a more experienced climber. However, the trainee has to remember every nuance of the climb, since the sequence of movements cannot be performed in parallel to the experienced climber. As a solution to this problem, we present a video recording and replay system for climbing. The replay component allows for different in-situ video feedback methods. We investigated the video feedback component of the system by studying two example visualization techniques, i.e. a life-sized in-place projection and a real-time third-person view of the climber, augmented by a video showing a successful ascent. The latter is presented to the user on both Google Glass and a projected display. The results indicate that a life-sized projection was perceived as easiest to follow, while most of the climbers had problems with the context switches between the augmented video and the climbing wall. These findings can aid in the design of assistance systems that teach complex movements.

We define the concept of Dynamic Passive Haptic Feedback (DPHF) for virtual reality by introducing the weight-shifting physical DPHF proxy object Shifty. This concept combines actuators known from active haptics and physical proxies known from passive haptics to construct proxies that automatically adapt their passive haptic feedback. We describe the concept behind our ungrounded weight-shifting DPHF proxy Shifty and the implementation of our prototype. We then investigate how Shifty can, by automatically changing its internal weight distribution, enhance the user's perception of virtual objects interacted with in two experiments. In a first experiment, we show that Shifty can enhance the perception of virtual objects changing in shape, especially in length and thickness. Here, Shifty was shown to increase the user's fun and perceived realism significantly, compared to an equivalent passive haptic proxy. In a second experiment, Shifty is used to pick up virtual objects of different virtual weights. The results show that Shifty enhances the perception of weight and thus the perceived realism by adapting its kinesthetic feedback to the picked-up virtual object. In the same experiment, we additionally show that specific combinations of haptic, visual and auditory feedback during the pick-up interaction help to compensate for visual-haptic mismatch perceived during the shifting process.

Wearable sports devices like GPS watches and heart rate monitors are ubiquitous in sports like running or road cycling and enable the users to receive real-time performance feedback. Although rock climbing is a trending sport, there are little to no consumer electronics available to support rock climbing training during exercise. In this paper, we investigated the acceptance and appropriateness of wearables in climbing on different body parts. Based on an online survey with 54 climbers, we designed a wearable device and conducted a perception study with 12 participants in a climbing gym. Using vibro-tactile, audible, and visual cues while climbing an easy route and a hard route, requiring high physical and cognitive load, we found that the most suited notification channel is sound, directly followed by vibro-tactile output. Light has been found to be inappropriate for the use in the sport of climbing.

In this paper we investigate "bottom-up" gamification, i.e. providing users with the option to gamify an experience on their own. To this end, we review commonly used gamification elements in terms of their suitability for such an approach and present the results of an online questionnaire (N=75) complemented by semi-structured interviews with employees of a manufacturing company (N=8). In a twelve-day-long study (N=20) we investigated the usefulness of a task managing app implementing a "bottom-up" gamification concept. With these studies, we derived requirements "bottom-up" applications should fulfill. The study results reveal that people want to use such an approach and are open to the creation of their own gamified experience, thus suggesting that "bottom-up" can be an alternative to "top-down" gamification often used today.

Practical wearable gesture tracking requires that sensors align with existing ergonomic device forms. We show that combining EMG and pressure data sensed only at the wrist can support accurate classification of hand gestures. A pilot study with unintended EMG electrode pressure variability led to exploration of the approach in greater depth. The EMPress technique senses both finger movements and rotations around the wrist and forearm, covering a wide range of gestures, with an overall 10-fold cross validation classification accuracy of 96%. We show that EMG is especially suited to sensing finger movements, that pressure is suited to sensing wrist and forearm rotations, and their combination is significantly more accurate for a range of gestures than either technique alone. The technique is well suited to existing wearable device forms such as smart watches that are already mounted on the wrist.

Mobile gaze-based interaction has been emerging over the last two decades. Head-mounted eye trackers as well as remote systems are used to determine people's gaze (e.g., on a display). However, most state-of-the-art systems need calibration prior to usage. When using a head-mounted eye tracker, many factors (e.g., changes of eye physiology) can influence the stability of the calibration leading to less accuracy over time. Re-calibrating the system at certain time intervals is cumbersome and time-consuming. We investigate methods to minimize the time needed and optimize the process. In a user study with 16 participants, we compared partial re-calibrations with different numbers of calibration points and types of adaptation strategies. In contrast to a full calibration with nine points, the results show that a re-calibration with only three points results in 60% less time needed and achieves a similar accuracy.

We investigate a crowd-based approach to enhance the outcome of optical character recognition in the domain of receipt capturing to keep track of expenses. In contrast to existing work, our approach is capable of extracting single products and provides categorizations for both articles and expenses, through the use of microtasks which are delegated to an unpaid crowd. To evaluate our approach, we developed a smartphone application based on a receipt analysis and an online questionnaire in which users are able to track expenses by taking photos of receipts, and solve microtasks to enhance the recognition. To provide additional motivation to solve these tasks, we make use of gamification. In a three-week-long user study (N=12), we found that our system is appreciated, that our approach reduces the error rate of captured receipts significantly, and that the gamification provided additional motivation to contribute more and thereby enrich the database.

Interacting with ridesharing systems is a complex spatiotemporal task. Traditional approaches rely on the full disclosure of a client's trip information to perform ride matching. However during poor service conditions of low supply or high demand, this requirement may mean that a client cannot find any ride matching their intentions. To address this within real-world road networks, we extend our map-based opportunistic client user interface concept, i.e., launch pads, from a discrete to a continuous space-time representation of vehicle accessibility to provide a client with a more realistic choice set. To examine this extension under different conditions, we conduct two computational experiments. First, we extend our previous investigation into the effects of varying vehicle flexibility and population size on launch pads and a client's probability of pick-up, describing the increased opportunity. Second, observing launch pads within a real-world road network, we analyze aspects of choice and propose necessary architecture improvements. The communication of ride share potential using launch pads provides a client with a simple yet flexible means of interfacing with on-demand transportation.

Today, sports and activity trackers are ubiquitous. Especially runners and cyclists have a variety of possibilities to record and analyze their workouts. In contrast, climbing did not find much attention in consumer electronics and human-computer interaction. If quantified data similar to cycling or running data were available for climbing, several applications would be possible, ranging from simple training diaries to virtual coaches or usage analytics for gym operators. This paper introduces a system that automatically recognizes climbed routes using wrist-worn inertia measurement units (IMUs). This is achieved by extracting features of a recorded ascent and use them as training data for the recognition system. To verify the recognition system, cross-validation methods were applied to a set of ascent recordings that were assessed during a user study with eight climbers in a local climbing gym. The evaluation resulted in a high recognition rate, thus proving that our approach is possible and operational.

Sorting garbage is a relevant topic in many countries as it contributes to environmental protection. Empirical evidence suggests that not all people separate waste, potentially because they do not know how to do it correctly or are simply not motivated enough. We present the results of an online study (N=184) investigating people's capabilities for classifying waste, their capabilities to improve in this task over time and their current garbage separation behavior. The study confirms that the Wisdom of Crowds is applicable in this context as the crowd produces only half as many errors as the individual and feedback helps participants to improve. Based on this, we introduce the idea of a crowd classifying waste in a game, with their classification result then being used as feedback on gamified public trash cans to educate both the crowd playing the game and people using the trash can playfully.

Nowadays, fans post status updates on social media that describe and express their opinion about live sports events. By doing this, they not only comment on the game but also provide information about the current state of play. In this paper we present TwitSoccer, a real-time live soccer score ticker crowd-sourced from tweets. TwitSoccer combines a pattern-based approach, to process the tweets, with a soccer knowledge-database. This allows to detect reoccurring events, such as goals or red cards, more precisely and ensures that these events are assigned to the right teams. The results of our evaluation for 42 soccer matches of the German Bundesliga, show that TwitSoccer automatically detects soccer events reliably with a recall value of 94% for goals and 71% for red cards which resembles a major enhancement over existing work.

Often, we operate mobile devices using only one hand. The hand thereby serves two purposes: holding the device and operating the touch screen with the thumb. The current trend of increasing screen sizes however, makes it close to impossible to reach all parts of the screen (especially the top area) for users with average hand sizes. One solution is to offer adaptive user interfaces for such one-handed interactions. These modes have to be triggered manually and thus induce a critical overhead. They are further designed to bring all content closer, regardless of whether the phone is operated with the left or right hand. In this paper, we present an algorithm that allows determining the users' interacting hand from their unlocking behavior. Our algorithm correctly distinguishes one- and two-handed usage as well as left- and right handed unlocking in 98.51% of all cases. This is achieved through a k-nearest neighbor comparison of the internal sensor readings of the smartphone during the unlocking process.

Our clothes are objects we interact with constantly. They represent our attitude and the circumstances we live in. In the following we will present an example of how this information can be used. Scarfy is a system based on a scarf which augments the natural interaction between a person and her clothes. Scarfy is able to detect the way it is tied and it can deliver information by shape-change and vibration. In this paper we explore the technical and fabrication approaches of technologically extended clothes and their input and output generation and discuss future applications.

Mobile gaze-based interaction with multiple displays may occur from arbitrary positions and orientations. However, maintaining high gaze estimation accuracy in such situa-tions remains a significant challenge. In this paper, we present GazeProjector, a system that combines (1) natural feature tracking on displays to determine the mobile eye tracker's position relative to a display with (2) accurate point-of-gaze estimation. GazeProjector allows for seam-less gaze estimation and interaction on multiple displays of arbitrary sizes independently of the user's position and orientation to the display. In a user study with 12 partici-pants we compare GazeProjector to established methods (here: visual on-screen markers and a state-of-the-art video-based motion capture system). We show that our approach is robust to varying head poses, orientations, and distances to the display, while still providing high gaze estimation accuracy across multiple displays without re-calibration for each variation. Our system represents an important step towards the vision of pervasive gaze-based interfaces.

Mobile phones have evolved significantly in recent years from single-purpose communication devices to multi-purpose computing devices. Despite this evolution, the interaction model for how incoming calls are handled has barely changed. Current-generation smartphones still use abrupt full-screen notifications to alert users to incoming calls, demanding a decision to either accept or decline the call. These full-screen notifications forcibly interrupt whatever activity the user was already engaged in. This might be undesirable when the user's primary task was more important than the incoming call. This paper explores the design space for how smartphones can alert users to incoming calls. We consider designs that allow users to postpone calls and also to multiplex by way of a smaller partial-screen notification. These design alternatives were evaluated in both a small-scale controlled lab study as well as a large-scale naturalistic in-the-wild study. Results show that a multiplex design solution works best because it allows people to continue working on their primary task while being made aware that there is a caller on the line. The contribution of this work is an enhanced interaction design for handling phone calls, and an understanding of how people use it for handling incoming calls.

To maintain a mobile form factor, the screen real estate of a mobile device canIn this paper we present SurfacePhone; a novel configuration of a projector phone which aligns the projector to project onto a physical surface to allow tabletop-like interaction in a mobile setup. The projection is created behind the upright standing phone and is touch and gesture-enabled. Multiple projections can be merged to create shared spaces for multi-user collaboration. We investigate this new setup, starting with the concept that we evaluated with a concept prototype. Furthermore we present our technical prototype, a mobile phone case with integrated projector that allows for the aforementioned interaction. We discuss its technical requirements and evaluate the accuracy of interaction in a second user study. We conclude with lessons learned and design guidelines.

With the increasing availability of smartwatches the question of suited input modalities arises. While direct touch input comes at the cost of the fat-finger problem, we propose to use a dynamic peephole to explore larger content such as websites or maps. In this paper, we present the results of a user study comparing the performance of static and dynamic peephole interactions for a map navigation task on a smartwatch display. As a first method, we investigated the static peephole methaphor where the displayed map is moved on the device via direct touch interaction. In contrast, for the second method - the dynamic peephole - the device is moved and the map is static with respect to an external frame of reference. We compared both methods in terms of task performance and perceived user experience. The results show that the dynamic peephole interaction performs significantly more slowly in terms of task completion time.

With the recent increase in sales figures of tablet computers, a corresponding boost of mobile media consumption on such devices can be observed. Even though tablets provide a comparably larger screen in contrast to smartphones - which are taking the main share of mobile media consumption - the experience of consuming media content on tablets is still limited. For movie theatres the notion of 4 dimensional (4D) films exist, describing a 3D movie that is enhanced with ambient feedback such as environmental light, air streams or moist. In this paper we present ambiPad, a tablet prototype that enriches mobile digital media with ambient feedback around the tablet's display. Besides a light emitting frame, ambiPad allows for thermal stimuli as well. We report on the results of a qualitative user study, in which the feedback channels of ambiPad were rated as highly attractive and desirable by the participants.

The number of available mobile applications is steadily increasing. People have rapidly adopted application stores as means to customize their devices with various functionalities that go beyond communication. Understanding the principles of mobile application usage is crucial for supporting users within this new ecosystem. In this paper, we investigate how people organize applications they have installed on their devices. We asked more than 130 participants for their habits for icon arrangement and collected more than 1,400 screenshots of their devices' menus to further ground our findings. Based on this data we can distinguish five different concepts for arranging icons on smartphone menus, e.g. based on application usage frequency and applications' functional relatedness. Additionally, we investigated how these concepts emerge in relation to frequency of application installations, removals and icon rearrangements, as well as users' experience levels. Finally we discuss implications for the design of smartphone launchers, and highlight differences to icon arrangement on stationary computers.

Existing ride sharing systems for commuting in urban environments are rigid. They rely on the communication of discrete, spatio-temporal constraints from both vehicle and client to perform ride-matching. From a client user perspective these approaches are problematic, leading to location-privacy issues and the use of additional communication channels for ad-hoc negotiation which cannot be immediately quantified. To account for these aspects, we develop a dynamic, intuitive interface technique called launch pads and a centralized system architecture, which together simplify the ride-matching process whilst preserving location-privacy. The results of two experiments reveal the latent potential existing within ride sharing systems if vehicle flexibility is quantified and incorporated into a representation of accessibility. The communication via launch pads provides a client with means to fully exploit this potential.

Digital technologies are rapidly finding their way into urban spaces. One prominent example is media façades. Due to their size, visibility and their technical capabilities, they offer great potential for interaction and for becoming the future displays of public spaces. To explore their potential, researchers have recently started to develop interactive applications for various media façades. Existing development tools are mostly tailored to one specific media façade in one specific setting. They usually provide limited means to incorporate interaction by a user, and the applications developed are limited to running on only one particular media façade. In this paper, we present a flexible, generalized media façade toolkit, which is capable of mimicking arbitrary media façade installations. The toolkit is capable of running interactive applications on media façades with different form factors, sizes and technical capabilities. Furthermore, it ensures application portability between different media façades and offers the possibility of providing interactivity by enabling user input with different modalities and different interaction devices.

Mobile phones have evolved from communication to multi-purpose devices that assist people with applications in various contexts and tasks. The size of the mobile ecosystem is steadily growing and new applications become available every day. This increasing number of applications makes it difficult for end-users to find good applications. Recommender systems suggesting mobile applications are being built to help people to find valuable applications. Since the nature of mobile applications differs from classical items to be recommended (e.g. books, movies, other goods), not only can new approaches for recommendation be developed, but also new paradigms for evaluating performance of recommender systems are advisable. During the lifecycle of mobile applications, different events can be observed that provide insights into users' engagement with particular applications. This gives rise to new approaches for evaluation of recommender systems. In this paper, we present AppFunnel: a framework that allows for usage-centric evaluation considering different stages of application engagement. We present a case study and discuss capabilities for evaluating recommender engines by applying metrics to the AppFunnel.

While mobile phones made a significant evolution in recent years from single-purpose communication devices to multi-purpose devices, the fundamental design of phone call applications did not evolve accordingly. While its implementation leveraged from new hardware and software capabilities, the fundamental decisions people are able to make when they receive a call did not change. Currently, when a call comes in, a modal dialog opens where the callee can either decline or accept the call. A recent study found that the current user interfaces of phone call applications (phone UIs) often lead to an increased overhead when application usage is being interrupted by phone calls [6]. In this paper, we revisit phone call UIs for multipurpose smartphones. We contribute a new design space for mobile phone call UIs, going beyond the simple accept-or-decline dilemma. We present a prototype implementation and discuss open challenges.

Some recent mobile devices have autostereoscopic displays that enable users to perceive stereoscopic 3D without lenses or filters. This might be used to improve depth discrimination of objects overlaid to a camera viewfinder in augmented reality (AR). However, it is not known if autostereoscopy is useful in the viewing conditions typical to mobile AR. This paper investigates the use of autostereoscopic displays in an psychophysical experiment with twelve participants using a state-of-the-art commercial device. The main finding is that stereoscopy has a negligible if any effect on a small screen, even in favorable viewing conditions. Instead, the traditional depth cues, in particular object size, drive depth discrimination.

With the increasing distribution of multi-touch capable devices multi-touch interaction becomes more and more ubiquitous. Especially the interaction with complex data (e.g. medical or geographical data), which until today mostly rely on mice and keyboard input or intense instrumentation, can benefit from this development. Multi-touch interaction offers new ways to deal with 3D data allowing a high degree of freedom (DOF) without instrumenting the user. This paper evaluates indirect multi-touch 3D selection techniques that can be used to interact with stereoscopic data. In this paper two gestural multi-touch selection techniques are presented and investigated with respect to positions on a stereoscopic multi-touch display and special consideration of objects displayed with different parallaxes. In an experiment it was shown that position and parallax have a significant impact on the interaction.

A key issue with state-of-the-art mobile eye trackers, particularly during long-term recordings in daily life, is the need for cumbersome and time consuming (re)calibration. To reduce this burden, in this paper we investigate the feasibility of adapting the calibration obtained for one user to another. Calibration adaptation is automatically performed using a light-weight linear translation. We compare three different methods to compute the translation: "multi-point", where all calibration-points are used, "1-point", and "0-point" that uses only an external parameter. We evaluate these methods in a 6-participant user study in a controlled laboratory setting by measuring the error in visual angle between the predicted gaze point and the true gaze point. Our results show that, averaged across all participants, the best adapted calibration is only 0.8° (mean) off the calibration obtained for that specific user. We also show the potential of the 1-point and 0-point methods compared to the time-consuming multi-point computation.

The increasing number of media façades is a prominent example of the digital augmentation of urban spaces. Many media façades cover most of the outer shell of a building and come with a 3D form factor. They offer great potential for remote interaction in which the interactive area goes beyond the parts of the façade that are visible from the user's current perspective. Common interaction techniques often focus on a fixed part of the media façade. This restricts exploiting the full capabilities and the potential of such gigantic screens. In this paper we describe how to apply cartographic map projections to create 2D map representations of media façades to address this problem. We describe how a continuous interaction with the media façade is possible, independent of the form factor. We analyze existing media façades and provide a set of guidelines for how to create façade maps for different form factors.

Media façades in urban spaces offer great potential for new forms of collaborative multi-user interaction. Beyond interaction, they also allow for connecting people and for triggering social interaction between the different users. We report on our experiences of how simultaneous interaction with a media façade at-a-distance can engage social offline interaction between users. We built an application that allows for simultaneous painting on a façade, and gathered informal feedback during the ARS Electronica Festival in Linz, Austria.

During sales conversations, gestures and mimics are of high importance to communicate information about a product. One prominent example for such sales gestures is the meat and cheese counter, which is one of the remaining spots in supermarkets where sales persons interact with customers. Interactions at such counters in supermarkets normally follow a simple protocol. The customer points at an item of choice. The employee takes out the item and, in most of the cases the product needs to be cut to fit the amount the customer wants to buy. Often it is ambiguous about what specific product the customer and the employees are talking about. Up to now, there are just a few efforts in HCI research to enrich communication at the point of sale. In this paper we report and analyze one scenario in which an intelligent natural user interface can support communication between customer and employee in a sales conversation. Furthermore, we report on our prototype that is able to track pointing gestures by using a depth camera and to display information about items pointed at.

We introduce spatially-aware virtual workspaces for the mobile environment. The notion of virtual workspaces was initially conceived to alleviate mental workload in desktop environments with limited display real-estate. Using spatial properties of mobile devices, we translate this approach and illustrate that mobile virtual workspaces greatly improve task performance for mobile devices. In a first study, we compare our spatially-aware prototype (mSpaces) to existing context switching methods for navigating amongst multiple tasks in the mobile environment. We show that users are faster, make more accurate decisions and require less mental and physical effort when using spatially-aware prototypes. We furthermore prototype pSpaces and m+pSpaces, two spatially-aware systems equipped with pico-projectors as auxiliary displays to provide dual-display capability to the handheld device. A final study reveals advantages of each of the different configurations and functionalities when comparing all three prototypes. Drawing on these findings, we identify design considerations to create, manipulate and manage spatially-aware virtual workspaces in the mobile environment.

Smartphone users might be interrupted while interacting with an application, either by intended or unintended circumstances. In this paper, we report on a large-scale observational study that investigated mobile application interruptions in two scenarios: (1) intended back and forth switching between applications and (2) unintended interruptions caused by incoming phone calls. Our findings reveal that these interruptions rarely happen (at most 10% of the daily application usage), but when they do, they may introduce a significant overhead (can delay completion of a task by up to 4 times). We conclude with a discussion of the results, their limitations, and a series of implications for the design of mobile phones.

Multi-touch interaction offers opportunities to interact with complex data. Especially the exploration of geographical data, which until today mostly relies on mice and keyboard input, could benefit from this interaction paradigm. However, the gestures that are required to interact with complex systems like Geographic Information Systems (GIS) increase in difficulty with every additional functionality. This paper describes a novel interaction approach that allows non-expert users to easily explore geographic data using a combination of multi-touch gestures and handpostures. The use of the additional input modality -- handpose -- is supposed to avoid more complex multi-touch gestures. Furthermore the screen of a wearable device serves as another output modality that on one hand avoids occlusion and on the other hand serves as a magic lens.

In the last years 3D is getting more and more popular. Besides the increasing number of movies for 3D stereoscopic cinemas and television, serious steps have also been undertaken in the field of 3D gaming. Games with stereoscopic 3D output are now available not only for gamers with high-end PCs but also on handheld devices equipped with 3D autostereoscopic displays.

There are a number of mobile shopping aids and recommender systems available, but none can be easily used for a weekly shop at a local supermarket. We present a minimal, mobile and fully functional lambent display that clips onto any shopping trolley handle, intended to nudge people when choosing what to buy. It provides salient information about the food miles for various scanned food items represented by varying lengths of lit LEDs on the handle and a changing emoticon comparing the average miles of all the products in the trolley against a social norm. When evaluated in situ, the lambent handle display nudged people to choose products with fewer food miles than the items they selected using their ordinary shopping strategies. People also felt guilty when the average mileage of the contents of their entire shopping trolley was above the social norm. The findings are discussed in terms of how to provide different kinds of product information that people care about, using simple lambent displays.

The electronic market has rapidly grown in the last few years. However, despite this success, consumers still enjoy visiting a store with real products. Therefore various common technologies have been installed in supermarkets to support the customer's shopping process and experience. In this paper, we introduce the IRL SmartCart - an instrumented shopping cart that acts as a user interface to support the shopping process. We show how RFID technology enables recognizing products that are put in the cart's basket. We are also able to determine the cart's position in an instrumented shopping environment. User input and visual output are possible by means of a touch screen, which is fitted in the IRL SmartCart's handle to support different tasks involved in the shopping process. We present and discuss different location- and context-based services that run on the cart interface system, e.g. a personalized shopping list sorted corresponding to the products in the user's vicinity or a navigation service to different products that the customer is searching for.

Most context-aware systems rely on physical sensors. To some extent, these systems are able to reason about a user's situation by means of the measured data. However, their overall uncertainty in modeling human behavior leads to ambiguity. A language for the mediation of context information between the user and the system is required to enable the user to adjust the machine's interpretation of his context. We describe how keywords that are attributed to activities by users themselves can act as such a mediator. We present results of a study that investigates the nature of this context attributes. The results demonstrate that different users use similar keywords to describe similar situations and different keywords to describe different situations. Therefore, algorithms developed to evaluate the semantic relatedness of tags and resources within folk-sonomies can be applied to exploit knowledge about the users' contexts from the keywords they have assigned. We discuss a prototype that recommends mobile services by enhancing its model with the described keywords.

While applications for mobile devices have become extremely important in the last few years, little public information exists on mobile application usage behavior. We describe a large-scale deployment-based research study that logged detailed application usage information from over 4,100 users of Android-powered mobile devices. We present two types of results from analyzing this data: basic descriptive statistics and contextual descriptive statistics. In the case of the former, we find that the average session with an application lasts less than a minute, even though users spend almost an hour a day using their phones. Our contextual findings include those related to time of day and location. For instance, we show that news applications are most popular in the morning and games are at night, but communication applications dominate through most of the day. We also find that despite the variety of apps available, communication applications are almost always the first used upon a device's waking from sleep. In addition, we discuss the notion of a virtual application sensor, which we used to collect the data.

Projector phones, handheld game consoles and many other mobile devices increasingly include more than one display, and therefore present a new breed of mobile Multi-Display Environments (MDEs) to users. Existing studies illustrate the effects of visual separation between displays in MDEs and suggest interaction techniques that mitigate these effects. Currently, mobile devices with heterogeneous displays such as projector phones are often designed without reference to visual separation issues; therefore it is critical to establish whether concerns and opportunities raised in the existing MDE literature apply to the emerging category of Mobile MDEs (MMDEs). This paper investigates the effects of visual separation in the context of MMDEs and contrasts these with fixed MDE results, and explores design factors for Mobile MDEs. Our study uses a novel eye-tracking methodology for measuring switches in visual context between displays and identifies that MMDEs offer increased design flexibility over traditional MDEs in terms of visual separation. We discuss these results and identify several design implications.

Localization and tracking technologies are rapidly evolving and have a huge impact on our daily life. Most of the available tracking or localization technologies use a coordinate based approach. They describe the current location through a set of coordinates. Although the position is described sufficiently with this set of coordinates, the circumjacent context is not. This work describes an approach to overcome this lack of contextual awareness for an industrial indoor localization. It introduces a service layer which enriches given coordinates based on existing blueprints with context information. The identified entities of the circumjacent context of the coordinates are communicated via a machine readable interface. The interface provides the logical entities geometry and a set of attributes which can be used to obtain further information like electrical or mechanical connections.

This paper presents a scale-independent (space and time) tracking framework for moving entities (objects, animals, humans). The framework allows for the integration, processing and visualization of spatio-temporal data obtained from heterogeneous sensor sources. The framework does not rely upon one tracking and localization technology for the data collection, but instead considers the combination of several technologies to obtain and calculate the best positioning result. Not only are typical localization and tracking technologies considered as data providers, but also keystroke sensors, like touch screens with a fixed location in the tracking environment. Different sizes and characteristics of moving entities are considered as well as different tracking environments or use cases in different domains. A prototypical implementation of basic functionalities processes the spatio-temporal data in a core and provides it by different internal and external interfaces. The modular framework consists of different usage modules. A visualization module is implemented as a first module for the framework. Thus the display of an existing three-dimensional environmental model with the sensors and the visualization of the tracked entities are possible. Furthermore, sensors can be added, changed and managed in the three-dimensional scene. The development of further modules for data analysis for different use cases allows for the analysis of large amounts of data, for example with machine learning techniques. Two different use cases (laboratory mice and service technicians) were used to develop and apply the framework.

We present iPiccer, a system that allows mobile camera phones users to interact with meta-tagged image material available from web repositories, such as flickr in two novel ways. Users of the system are able to infer photo tags from their location and orientation, but are also able to infer their location and orientation from their spatial photo tags. An implementation of iPiccerTaker is outlined, and the potential of this new form of interaction with web repository data is discussed.

We conducted a user study to investigate the effect of visual context in handheld augmented reality interfaces. A dynamic peephole interface (without visual context beyond the device display) was compared to a magic lens interface (with video see-through augmentation of external visual context). The task was to explore objects on a map and look for a specific attribute shown on the display. We tested different sizes of visual context as well as different numbers of items per area, i.e. different item densities. We found that visual context is most effective for sparse item distributions and the performance benefit decreases with increasing density. User performance in the magic lens case approaches the performance of the dynamic peephole case the more densely spaced the items are. In all conditions, subjective feedback indicates that participants generally prefer visual context over the lack thereof. The insights gained from this study are relevant for designers of mobile AR and dynamic peephole interfaces by suggesting when external visual context is most beneficial.

In many mid- to large-sized cities public maps are ubiquitous. One can also find a great number of maps in parks or near hiking trails. Public maps help to facilitate orientation and provide special information to not only tourists but also to locals who just want to look up an unfamiliar place while on the go. These maps offer many advantages compared to mobile maps from services like Google Maps Mobile or Nokia Maps. They often show local landmarks and sights that are not shown on standard digital maps. Often these 'You are here' (YAH) maps are adapted to a special use case, e.g. a zoo map or a hiking map of a certain area. Being designed for a fashioned purpose these maps are often aesthetically well designed and their usage is therefore more pleasant. In this paper we present a novel technique and application called PhotoMap that uses images of 'You are here' maps taken with a GPS-enhanced mobile camera phone as background maps for on-the-fly navigation tasks. We discuss different implementations of the main challenge, namely helping the user to properly georeference the taken image with sufficient accuracy to support pedestrian navigation tasks. We present a study that discusses the suitability of various public maps for this task and we evaluate if these georeferenced photos can be used for navigation on GPS-enabled devices.

Classic paper-based maps provide high-resolution, large-scale information and are ubiquitous in larger cities and even outdoors. In this paper we present a combination of their advantages with a mobile camera projector unit to create a new mobile and intuitive buddy finder interface. Current friend or buddy finder systems on mobile phones such as Google Latitude suffer from the small screen estate and display size of mobile devices. To overcome this problem we use a lightweight mobile camera projector unit to augment the paper map with a projected overlay of the buddies' positions. By using digital and geo-referenced representations of public maps no extra preperation for the tracking is needed. The idea is presented by enhancing Google Latitude, which allows users to browse the positions of their friends, to project the positions directly on the paper map without the cumbersome panning and zooming of a digital map on a small display.

This paper presents the general idea of image geo-mashups, which combines concepts from web mashups and augmented reality by adding geo-referenced data to a perspective image. The paper shows how to design and implement an augmented reality weather cam, that combines data from a steerable weather cam with additional sensor information retrieved from the web.

In today's information world, small personal computerized devices, such as PDAs, smart phones and other smart appliances, become widely available and essential tools in many situations. This ongoing penetration of computers into everyday life leads to so-called ubiquitous environments, where computational power and networking capabilities are available (and used) everywhere. The strive of providing personal services to users made user modeling capability an essential part of any ubiquitous application. Ubiquitous user modeling describes ongoing modeling and exploitation of user behavior with a variety of systems that share their user models. Currently, issues relating to ubiquitous user modeling are gaining more and more attention from research groups representing the user modeling, the human-computer interaction, and the ubiquitous computing research communities. The goal of this workshop is to bring together academic and industrial researchers from these communities to discuss the most innovative approaches to ubiquitous user modeling, to enhance the exchange of ideas and concepts, to determine the veins the research should proceed, and to go one step further towards personalization in ubiquitous computing.

Virtual globes have progressed from little-known technology to broadly popular software in a mere few years. We investigated this phenomenon through a survey and discovered that, while virtual globes are en vogue, their use is restricted to a small set of tasks so simple that they do not involve any spatial thinking. Spatial thinking requires that users ask "what is where" and "why"; the most common virtual globe tasks only include the "what". Based on the results of this survey, we have developed a multi-touch virtual globe derived from an adapted virtual globe paradigm designed to widen the potential uses of the technology by helping its users to inquire about both the "what is where" and "why" of spatial distribution. We do not seek to provide users with full GIS (geographic information system) functionality, but rather we aim to facilitate the asking and answering of simple "why" questions about general topics that appeal to a wide virtual globe user base.

This paper addresses challenges connected to the integration of small-size cameras into everyday objects and their impact on human computer interaction (HCI) paradigms. Along the lines of Weiser's vision of ubiquitous computing more and more everyday objects are equipped with sophisticated sensors, including accelerometers, temperature and pressure sensors as well as cameras. Many off-the-shelf mobile phones are equipped with a mega-pixel camera today. In the future it is likely that other objects of daily live use will come with integrated cameras as well, such as cups, books, or doors. In this paper we will briefly review some state-of-the-art HCI-related projects relying on embedded cameras and highlight related challenges in the context of vision algorithms and technologies.

A user study was conducted to compare the performance of three methods for map navigation with mobile devices. These methods are joystick navigation, the dynamic peephole method without visual context, and the magic lens paradigm using external visual context. The joystick method is the familiar scrolling and panning of a virtual map keeping the device itself static. In the dynamic peephole method the device is moved and the map is fixed with respect to an external frame of reference, but no visual information is present outside the device's display. The magic lens method augments an external content with graphical overlays, hence providing visual context outside the device display. Here too motion of the device serves to steer navigation. We compare these methods in a study measuring user performance, motion patterns, and subjective preference via questionnaires. The study demonstrates the advantage of dynamic peephole and magic lens interaction over joystick interaction in terms of search time and degree of exploration of the search space.

Normally, mobile pedestrian navigation systems use visually perceptible landmarks to guide their users through the environment. In this article we introduce concepts for the use of auditory perceptible landmarks in route descriptions. Such auditory perceptible landmarks complement their visual counterparts and also stand to be beneficial for certain groups like the visually impaired and the elderly.

This full-day tutorial will provide an overview on Intelligent User Interfaces in the context of the technological development associated with the Ubiquitous Computing paradigm. Based on a technological assessment of the main technological areas involved (miniaturization, communication and network structures, advanced display and speech technologies, as well as sensor integration and deployment) this tutorial will discuss the main approaches in context and user modeling, as well as present new interaction paradigms for ubiquitous computing environments. Examples of past and current research projects of the field will be used to explain and discuss the intersection of UbiComp and IUIs. Having discussed the technical foundations, the tutorial aims to explore the design and development of new pervasive applications for mobile devices and other distributed interfaces. It will explore how handhelds can interact with other surrounding devices, intelligent environments and with the social context. The tutorial will also look at how Participatory Design approaches and advanced in-situ evaluation techniques and ethno-methodologies such as living labs, on-the-field data collection, enactments, and 'Cultural Probes', can lead to the representation of significant communication systems.

In this paper we report preliminary findings from two user studies that on the one hand investigate how prosodic parameters of synthetic speech can influence the perceived impression of the speakers personality and on the other hand explores if and how people attribute personality to objects such as typical products of daily shopping. The results show that a) prosodic parameters have a strong influence on the perceived personality and can be partially used to achieve a desired impression and b) that subjects clearly attribute personalities to products. Both findings encourage us to continue our work on a dialogue shell for talking products.

The main objective of the third workshop on Multi-User and Ubiquitous User Interfaces (MU3I 2006) is to bring people with relevant backgrounds (e.g. interface design, CSCW, ubiquitous computing) together to discuss two key questions in this field: How can we build interfaces, which span multiple devices so that the user knows that they can be used to control a specific application? How can we build interfaces for public displays? Therefore, the main outcome of the workshop is expected to consists of further insights into those problems, potential solutions and a research agenda to investigate these further.

In this paper, we describe the results of a usability study on user preferences for multimodal interaction in an instrumented environment. The study was conducted in a public setting, and provides insight into modality preferences among users, and specific to among men and women. The returned results are also contrasted to the results of a former study based on the same evaluation procedures but conducted under a laboratory setting.

In this paper we regard the navigation aid provided by mobile navigation systems in a real environment and the effects of these mobile assistants to the development of spatial knowledge. Therefore, we report on a user study concerning the acquisition of spatial knowledge. This study sets up on a former study described by Krüger and colleagues and sheds light on problems concerning the acquisition of survey knowledge while being navigated by a mobile handheld PC.

We introduce the concept of product associated displays -PADs - as a way of providing visual feedback to users interacting with physical objects in an instrumented environment. PADs are projected public displays created at locations that can be intuitively associated with the objects they show information about. The concept is illustrated in a shopping scenario.

This second workshop on Multi-User and Ubiquitous User Interfaces aims at further investigating two major issues identified at last year's MU3I: control and consistency. The former relates to how a user gains control of devices in a ubiquitous computing environment, how control is passed, and how it is shared in such a setting. The second one concerns interfaces that span multiple devices or move from one set of devices to another. Both issues will be discussed in this year's workshop (with a focus on consistency.

In this paper, we describe a public navigation system which uses adaptive displays as directional signs. The displays are mounted to walls where they provide passersbys with directional information. Each sign is an autonomous, wirelessly networked digital displays connected to a central server. The signs are position-aware and able to adapt their display content in accordance with their current position. Advantages of such a navigation system include improved flexibility, dynamic adaptation and ease of setup and maintenance.

Tangible user interfaces provide access to virtual information through intuitive physical manipulation. However, feedback is mostly provided by displays in the environment instead of the TUI itself. In this paper we describe the design of Tuister, a tangible user interface with multiple embedded displays and sensors. We explain how Tuister can be used to browse and access hierarchical structures and briefly describe the current state of a prototype we're building.

Navigation services can be found in different situations and contexts: while connected to the web through a desktop PC, in cars, and more recently on PDAs while on foot. These services are usually well designed for their specific purpose, but fail to work in other situations. In this paper we present an approach that connects a variety of specialized user interfaces to achieve a personal navigation service spanning different situations. We describe the concepts behind the \bf BPN (BMW Personal Navigator), an entirely implemented system that combines a desktop event and route planner, a car navigation system, and a multi-modal, in- and outdoor pedestrian navigation system for a PDA. Rather than designing for one unified UI, we focus on connecting specialized UIs for desktop, in-car and on-foot use.

The issue of the seamless interleaving of interaction with a mobile device and stationary devices is addressed, in a typical situation of educational entertainment: the visit to a museum. Some of the salient elements of the described work are the emphasis on multimodality in the dynamic presentation and coherence throughout the visit.The adopted metaphor is of a kind of contextualized TV-like presentation, useful for engaging (young) visitors. On the mobile device, personal video clips are dynamically generated from personalized verbal presentations; on larger stationary screens distributed throughout the museum, further background material and additional information is provided. A virtual presenter follows the visitors in their experience and gives advice on both types of devices and on the museum itself.

Mobile and outdoor environments have long been out ofreach for speech engines due to the performance limitationsthat were associated with portable devices, and thedifficulties of processing speech in high-noise areas. Thispaper outlines an architecture for attaining robust speechrecognition rates in a mobile pedestrian indoor/outdoornavigation environment, through the use of a media fusionknowledge component.

The design of mobile navigation systems adapting to limited resources will be an important future challenge. Since typically several different means of transportation have to be combined in order to reach a destination, the user interface of such a system has to adapt to the user's changing situation. This applies especially to the alternating use of different technologies to detect the user's position, which should be as seamless as possible. This article presents a hybrid navigation system that relies on different technologies to determine the user's location and that adapts the presentation of route directions to the limited technical resources of the output device and the limited cognitive resources of the user.

We are exploring a class of decision-theoretic handheld systems that give a user personalized advice about how to explore an indoor area in search of products or information. An initial user test in a simple mockup of a shopping mall showed that even novice PDA users accepted the system immediately and were able to achieve their shopping goals faster than when using a paper map of the mall. A key issue is the extent to which spontaneous user behavior can be accommodated within this framework.

We describe a hybrid building navigation system consisting of stationary information booths and a mobile communication infrastructure feeding small portable devices. The graphical presentations for both the booths and the mobile devices are generated from a common source and for the common task of way finding, but they use different techniques to convey possibly different subsets of the relevant information. The form of the presentations is depending on technical limitations of the output media, accuracy of location information, and cognitive restrictions of the user. We analyze what information needs to be conveyed, how limited resources influence the presentation of this information, and argue, that by generating all different presentations in a common framework, a consistent appearance across devices can be achieved and that the different device classes can complement each other in facilitating the navigation task.

While car navigation systems are widely commercialized already today, pedestrian information systems are still in the early research stage. However, recent progress in mobile computing has opened perspectives for pedestrian navigation systems. In this context, graphics is and will still be an important modality to convey all types of route information. This paper will address the question how to generate graphics for navigation systems that help pedestrians, e.g., airport passengers, city tourists or conference attendees, to find their way in complex environments. We will discuss how the presentation of graphics can be tailored to various technical and cognitive constraints, and we will demonstrate our ideas within a scenario where a passenger of an airport gets navigational help from a stationary info booth and afterwards on her way via a handheld device (PDA). Both the 3D visualization at the info booth and the sketch-like presentation on the PDA are generated from the same data and by the same system, yet are adapted to the specific situation, output medium and user as far as possible.

The purpose of this paper is to present models, methods and techniques to control automatically the degree of details in graphics or animation in an intelligent way. Instead of just aiming at the technical advantages of such a reduction (i.e. saving computer memory and computational load), this work focuses on clarifying the intention of graphics or animation with the means of abstraction. The goal is to direct the viewer's attention to relevant parts of the graphics, without using metaobjects or - colours (e.g. arrows, blinking objects etc.) and to reduce the probability of distracting the viewer's attention by unimportant details.The paper describes the major problems of this task and provides a knowledge-based solution to yield an intelligent control of the level of detail, improving the expressivity of the graphics or animation and saving computational resources at the same time. To emphasize the plausibility of these ideas the graphical abstraction system ARP is presented and its results are discussed.