Delays in direct teleoperation decouple operator input from robot feedback. We frame this not as a unitary problem but as three facets of operator uncertainty: (1) communication, when commands take effect, (2) trajectory, how inputs map to motion, and (3) environmental, how external factors alter outcomes. We externalized each facet through predictive visualizations: Network, Path, and Envelope. In a controlled study with 24 participants (novices in telerobotics) navigating a simulated robot under a fixed 2.56 s round-trip delay, we compared these visualizations against a delayed-video baseline. Path significantly shortened task time, lowered perceived cognitive load, and reduced reliance on reactive “move-and-wait” behavior. Envelope lowered cognitive load but did not significantly reduce reactive behavior or improve performance, while Network had no measurable effect. These results indicate that predictive support is effective only when trajectory uncertainty is externalized, enabling operators to move from reactive to more proactive control.

Virtual Reality (VR) environments challenge users with varied input devices, interaction methods, and interface designs, resulting in a steep learning curve. Feedforward "informs the user about what the result of his action will be", and it allows to ease the process of learning of the end user by providing ways to represent the required action to perform, using contextualised previews that show how to complete a given interaction. Yet, creating effective direct feedforward without specialised tools remains a tedious process. We present BeatriXR, a flexible and adaptive toolkit that simplifies the creation of direct-feedforward configurations in VR. It enables users to build, visualise, and customise direct feedforward using virtual avatars, offering both in-world representations and on-screen comparisons of interaction options. Mapped to the recognised design-space by Muresan et al. (including Triggering, Previewing, and Exiting phases), BeatriXR streamlines development and helps ensure effective feedforward integration in VR environments. As additional support for the feedforward design task, BeatriXR features an LLM-powered decision-support layer that suggests possible configuration options. This guidance helps designers select optimal settings during development for adapting the presentation to user needs and the expected configuration of the interaction context. We conducted an exploratory review with XR domain experts who rated the UI for modifying feedforward settings, and assessed four LLM models in the task of suggesting possible configuration options. The participants’ feedback was positive and provided valuable insights for improving both the user interface, which was generally perceived positively, and the quality of the LLM-generated responses.

Machine learning systems are often inspected through 2D projections of high-dimensional representations using techniques such as t-SNE or UMAP. While these visualizations provide useful overviews of clustering and similarity, they are inherently static: they display only the existing data points and do not allow users to interactively explore a model’s decision space. We present an interactive exploration system — LAPEX — that uses a Variational Autoencoder (VAE) as a generative proxy over a model’s training distribution, turning the latent space into a navigable workspace for model sensemaking. Unlike static embeddings, the proxy provides an explicit decoding path from latent coordinates to inputs, enabling interaction patterns such as continuous sampling, interpolation between anchors, and region probing. We operationalize these capabilities through a set of interactive probes that augment a familiar scatter-plot overview with generative overlays for comparing transitions between classes and examining sparsely populated regions. A within-subject formative study ( N = 16) comparing an interactive VAE-based method to a static t-SNE baseline shows that generative interaction substantially improves counterfactual reasoning and influences how users assess model behavior in sparse or uncertain regions, while static embeddings sometimes provide clearer boundary perception. From these findings, we derive concrete design guidelines and architectural considerations for engineering interactive AI model exploration systems using generative latent representations.

Augmented Reality (AR) holds significant potential to facilitate users in executing manual tasks. For effective support, however, we need to understand how showing movement instructions in AR affects how well people can follow those movements in real life. In this paper, we examine the degree to which users can synchronize the speed of their movements with speed cues presented through an AR environment. Specifically, we investigate the effects of timing in AR visual guidance. We assess performance using a highly realistic Mixed Reality (MR) welding simulation. Welding is a task that requires very precise timing and control over hand and arm motion. Our results show that upfront visual guidance (before manual task execution) alone often fails to transfer the knowledge of intended speeds, especially at higher target speeds. Live guidance (during manual task execution) during the activity provides more accurate speed results but typically requires a higher overshoot at the start. Optimal outcomes occur when visual guidance appears upfront and continues during the activity for users to follow through.

Cleanroom cleaning is a surface coverage task where the pattern should be followed correctly, and the entire surface should be covered. We investigate the efficacy of augmented reality (AR) by implementing various pattern guidance designs to enhance a cleanroom cleaning task. We developed an AR guidance system for cleaning procedures and evaluated four distinct pattern guidance methods: (1) breadcrumbs, (2) examples, (3) middle lines, and (4) outlines. We vary the instructions on the entire surface or as a single step. To measure performance, accuracy, and user satisfaction associated with each guidance method, we conducted a large-scale (n=864) between-subjects study. Our findings indicate that single step instructions proved to be more intuitive and efficient than full instructions, especially for the breadcrumbs. We also discussed the implications of our results for the development of AR applications for surface coverage and pattern optimization.

Text input in Virtual Reality (VR) is crucial for communication, search, and productivity. We compared four keyboard designs for VR text entry, leveraging the flexibility and the tracking options of a 3D environment. We used the Dvorak layout to control for experience differences. The designs were: (a) a floating keyboard with touch input, (b) a keyboard attached on the back of the hand with touch input, (c) a floating keyboard with eye tracking and pinch input, and (d) a keyboard laid out over a rolling shape with touch input. Designs (b), (c), and (d) can move in 3D space, while design (a) is static. Design (d) had similar efficiency to design (a) but with better usability and lower Physical Demand. Design (b) led to higher Physical Demand, Effort, and Frustration. Design (c) had lower Physical Demand but higher Mental Demand, Effort, and error rates. Typing speeds averaged 6.51 WPM (1.24% error rate) for (a), 5.56 WPM (3.82% error rate) for (b), 5.33 WPM (1.43% error rate) for (c), and 6.70 WPM (1.64% error rate) for (d).

Navigating the interaction landscape of Virtual Reality (VR) and Augmented Reality (AR) presents significant complexities due to the plethora of available input hardware and interaction modalities, compounded by spatially diverse visual interfaces. Such complexities elevate the likelihood of user errors, necessitating frequent backtracking. To address this, we introduce ViRgilites, a virtual guidance framework that delivers multi-level feedforward information covering the available interaction techniques as well as the future possibilities to interact with virtual objects, anticipating the interaction effects and how they fit with the overall user's goal. ViRgilites is engineered to facilitate task execution, empowering users to make informed decisions about action methodologies and alternative courses of action. This paper presents the architecture and functionality of ViRgilites and demonstrates its efficacy through evaluation with a formative user study.

The majority of errors in making processes can be tracked back to errors in dimensional specifications. While technical aspects of measurement, such as precision and speed have been extensively studied in metrology, the user aspects of measurement received significantly less attention. While little research exists that specifically addresses the user aspects of handling dimensions, various systems have been built that embed new interactive modalities, processes, and techniques which significantly impact how users deal with dimensions or conduct measurements. However, these features are mostly hidden in larger system contributions. To uncover and articulate these techniques, we conducted a holistic literature survey on measurement practices in crafting techniques and systems for rapid prototyping. Based on this survey, we contribute 10 measurement patterns, which describe reusable elements and solutions for common difficulties when dealing with dimensions throughout workflows for making physical artifacts.

In many jobs, workers execute precise line tracing tasks; welding, spray painting, or chiseling, for example. Training and support for such tasks can be done using VR and AR. However, to enable workers to achieve the required precision in movement and timing, the effect of visual guidance on continuous movement needs to be explored. In VR environments, we want to ensure people are trained so that the obtained skill is transferable to a real-world context, whereas, in AR, we want to ensure an ongoing task can be completed successfully when adding visual guidance. To simulate these various contexts, we employ a VR environment to investigate the effectiveness of different visualizations for motion-based guidance in a line tracing task. We tested five different visualizations, including faster and slower arrows on the pen, the same arrows on the line, a dynamic graph on the pen or line, and a ghost object to follow. Each visualization was tested with the same set of five lines of different target speeds (2cm/s to 10 cm/s in steps of 2 cm/s) with a training line of 5 cm/s. Our results show that the example ghost on the line turns out to be the most efficient visualization for allowing users to achieve a specific speed. Users also perceived this visualization as the most engaging and easy to use. These findings have significant implications for the development of AR-based guidance systems, specifically in the realm of speed control, across diverse domains such as industrial applications, training, and entertainment.

Welcome to this issue of the Proceedings of the ACM on Human-Computer Interaction, bringing together contributions from the community on Engineering Interactive Computing Systems (EICS). The EICS track of the PACM-HCI is the primary venue for research contributions at the intersection of Human-Computer Interaction (HCI) and Software Engineering. This year, over the three rounds of submissions, for the issue of PACM-HCI we received 68 valid submissions (out of 90 submissions in total), of which we carefully selected 19 papers, bringing our acceptance rate to 27.9%. The result of this selection process is presented in this issue of the Proceedings of the ACM.

Operators’ well-being is a key factor for the success of industrial production processes. Even though research has studied the well-being aspects of the industry, such as support and improvement of ergonomics, there is still a long way to go to achieve a sustainable and healthy work context for manufacturing industry. We believe the Human-Computer Interaction community can contribute by developing research on worker well-being in real-life settings. This workshop intends to offer a venue for HCI researchers that focus on worker well-being for the manufacturing industry and other industry domains.

Smart home inhabitants can specify trigger-condition-action rules to control the home's behavior. As the number of rules and their complexity grow, however, so does the probability of issues such as inconsistencies and redundancies. These can lead to unintended behavior, including security vulnerabilities and wasted resources, which harms the inhabitants' trust in the system. Existing approaches to handle unintended behavior typically require inhabitants to define all-encompassing, permanent solutions by modifying the rules. Although this is fitting in certain situations, some unforeseen situations might occur. We argue that the user always must have the last word to avoid unwanted behaviors, without altering the overall behavior. With FortClash, we present an approach to predict many different types of unintended behavior, and contribute four novel mechanisms to mediate them that rely on making one-time exceptions. With FortClash, inhabitants gain a new tool to deal with unintended behavior in the short-term that is compatible with existing long-term approaches such as editing rules.

As robots are equipped with software that makes them increasingly autonomous, it becomes harder for humans to understand and control these robots. Human users should be able to understand and, to a certain amount, predict what the robot will do. The software that drives a robotic system is often very complex, hard to understand for human users, and there is only limited support for ensuring robotic systems are also intelligible. Adding intelligibility to the behavior of a robotic system improves the predictability, trust, safety, usability, and acceptance of such autonomous robotic systems. Applying intelligibility to the interface design can be challenging for developers and designers of robotic systems, as they are expert users in robot programming but not necessarily experts on interaction design. We propose Choreobot, an interactive, online, and visual dashboard to use with our reference framework to help identify where and when adding intelligibility to the interface design is required, desired, or optional. The reference framework and accompanying input cards allow developers and designers of robotic systems to specify a usage scenario as a set of actions and, for each action, capture the context data that is indispensable for revealing when feedforward is required. The Choreobot interactive dashboard generates a visualization that presents this data on a timeline for the sequence of actions that make up the usage scenario. A set of heuristics and rules are included that highlight where and when feedforward is desired. Based on these insights, the developers and designers can adjust the interactions to improve the interaction for the human users working with the robotic system.

International audience

The promise of on-body interactions has led to widespread development of wearable displays. They manifest themselves in highly variable shapes and form, and are realized using technologies with fundamentally different properties. Through an extensive survey of the field of wearable displays, we characterize existing systems based on key qualities of displays and wearables, such as location on the body, intended viewers or audience, and the information density of rendered content. We present the results of this analysis in an open, web-based interactive design space that supports exploration and refinement along various parameters. The design space, which currently encapsulates 129 cases of wearable displays, aims to inform researchers and practitioners on existing solutions and designs, and enable the identification of gaps and opportunities for novel research and applications. Further, it seeks to provide them with a thinking tool to deliberate on how the displayed content should be adapted based on key design parameters. Through this work, we aim to facilitate progress in wearable displays, informed by existing solutions, by providing researchers with an interactive platform for discovery and reflection.

Virtual Reality (VR) allows simulation of machine control panels without physical access to the machine, enabling easier and faster initial exploration, testing, and validation of machine panel designs. However, haptic feedback is indispensable if we want to interact with these simulated panels in a realistic manner. We present HapticPanel, an encountered-type haptic system that provides realistic haptic feedback for machine control panels in VR. To ensure a realistic manipulation of input elements, the user’s hand is continuously tracked during interaction with the virtual interface. Based on which virtual element the user intends to manipulate, a motorized panel with stepper motors moves a corresponding physical input element in front of the user’s hand, enabling realistic physical interaction.

Abstract Feedback and feedforward are two fundamental mechanisms that support users’ activities while interacting with computing devices. While feedback can be easily solved by providing information to the users following the triggering of an action, feedforward is much more complex as it must provide information before an action is performed. For interactive applications where making a mistake has more impact than just reduced user comfort, correct feedforward is an essential step toward correctly informed, and thus safe, usage. Our approach, Fortunettes, is a generic mechanism providing a systematic way of designing feedforward addressing both action and presentation problems. Including a feedforward mechanism significantly increases the complexity of the interactive application hardening developers’ tasks to detect and correct defects. We build upon an existing formal notation based on Petri Nets for describing the behavior of interactive applications and present an approach that allows for adding correct and consistent feedforward.

Listening to music while being on the move is common in our headphone society. However, if we want assistance in navigation from our smartphone, existing approaches either demand exclusive playback through the headphones or impact the listening experience of the music. We present a field evaluation of Attracktion, a spatial audio navigation system that leverages the access to single stems in a multi-track recording to minimize the impact on the listening experience. We compared Attracktion against current turn-by-turn navigation instructions in a field-study with 22 users and found that users perceived acoustic overlays with additional navigation information to have no impact on the listening experience. In terms of path efficiency, errors, and mental workload, Attracktion is on par with spoken turn-by-turn navigation instructions, and users liked it for the aspect of serendipity.

In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) on an interface. This article contributes individualised predictive models of visual search, and a computational approach to restructure graphical layouts for an individual user such that features on a new, unvisited interface can be found quicker. It explores four technical principles inspired by the human visual system (HVS) to predict expected positions of features and create individualised layout templates: (I) the interface with highest frequency is chosen as the template; (II) the interface with highest predicted recall probability (serial position curve) is chosen as the template; (III) the most probable locations for features across interfaces are chosen (visual statistical learning) to generate the template; (IV) based on a generative cognitive model, the most likely visual search locations for features are chosen (visual sampling modelling) to generate the template. Given a history of previously seen interfaces, we restructure the spatial layout of a new (unseen) interface with the goal of making its features more easily findable. The four HVS principles are implemented in Familiariser, a web browser that automatically restructures webpage layouts based on the visual history of the user. Evaluation of Familiariser (using visual statistical learning) with users provides first evidence that our approach reduces visual search time by over 10%, and number of eye-gaze fixations by over 20%, during web browsing tasks.

We present JigFab, an integrated end-to-end system that supports casual makers in designing and fabricating constructions with power tools. Starting from a digital version of the construction, JigFab achieves this by generating various types of constraints that configure and physically aid the movement of a power tool. Constraints are generated for every operation and are custom to the work piece. Constraints are laser cut and assembled together with predefined parts to reduce waste. JigFab's constraints are used according to an interactive step-by-step manual. JigFab internalizes all the required domain knowledge for designing and building intricate structures, consisting of various types of finger joints, tenon & mortise joints, grooves, and dowels. Building such structures is normally reserved for artisans or automated with advanced CNC machinery.

Abstract Physical exercise training and medication compliance are primary components of cardiac rehabilitation. When rehabilitating independently at home, patients often fail to comply with their prescribed medication and find it challenging to interpret exercise targets or be aware of the expected efforts. Our work aims to assist cardiac patients in understanding their condition better, promoting medication adherence and motivating them to achieve their exercise targets in a tele-rehabilitation setting. We introduce a patient-centric intelligible visualization approach to present prescribed medication and exercise targets to patients. We assessed efficacy of intelligible visualizations on patients’ comprehension in two lab studies. We evaluated the impact on patient motivation and health outcomes in field studies. Patients were able to adhere to medication prescriptions, manage their physical exercises, monitor their progress and gained better self-awareness on how they achieved their rehabilitation targets. Patients confirmed that the intelligible visualizations motivated them to achieve their targets better. We observed an improvement in overall physical activity levels and health outcomes of patients. Research Highlights Presents challenges currently faced in cardiac tele-rehabilitation. Demonstrates how intelligibility was applied to two core aspects of cardiac rehabilitation- promoting medication adherence and physical exercise training. Lab., field and clinical studies to demonstrate efficacy of intelligible visualization, impact on patient motivation and resultant health outcomes. Reflection on how similar HCI approaches could be leveraged for technology-supported management of critical health conditions such as cardiac diseases.

Feedback is commonly used to explain what happened in an interface. What if questions, on the other hand, remain mostly unanswered. In this paper, we present the concept of enhanced widgets capable of visualizing their future state, which helps users to understand what will happen without committing to an action. We describe two approaches to extend GUI toolkits to support widget-level feedforward, and illustrate the usefulness of widget-level feedforward in a standardized interface to control the weather radar in commercial aircraft. In our evaluation, we found that users require less clicks to achieve tasks and are more confident about their actions when feedforward information was available. These findings suggest that widget-level feedforward is highly suitable in applications the user is unfamiliar with, or when high confidence is desirable.

Translation environments offer various translation aids to support professional translators. However, translation aids typically provide only limited justification for the translation suggestions they propose. In this paper we present Intellingo, a translation environment that explores intelligibility for translation aids, to enable more sensible usage of translation suggestions. We performed a comparative study between an intelligible version and a non-intelligible version of Intellingo. The results show that although adding intelligibility does not necessarily result in significant changes to the user experience, translators can better assess translation suggestions without a negative impact on their performance. Intelligibility is preferred by translators when the additional information it conveys benefits the translation process and when this information is not part of the translator's readily available knowledge.

We present a scalable Do-It-Yourself (DIY) fabrication workflow for prototyping highly stretchable yet robust devices using a CO2 laser cutter, which we call Silicone Devices. Silicone Devices are self-contained and thus embed components for input, output, processing, and power. Our approach scales to arbitrary complex devices as it supports techniques to make multi-layered stretchable circuits and buried VIAs. Additionally, high-frequency signals are supported as our circuits consist of liquid metal and are therefore highly conductive and durable. To enable makers and interaction designers to prototype a wide variety of Silicone Devices, we also contribute a stretchable sensor toolkit, consisting of touch, proximity, sliding, pressure, and strain sensors. We demonstrate the versatility and novel opportunities of our technique by prototyping various samples and exploring their use cases. Strain tests report on the reliability of our circuits and preliminary user feedback reports on the user-experience of our workflow by non-engineers.

We present the highlights of the now finished 4-year SCATE project. It was completed in February 2018 and funded by the Flemish Government IWT-SBO, project No. 130041.1

Keeping track of design processes is a cumbersome task due to the apparently unconstrained and unstructured nature of creative work. Traceability is fundamental to revisit and reflect on the design narratives that describe artefact evolution. In this paper, we aim to identify what characteristics are necessary to facilitate traceability of creative design processes. For this end, we use a functional prototype to connect artefacts, design rationale, and decisions in a shared workspace. We evaluated this prototype for 15 weeks with six pairs of students engaged in a user-centered design project. Our findings show that having a lean repository of artefacts annotated with design rationale can facilitate tracking progress in different phases of the process. We found that creating a record of the participants' design work is useful to reflect on and for team agreement, ensure consistency of evolving artefacts, and help in planning future steps in the design project.

In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) in familiar locations. This paper contributes computational approaches to restructuring layouts such that features on a new, unvisited interface can be found quicker. We explore four concepts of familiarisation, inspired by the human visual system (HVS), to automatically generate a familiar design for each user.

We present StrutModeling, a computationally enhanced construction kit that enables users without a 3D modeling background to prototype 3D models by assembling struts and hub primitives in physical space. Physical 3D models are immediately captured in software and result in readily available models for 3D printing. Given the concrete physical format of StrutModels, modeled objects can be tested and fine tuned in the presence of existing objects and specific needs of users. StrutModeling avoids puzzling with pieces by contributing an adjustable strut and universal hub design. Struts can be adjusted in length and snap to magnetic hubs in any configuration. As such, arbitrarily complex models can be modeled, tested, and adjusted during the design phase. In addition, the embedded sensing capabilities allow struts to be used as measuring devices for lengths and angles, and tune physical mesh models according to existing physical objects.

We present Handly, an integrated upper-limb rehabilitation system for persons with a neurological disorder. Handly consists of tangibles for training four hand tasks with specific functional handgrips and a motivational game. These persons need intensive personalized therapy in order to sustain or enhance their abilities and aptitude in daily activities. Conventional approaches require intense support by a therapist or usage of expensive robots, both of which are not a sustainable model of training. Handly combines tangibles specifically designed for repetitive task-oriented motor skill training of typical daily activities with serious gaming, thus offering a comprehensive approach. A feedback mechanism guides the user and an accessible game environment stimulates individual training. We performed an initial evaluation of Handly with eight persons with Multiple Sclerosis. We show that a tangible-virtual environment for intensive upper-limb rehabilitation is feasible and that this approach is highly appreciated by the patients.

In this paper, we set out to find what encompasses an appropriate visual language for information presented on near-eye out-of-focus displays. These displays are positioned in a user's peripheral view, very near to the user's eyes, for example on the inside of the temples of a pair of glasses. We explored the usable display area, the role of spatial and retinal variables, and the influence of motion and interaction for such a language. Our findings show that a usable visual language can be accomplished by limiting the possible shapes and by making clever use of orientation and meaningful motion. We found that especially motion is very important to improve perception and comprehension of what is being displayed on near-eye out-of-focus displays, and that perception is further improved if direct interaction with the content is allowed.

Persons that suffered from a cardiac disease are often recommended to integrate a sufficient level of physical exercise in their daily life. Initially, cardiac rehabilitation takes place in a closely monitored setting in a hospital or a rehabilitation center. Sustaining the effort once the patient has left the ambulatory, supervised environment is a challenge, and drop-out rates are high. Emerging approaches such as telemonitoring and telerehabilitation have been proven to show the potential to support the cardiac patient in adhering to the advised physical exercise. However, most telerehabilitation solutions only support a limited range of physical exercise, such as step-counting during walking. We propose BoB (Back on Bike), a mobile application that guides cardiac patients while cycling. Design choices are explained according to three pillars: ease of use, reduce fear, and direct and indirect motivation. In this paper, we report the results from a field study with cardiac patients.

Human-Robot Collaboration is increasingly considered in manufacturing to better combine the strengths of humans and robots. Establishing this human-robot collaboration may require multi-modal interaction; input to and output from the robot can both use multiple channels in sequence or in parallel. Designing effective interaction requires the expertise from different domains, possibly originating from people with different backgrounds. In our work we explore how composite events — hierarchical composition of events — can be used in a way that eases the communication within a multi-disciplinary team. In this paper, we present how the concept of composite events can be used to create different layers of abstraction that can be used to ease prototyping and discussion of human-robot collaboration with stakeholders through a supporting tool called Hasselt UIMS. At the lower level(s) of abstraction, the composite events can be mapped to the message-based communication as implemented in the Robotic Operating System (ROS), which is used to program collaborative robots, such as the Baxter robot from Rethink Robotics.

Nonverbal and unconscious behaviour is an important component of daily human-human interaction. This is especially true in situations such as public speaking, job interviews or information sensitive conversations, where researchers have shown that an increased awareness of one's behaviour can improve the outcome of the interaction. With wearable technology, such as Google Glass, we now have the opportunity to augment social interactions and provide realtime feedback on one's behaviour in an unobtrusive way. In this paper we present Logue, a system that provides realtime feedback on the presenters' openness, body energy and speech rate during public speaking. The system analyses the user's nonverbal behaviour using social signal processing techniques and gives visual feedback on a head-mounted display. We conducted two user studies with a staged and a real presentation scenario which yielded that Logue's feedback was perceived helpful and had a positive impact on the speaker's performance.

We present PaperPulse, a design and fabrication approach that enables designers without a technical background to produce standalone interactive paper artifacts by augmenting them with electronics. With PaperPulse, designers overlay pre-designed visual elements with widgets available in our design tool. PaperPulse provides designers with three families of widgets designed for smooth integration with paper, for an overall of 20 different interactive components. We also contribute a logic demonstration and recording approach, Pulsation, that allows for specifying functional relationships between widgets. Using the final design and the recorded Pulsation logic, PaperPulse generates layered electronic circuit designs, and code that can be deployed on a microcontroller. By following automatically generated assembly instructions, designers can seamlessly integrate the microcontroller and widgets in the final paper artifact.

The increasing number of smart objects in our everyday life shapes how we interact beyond the desktop. In this workshop we discussed how the interaction with these smart objects should be designed from various perspectives. This year's workshop put a special focus on affective computing with smart objects, as reflected by the keynote talk.

The creation of prototypes and their iterative adaptation are important stages in the time-consuming development process of a multimodal system. This reality has brought about the appearance of many specialized toolkits intended for facilitating the prototyping of multimodal systems. Using these toolkits, some functionalities of the intended prototype can be specified by means of a visual language instead of programming code. This article reports on a comparative study of a representative set of graphical toolkits for rapid prototyping of multimodal systems. It presents a set of criteria to identify the scope of a toolkit. According to their scopes, toolkits can be clustered into three classes called flow-based, state-based and token-based. Toolkits within each class do not only offer similar benefits to their users, but also exhibit resemblance in their underlying visual languages.

The misalignment between the timeframe of systems and that of their users can cause problems, especially when the system relies on implicit interaction. It makes it hard for users to understand what is happening and leaves them little chance to intervene. This paper introduces the design concept of slow-motion feedback, which can help to address this issue. A definition is provided, together with an overview of existing applications of this technique.

We present the concept of highly deformable mobile devices that can be transformed into various special-purpose controls in order to bring physical controls to mobile devices. Physical controls have the advantage of exploiting people's innate abilities for manipulating physical objects in the real world. We designed and implemented a prototype, called Paddle, to demonstrate our concept. Additionally, we explore the interaction techniques enabled by this concept and conduct an in-depth study to evaluate our transformable physical controls. Our findings show that these physical controls provide several benefits over traditional touch interaction techniques commonly used on mobile devices.

Omni-directional video (ODV) is a novel medium that offers viewers a 360º panoramic recording. This type of content will become more common within our living rooms in the near future, seeing that immersive displaying technologies such as 3D television are on the rise. However, little attention has been given to how to interact with ODV content. We present a gesture elicitation study in which we asked users to perform mid-air gestures that they consider to be appropriate for ODV interaction, both for individual as well as collocated settings. We are interested in the gesture variations and adaptations that come forth from individual and collocated usage. To this end, we gathered quantitative and qualitative data by means of observations, motion capture, questionnaires and interviews. This data resulted in a user-defined gesture set for ODV, alongside an in-depth analysis of the variation in gestures we observed during the study.

During remote video-mediated assistance, instructors often guide workers through problems and instruct them to perform unfamiliar or complex operations. However, the workers' performance might deteriorate due to stress. We argue that informing biofeedback to the instructor, can improve communication and lead to lower stress. This paper presents a thorough investigation on mental workload and stress perceived by twenty participants, paired up in an instructor-worker scenario, performing remote video-mediated tasks. The interface conditions differ in task, facial and biofeedback communication. Two self-report measures are used to assess mental workload and stress. Results show that pairs reported lower mental workload and stress when instructors are using the biofeedback as compared to using interfaces with facial view. Significant correlations were found on task performance with reducing stress (i.e. increased task engagement and decreased worry) for instructors and declining mental workload (i.e. increased performance) for workers. Our findings provide insights to advance video-mediated interfaces for remote collaborative work.

We present ReHoblet; a physical rehabilitation game on tablets, designed to be used in a residential setting. ReHoblet trains two gross motor movements of the upper limbs by lifting (up-down) and transporting (left-right) the tablet to control a simple platform game. By using its accelerometers and gyroscope, the tablet is capable of detecting movements made by the user and steer the interaction based on this data. A formative evaluation with five Multiple Sclerosis (MS) patients and their therapists showed high appreciation for ReHoblet. Patients stated they liked ReHoblet not only to improve their physical abilities, but to train on performing technology-related tasks. Based on the results, we reflect on tablet-based games in home-based rehabilitation.

Non-verbal and unconscious behavior plays an important role for efficient human-to-human communication but are often undervalued when training people to become better communicators. This is particularly true for public speakers who need not only behave according to a social etiquette but do so while generating enthusiasm and interest for dozens if not hundreds of other persons. In this paper we propose the concept of social augmentation using wearable computing with the goal of giving users the ability to continuously monitor their performance as a communicator. To this end we explore interaction modalities and feedback mechanisms which would lend themselves to this task.

Smartphones leak personal information about their owner when they use it to connect to the Internet. Despite recent coverage of these issues in popular media, raising awareness remains problematic since it remains largely invisible to the users. We designed a system, SASQUATCH, consisting of a netwo

Persons with a neurological disorder are confronted with significantly reduced physical abilities during their daily activities. Physiotherapy, for these patients mainly provided in rehabilitation centres, utilizes tangible, real-world objects in training for the upper limbs. Only by intensely and frequently exercising, patients have a chance to sustain or enhance their functional performance. Our research explores pervasive technologies and tangible objects to provide motivating, technology-supported training systems in a residential environment for independent use by these patients. In this paper, we describe our pervasive training system 'PhysiCube', consisting of prototypes 'LiftACube' and 'ReachACube'. PhysiCube takes advantage of tangible interactions and games to provide motivating physical training for the upper limbs. An evaluation with therapists showed great appreciation for our prototypes. Reflections on the technical setup and use of tangible interaction for these pervasive rehabilitation systems in a residential setting are elaborated upon.

Modern knowledge work consists of both individual and highly collaborative activities that are typically composed of a number of configuration, coordination and articulation processes. The desktop interface today, however, provides very little support for these processes and rather forces knowledge workers to adapt to the technology. We introduce co-Activity Manager, an activity-centric desktop system that (i) provides tools for ad hoc dynamic configuration of a desktop working context, (ii) supports both explicit and implicit articulation of ongoing work through a built-in collaboration manager and (iii) provides the means to coordinate and share working context with other users and devices. In this paper, we discuss the activity theory informed design of co-Activity Manager and report on a 14 day field deployment in a multi-disciplinary software development team. The study showed that the activity-centric workspace supports different individual and collaborative work configuration practices and that activity-centric collaboration is a two-phase process consisting of an activity sharing and per-activity coordination phase.

Feedback and affordances are two of the most well-known principles in interaction design. Unfortunately, the related and equally important notion of feedforward has not been given as much consideration. Nevertheless, feedforward is a powerful design principle for bridging Norman's Gulf of Execution. We reframe feedforward by disambiguating it from related design principles such as feedback and perceived affordances, and identify new classes of feedforward. In addition, we present a reference framework that provides a means for designers to explore and recognize different opportunities for feedforward.

Intelligent User Interfaces can benefit from having knowledge on the user's emotion. However, current implementations to detect affective states, are often constraining the user's freedom of movement by instrumenting her with sensors. This prevents affective computing from being deployed in naturalistic and ubiquitous computing contexts. In this paper, we present a novel system called mASqUE, which uses a set of association rules to infer someone's affective state from their body postures. This is done without any user instrumentation and using off-the-shelf and non-expensive commodity hardware: a depth camera tracks the body posture of the users and their postures are also used as an indicator of their openness. By combining the posture information with physiological sensors measurements we were able to mine a set of association rules relating postures to affective states. We demonstrate the possibility of inferring affective states from body postures in ubiquitous computing environments and our study also provides insights how this opens up new possibilities for IUI to access the affective states of users from body postures in a nonintrusive way.

Smart objects can be smart because of the information and communication technology that is added to human-made artifacts. It is not, however, the technology itself that makes them smart but rather the way in which the technology is integrated, and their smartness surfaces through how people are able to interact with these objects. Hence, the key challenge for making smart objects successful is to design usable and useful interactions with them. We list five features that can contribute to the smartness of an object, and we discuss how smart objects can help resolve the simplicity-featurism paradox. We conclude by introducing the three articles in this special issue, which dive into various aspects of smart object interaction: augmenting objects with projection, service-oriented interaction with smart objects via a mobile portal, and an analysis of input-output relations in interaction with tangible smart objects.

Activity-Based Computing (ABC) has been proposed as an organisational structure for local desktop management and knowledge work. Knowledge work, however, typically occurs in partially overlapping subgroups and involves the use of multiple devices. We introduce co-Activity Manager, an ABC approach that (i) supports activity sharing for multiple collaborative contexts, (ii) includes collaborative tools into the activity abstraction and (iii) supports multiple devices by seamlessly integrated cloud support for documents and activity storage. Our 14 day field deployment in a multidisciplinary software development team showed that activity sharing is used as a starting point for long-term collaboration while integrated communication tools and cloud support are used extensively during the collaborative activities. The study also showed that activities are used in different ways ranging from project descriptions to to-do lists, thereby confirming that a document-driven activity roaming model seems to be a good match for collaborative knowledge work

Interactive surfaces have great potential for co-located collaboration because of their ability to track multiple inputs simultaneously. However, the multi-user experience on these devices could be enriched significantly if touch points could be associated with a particular user. Existing approaches to user identification are intrusive, require users to stay in a fixed position, or suffer from poor accuracy. We present a non-intrusive, high-accuracy technique for mapping touches to their corresponding user in a collaborative environment. By mounting a high-resolution camera above the interactive surface, we are able to identify touches reliably without any extra instrumentation, and users are able to move around the surface at will. Our technique, which leverages the back of users' hands as identifiers, supports walk-up-and-use situations in which multiple people interact on a shared surface.

Informing caregivers by providing them with contextual medical information can significantly improve the quality of patient care activities. However, information flow in hospitals is still tied to traditional manual or digitised lengthy patient record files that are often not accessible while caregivers are attending to patients. Leveraging the proliferation of pervasive awareness technologies (sensors, actuators and mobile displays), recent studies have explored this information presentation aspect borrowing theories from context-aware computing, i.e., presenting subtle information contextually to support the activity at hand. However, the understanding of the information space (i.e., what information should be presented) is still fairly abstruse, which inhibits the deployment of such real-time activity support systems. To this end, this paper first presents situated glyphs, a graphical entity to encode situation specific information, and then presents our findings from an in-situ qualitative study addressing the information space tailored to such glyphs. Applying technology probes using situated glyphs and different glyph display form factors, the study aimed at uncovering the information space pertained to both primary and secondary medical care. Our analysis has resulted in a large set of information types as well as given us deeper insight on the principles for designing future situated glyphs. We report our findings in this paper that we expect would provide a solid foundation for designing future assistive systems to support patient care activities.

The number of smart objects in our everyday life is steadily increasing. In this workshop we discuss how the interaction with these smart objects should be designed from various perspectives.

When creating applications that should be available on multiple computing platforms, designers have to cope with different design tools and user interface toolkits. Incompatibilities between these design tools and toolkits make it hard to keep multi-device user interfaces consistent. This paper presents Jelly, a flexible design environment that can target a broad set of computing devices and toolkits. Jelly enables designers to copy parts of a user interface from one device to another and to maintain the different user interfaces in concert using linked editing. Our approach lowers the burden of designing multi-device user interfaces by eliminating the need to switch between different design tools and by providing tool support for keeping the user interfaces consistent across different platforms and toolkits.

Bringing together heterogeneous computing devices and appliances gives rise to a spontaneous environment where resources exchange messages, such as a mobile phone telling the car's stereo to mute. We also witness computer-augmented resources become physically simpler to use (e. g. less buttons) but become more complex to handle in their digital dimension (e. g. overloaded user interfaces). As a consequence, the behavior of the pervasive applications leveraging these resources gets even more complex to understand and configure. This demands for tools that help developers and end-users inspect and manipulate the current state of the pervasive computing environment during execution time. We present models and tools that support the development and deployment of applications that can be observed at runtime, by means of the messages they exchange, the properties they manipulate and the rules they define.

Pervasive applications are typically realized through ad-hoc service and user interface compositions. While many tools focus on the development of pervasive services by masking the complex technical side of a pervasive computing environment, the deployment of an application as a whole -- i.e. a set of services and user interfaces -- is often forgotten. We present an alternative design strategy and tool for pervasive applications in which pervasiveness is not considered a handicap, but rather as a situation that draws extra attention to the deployment of applications. By crafting pervasive applications and their services as independent context consumers and producers, we illustrate how the behaviour of a pervasive application deployed using our approach can be observed while it executes.

Users often become frustrated when they are unable to understand and control a pervasive computing environment. Previous studies have shown that allowing users to pose why and why not questions about context-aware applications resulted in better understanding and stronger feelings of trust. Although why and why not questions have been used before to aid in debugging and to clarify graphical user interfaces, it is currently not clear how they can be integrated into pervasive computing systems. We explain in detail how we have extended an existing pervasive computing framework with support for why and why not questions. This resulted in PervasiveCrystal, a system for asking and answering why and why not questions in pervasive computing environments.

Efficient discovery of nearby devices and services is one of the preconditions to obtain a usable pervasive environment. Typical user interfaces in these environments hide the heterogeneity of the environment for end-users which often makes it hard to perceive the provided functionality. We present Pervasive Maps, an approach and tool that allows to create an intuitive user interface for exploring and controlling the environment. Pervasive Maps offers user-oriented views on the user's environment based on pictures of this environment. We show how users can model, explore and finally interact with complex pervasive environments using Pervasive Maps.

Multi-device user interface design mostly implies creating suitable interface for each targeted device, using a diverse set of design tools and toolkits. This is a time consuming activity, concerning a lot of repetitive design actions without support for reusing this effort in later designs. In this paper, we propose D-Macs: a design tool that allows designers to record their design actions across devices, to share these actions with other designers and to replay their own design actions and those of others. D-Macs lowers the burden in multi-device user interface design and can reduce the necessity for manually repeating design actions.

As the variety in network service platforms and end user devices grows rapidly, content providers must constantly adapt their production system to support these new technologies. In this paper, we present a middleware platform for deploying highly interactive (television) applications over a diverse collection of networks and end user devices. As the user interface of such interactive applications may vary depending on the capabilities of the different target devices, our middleware uses UIML for the description of generic user interfaces. Our middleware platform also provides a pluggable support for new networks. A factor that highly complicates the design is the need for strict synchronization between an interactive application and video or audio data that is broadcasted. In order to support a maximum of functionality, downloadable application logic is used to provide the interactive services. As a test case, an evaluation setup was built, targeting both set-top boxes and mobile phones.

In current mobile user interface design tools, it is time consuming to export a design to the target device. This makes it hard for designers to iterate over the user interfaces they are creating. We propose Gummy-live, a GUI builder for mobile devices allowing designers to test and observe immediately on the target device each step they take in the GUI builder. This way, designers are stimulated to iteratively test and refine user interface prototypes in order to take the target device characteristics into account.

introduction Share on Introduction to the special issue on UIDL for next-generation user interfaces Editors: Orit Shaer Wellesley College Wellesley CollegeView Profile , Robert J. K. Jacob Tufts University Tufts UniversityView Profile , Mark Green University of Ontario Institute of Technology University of Ontario Institute of TechnologyView Profile , Kris Luyten Hasselt University Hasselt UniversityView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 16Issue 4Article No.: 16pp 1–3https://doi.org/10.1145/1614390.1614391Published:27 November 2009Publication History 8citation870DownloadsMetricsTotal Citations8Total Downloads870Last 12 Months3Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Designers still often create a specific user interface for every target platform they wish to support, which is time-consuming and error-prone. The need for a multi-platform user interface design approach that designers feel comfortable with increases as people expect their applications and data to go where they go. We present Gummy, a multi-platform graphical user interface builder that can generate an initial design for a new platform by adapting and combining features of existing user interfaces created for the same application. Our approach makes it easy to target new platforms and keep all user interfaces consistent without requiring designers to considerably change their work practice.

This paper presents how ad-hoc co-located collaborations can be supported with an arbitrary number of users that only have acces to small-size mobile displays. Our approach is based on tracking of these personal displays that share the same information space. All displays involved in the collaboration act as autonomous windows on a set of data items (the information space) positioned on a shared virtual canvas. Data items are identified by their three-dimensional location in physical space and can be manipulated through the displays that serve as windows on the shared canvas. Each display is tracked in physical space and is aware of its own location. Since different mobile displays can access and manipulate the same information space, a distributed locking mechanism makes sure the data stays consistent during simultaneous access of data in this information space.

This paper presents a modular runtime architecture supporting our model-based user interface design approach for designing context-aware, distributable user interfaces for ambient intelligent environments.

In this paper we show how an interactive system can be distributed among several peer devices. By taking advantage of the current trend towards ambient intelligent environments, we can make use of a combination of computing resources in the surrounding of the user to function as one logical interactive system; an interaction space. Our approach relies on the fact that nowadays most computing resources are network enabled and publish their device profile using some special purpose protocol. For this reason, federations of devices that support the tasks of the user can be composed automatically according to the requirements of these tasks. This distribution of the user interface over a federation of devices can be local or nonlocal. It raises the opportunity for supporting collaborative tasks with the same user interface with little or no extra effort from the user interface designer. Future tools supporting the design, creation and deployment of distributed interactive systems using device federations should maintain usability and usefulness of a dynamic distributed system. We use two different metrics to cope with these problems: interface completeness and interface continuity.

The usage of computing systems has evolved dramatically over the last years. Starting from a low level procedural usage, in which a process for executing one or several tasks is carried out, computers now tend to be used in a problem oriented way. Future computer usage will be more centered around particular services, and will not be focused on platforms or applications. These services should be independent of the technology used to interact with them. In this paper an approach will be presented which provides a uniform interface to such services, without any dependence on modality, platform or programming language. Through the usage of general user interface descriptions, presented in XML, and converted using XSLT, a uniform framework is presented for runtime migration of user interfaces. As a consequence, future services will become easily extensible for all kinds of devices and modalities. Special attention goes out to a component-based software development approach. Services represented by and grouped in components can offer a special interface for modal- and device-independent rendering. Components become responsible for describing their own possibilities and constraints for interacting. An implementation serving as a proof of concept, a runtime conversion of a joystick in a 3D virtual environment into a 2D dialog-based user interface, is developed.