The cube is an object often used for tangible interaction across many fields of application, notably by manipulating cubes or performing gestures with or on them. Although some system- or designer-defined gestures exist for tangible cubes, user-defined gestures for them remain unexplored. To bridge this gap, we conducted three gesture elicitation studies, each using an identical experimental protocol on a different sample of 30 participants proposing a gesture for 13, 16, and 20 referents controlling a smart home appliance, respectively. Based on the total of 1,440 gesture proposals, we established a classification of user-defined gestures for designing tangible interaction with cubes in smart home control structured into seven major categories, which are further broken down into 55 subcategories. This classification is abstracted from properties such as the number of fingers and hands used to manipulate a cube, the repetition of gestures, their combination, and the time of articulation. A Poisson law estimates the number of participants to be recruited to obtain a desired number of gesture categories.

This paper examines how awareness interventions relate to emotional responses to invisible risks in immersive systems, with an empirical focus on virtual reality. Across two complementary studies, we examine emotional responses elicited by a lecture and by narrative design-fiction storyboards focusing on biometric profiling and overengagement—two risks that undermine user agency and autonomy. The interventions elicit distinct emotional patterns, pointing to different ways in which invisible risks become salient across cognitive, affective, and moral dimensions. Qualitative responses indicate recognition of manipulation, loss of agency, and ethical concern. The results suggest that concise narrative scenarios can render otherwise imperceptible risks concrete and relatable.

Adaptive user interfaces dynamically change their content, presentation, and behavior to optimize the user experience, which has been primarily evaluated using classic usability measures but to a lesser extent by using neurological measures. While the perceived preference of specific user interface elements, such as graphical adaptive menus, has already been studied, no consensus exists regarding their performance and how to substitute a static menu with an adaptive one. To gain insights into how graphical adaptive menus could influence the user experience and to identify any correlation between users’ performance and their preferences, we conducted an experiment in which forty participants used twenty graphical adaptive menus while their brain activity was captured by employing electroencephalography to derive four measures ( i.e. , cognitive load, engagement, attraction, and memorization). User performance was measured using task completion time, specifically the time to select menu items. Statistical analysis suggested which graphical adaptive menus were significantly better or worse than the static menu, our baseline. These results are used as the basis to suggest implications for software developers and researchers to design more effective adaptive user interfaces.

User interface requirements elicitation is an early and critical stage in the software development life cycle: requirements are elicited from stakeholders, and user interface elements that should satisfy these requirements are also created, shared, and edited. To effectively capture the diverse perspectives of stakeholders on these requirements, crowd elicitation is a promising approach to leverage the collective knowledge of a time- and space-distributed group of stakeholders to manage user interface requirements and their associated user interface elements. Annotating these elements manually in freeform digital ink leads to results that are difficult to interpret and time-consuming to exploit by designers and developers. Instead, we propose that stakeholders annotate user interface elements by commonly-agreed gestures that will be automatically recognized to populate 10 annotation metrics. To this end, we present Crowditivity , a publicly available web application that effectively harnesses crowd elicitation in three related activities: (1) collecting and managing user interface functional and non-functional requirements according to an adapted Volere Shell template; (2) assigning user interface elements of a mockup or prototype to requirements to satisfy them; and (3) enabling stakeholders to annotate these elements by ink-based gestures. A 2D multi-stroke gesture classifier is specifically tailored to recognize gesture classes corresponding to 18 annotation types, that are empirically validated through a gesture elicitation study ( n 1 = 26) confirmed by an identification study ( n 2 = 59). Through an evaluation of a case study for a web application, we investigated how eight representative designers valued the Crowditivity platform. We finally define and illustrate a set of ten metrics to quantify the output of Crowditivity in terms of annotations and annotators resulting from the gesture recognition.

2D-3D gesture interaction most often associates a recognized gesture with a single command, such as “Swipe right” to navigate to the next screen. When gestures are performed one after another in a gesture sequence, each gesture still needs to be associated with one separate command at a time. Increasing the number of such gestures to realize more commands is not favorable to the end user, who memorizes and uses only a limited number of gestures. Instead of relying on gesture sequences, we can expand the range of executable commands to perform more complex tasks by gesture composition. To this end, we define an instruction as an action applied to a device in an environment, together with its valued parameters, such as “Increase the brightness of the bedroom lamp to 10“. Elementary gestures, preferably congruent and hierarchical, are composed into a compound gesture. The instructions can be combined into macro-instructions to summarize gesture composition. To put these definitions in practice, we developed Emeriti , a framework for the recognition of elementary and compound gestures in both 2D (uni- or multistroke gestures recognized on an interactive surface) and 3D (point trajectories in space) by composition. We demonstrate the effectiveness and usefulness of turning gesture sequences into gesture composition using (macro-)instructions in an application simulating a smart home in which every device in every room can be controlled by compound gestures with parameters. This makes it possible to multiply the number of possible commands for more complex tasks without considerably increasing the number of gestures.

Figure 1: Photograph captured during one of our missions at the Mars Desert Research Station, depicting an environment characterized by extreme constraints, isolation, and confinement, with harsh conditions that can affect physical and psychological well-being.Note the deserted landscape, the strong sunlight, and the bulky equipment.In this setting, closely mimicking a Mars analog environment, we conducted two user experience studies exploring various pragmatic and hedonic dimensions of interaction with computer systems, with a baseline represented by user experience measured in a conventional environment.

Large Language Models (LLMs) such as Chat-GPT demonstrate significant potential in the medical domain and are often evaluated using multiple-choice questions (MCQs) modeled on exams like the USMLE.However, such benchmarks may overestimate true clinical understanding by rewarding pattern recognition and test-taking heuristics.To investigate this, we created a fictional medical benchmark centered on an imaginary organ, the Glianorex, allowing us to separate memorized knowledge from reasoning ability.We generated textbooks and MCQs in English and French using leading LLMs, then evaluated proprietary, open-source, and domain-specific models in a zero-shot setting.Despite the fictional content, models achieved an average score of 64%, while physicians scored only 27%.Fine-tuned medical models outperformed base models in English but not in French.Ablation and interpretability analyses revealed that models frequently relied on shallow cues, test-taking strategies, and hallucinated reasoning to identify the correct choice.These results suggest that standard MCQ-based evaluations may not effectively measure clinical reasoning and highlight the need for more robust, clinically meaningful assessment methods for LLMs.

Effective decision-making often relies on timely insights from complex visual data. While Information Visualization (InfoVis) dashboards can support this process, they rarely adapt to users’ cognitive state, and less so in real time. We present SYMBIOTIK, an intelligent, context-aware adaptive visualization system that leverages neurophysiological signals to estimate mental workload (MWL) and dynamically adapt visual dashboards using reinforcement learning (RL). Through a user study with 120 participants and three visualization types, we demonstrate that our approach improves task performance and engagement. SYMBIOTIK offers a scalable, real-time adaptation architecture, and a validated methodology for neuroadaptive user interfaces.

We introduce TapStrapGest , a novel solution for customizable ring-based multi-finger gestures, encompassing the process from gesture elicitation to gesture recognition. Recognizing the growing demand for intuitive and customizable gesture interaction with fingers, TapStrapGest uses Tap Strap to enable users to perform simple and complex multi-finger gestures using smart rings. We conducted a gesture elicitation study, detailing the systematic process of soliciting and refining a custom set of user-defined ring-based finger gestures through participatory design and ergonomic considerations, including thinking time, goodness of fit, and memorization. Subsequently, we delve into the technical underpinnings of gesture recognition. We reduce the dimensionality of a dataset of 27 gesture classes from 21 to 15 by filtering, then from 15 to 5 by a Principal Component Analysis. We implement and compare four machine learning algorithms to show that a Quadratic Discriminant Analysis (precision=99.33%, recall=99.26%, and F1-score=99.26%) outperforms three other machine learning classifiers, i.e., a Linear Discriminant Analysis, a Support Vector Machines, and a Random Forest, as well as existing recognizers from the literature, to accurately recognize such gestures without the need to call for Deep Learning. Through a performance analysis, we demonstrate that TapStrapGest is a versatile and admissible solution for ring-based multi-finger gesture interaction, opening avenues for "eyes-free" or "screen-free" human-computer interaction in various domains.

The wide range of algorithms for recognizing two-dimensional stroke gestures raises a crucial question: which recognizer should be selected depending on the gesture set and its context of use? Since the context of use expresses a category of users performing interactive tasks on a device in a physical environment, many contextual conditions are feasible. Recognizers have been tested in only a few of them, making them difficult to compare and select. A recognizer suitable for one context of use does not necessarily translate to another. Recognizer variables, such as the number of resampling points, the number of templates, and the number of users, vary from one recognizer to another, making them challenging to specify for practitioners and heterogeneous to compare for researchers. To address these issues, this article performs comparative tests of 16 recognizers (15 state-of-the-art recognizers and one for rotation invariance) in 11 gesture sets (9 reference sets and 2 for multi-device rotation invariance) against their accuracy, speed, and algorithmic complexity, in user-(in)dependent scenarios based on the variables mentioned above. In particular, the impact of two contextual dimensions, i.e., the visual impairments of end users and input devices, on accuracy is further studied. The comparative testing results in an empirically validated decision table for the researcher to select a recognizer depending on contextual conditions and a flow chart for the practitioner to be guided. We break down our design implications into two categories: gesture acquisition, recognition, and gesture set design. We also publicly release Gester , a JavaScript software for conducting comparative testing of recognizers under various contextual conditions, allowing the practitioner to test a recognizer in a project and the researcher to benchmark recognizers.

The performance of state-of-the-art stroke gesture recognizers has been extensively tested against accuracy and speed in the user-dependent or user-independent procedure on a wide variety of gesture sets. These tests were carried out under heterogeneous experimental conditions, making the results non-comparable from one source to another, and calling for a consistent assessment that ensures reproducibility of the results. When a new gesture set should be incorporated in a gesture-based user interface and/or when a new recognizer is made available, this gesture set needs to be tested with existing and new recognizers to determine which recognizer offers the best performance in a given context of use. To this end, this technical note presents Gester , a software that automates the performance testing of stroke gesture recognizers in terms of accuracy (through three recognition rates) and speed (through four execution times). It exports data for visualizing confusion matrices and confusion wheels. We elaborate on typical use cases supported by Gester , which include the two traditional testing procedures, i.e., user-dependent and user-independent scenarios, and provide two new ones, i.e., dataset-dependent and dataset-independent, to test the performance of recognizers under new conditions. Then, we illustrate a series of activities.

The evolving landscape of distributed user interfaces requires the prototyping stage also be distributed between users, tasks, platforms, and environments. To create a cohesive distribution of the user interface elements in such ecosystems, paired sketching has emerged as a collaborative design method that leverages multiple stakeholders’ strengths, including designers, developers, and end users, working in pairs. In the context of developer experience applied to paired sketching for distributed user interfaces, we decomposed a workflow into four disciplines according to the Software and Systems Process Engineering Meta-Model (SPEM) notation. First, we defined a protocol to deploy paired sketching of distributed user interfaces, supported by UbiSketch , a collaborative software environment tailored featuring sketch recognition and whiteboarding. Second, to evaluate paired sketching for engineering interactive systems, we conducted an experiment involving five pairs of stakeholders who sketched a distributed user interface for inside-the-vehicule interaction distributed on four platforms: smartphone, tablet, pen display, and tabletop. Empirical results from questionnaires, reactivity, intention, perceived satisfaction, and free comments, suggest a preference order in which the tabletop is ranked first, followed by the tablet, smartphone, and pen display. Based on these results, we discuss the potential of paired sketching for distributed user interfaces.

Isolated, Confined, and Extreme (ICE) environments, such as encountered in space exploration missions, pose unique physical and psychological challenges that influence user interactions with computer systems, yet remain considerably less documented compared to conventional settings. To investigate the impact of such environments on mobile interaction, we conducted an experiment involving a crew of analog astronauts operating a drone via a handheld controller in both a conventional Earth-based setting and an ICE environment represented by the extreme landscape of the Mars Desert Research Station. Our findings reveal how the user experience of mobile interaction evolves over multiple evaluation sessions conducted over a two-week period in the ICE environment, for which we analyze both pragmatic and hedonic dimensions, such as perceived efficiency, adaptability, novelty, usefulness, and trust. Based on our findings, we outline a set of implications for the design of mobile interaction intersecting space research through the distinctive lens of astronaut-drone interaction.

Modern in-vehicle infotainment systems connect to drivers’ personal digital devices, such as smartphones, to facilitate calls, notifications delivery, and access to online media and resources while driving. This communications process facilitates distributed user interfaces (DUIs) and allows casual in-vehicle interactions. However, such functionalities of modern vehicles still lack the extensive capabilities of state-of-the-art DUIs, which can offer a variety of interaction opportunities through features such as migration, redirection, adaptation, and granularity, among others. In this work, we explore DUIs in the specific context of smart vehicles, and introduce V2DUIs (Vehicle-to-DUIs) as a new addition to the V2X (Vehicle-to-Everything) family of smart vehicle concepts.

Affective computing has potential to enrich the development lifecycle of Graphical User Interfaces (GUIs) and of intelligent user interfaces by incorporating emotion-aware responses. Yet, affect is seldom considered to determine whether a GUI design would be perceived as good or bad. We study how physiological signals can be used as an early, effective, and rapid affective assessment method for GUI design, without having to ask for explicit user feedback. We conducted a controlled experiment where 32 participants were exposed to 20 good GUI and 20 bad GUI designs while recording their eye activity through eye tracking, facial expressions through video recordings, and brain activity through electroencephalography (EEG). We observed noticeable differences in the collected data, so we trained and compared different computational models to tell good and bad designs apart. Taken together, our results suggest that each modality has its own “performance sweet spot” both in terms of model architecture and signal length. Taken together, our findings suggest that is possible to distinguish between good and bad designs using physiological signals. Ultimately, this research paves the way toward implicit evaluation methods of GUI designs through user modeling.

OBJECTIVE: To optimize the training strategy of large language models for medical applications, focusing on creating clinically relevant systems that efficiently integrate into healthcare settings, while ensuring high standards of accuracy and reliability. MATERIALS AND METHODS: We curated a comprehensive collection of high-quality, domain-specific data and used it to train several models, each with different subsets of this data. These models were rigorously evaluated against standard medical benchmarks, such as the USMLE, to measure their performance. Furthermore, for a thorough effectiveness assessment, they were compared with other state-of-the-art medical models of comparable size. RESULTS: The models trained with a mix of high-quality, domain-specific, and general data showed superior performance over those trained on larger, less clinically relevant datasets (P < .001). Our 7-billion-parameter model Med5 scores 60.5% on MedQA, outperforming the previous best of 49.3% from comparable models, and becomes the first of its size to achieve a passing score on the USMLE. Additionally, this model retained its proficiency in general domain tasks, comparable to state-of-the-art general domain models of similar size. DISCUSSION: Our findings underscore the importance of integrating high-quality, domain-specific data in training large language models for medical purposes. The balanced approach between specialized and general data significantly enhances the model's clinical relevance and performance. CONCLUSION: This study sets a new standard in medical language models, proving that a strategically trained, smaller model can outperform larger ones in clinical relevance and general proficiency, highlighting the importance of data quality and expert curation in generative artificial intelligence for healthcare applications.

Background: Adaptive user interfaces have the advantage of being able to dynamically change their aspect and/or behaviour depending on the characteristics of the context of use, i.e. to improve user experience. User experience is an important quality factor that has been primarily evaluated with classical measures (e.g. effectiveness, efficiency, satisfaction), but to a lesser extent with physiological measures, such as emotion recognition, skin response, or brain activity. Aim: In a previous exploratory experiment involving users with different profiles and a wide range of ages, we analysed user experience in terms of cognitive load, engagement, attraction and memorisation when employing twenty graphical adaptive menus through the use of an Electroencephalogram (EEG) device. The results indicated that there were statistically significant differences for these four variables. However, we considered that it was necessary to confirm or reject these findings using a more homogeneous group of users. Method: We conducted an operational internal replication study with 40 participants. We also investigated the potential correlation between EEG signals and the participants’ user experience ratings, such as their preferences. Results: The results of this experiment confirm that there are statistically significant differences between the EEG variables when the participants interact with the different adaptive menus. Moreover, there is a high correlation among the participants’ user experience ratings and the EEG signals, and a trend regarding performance has emerged from our analysis. Conclusions: These findings suggest that EEG signals could be used to evaluate user experience. With regard to the menus studied, our results suggest that graphical menus with different structures and font types produce more differences in users’ brain responses, while menus which use colours produce more similarities in users’ brain responses. Several insights with which to improve users’ experience of graphical adaptive menus are outlined.

Browsing multimedia objects, such as photos, videos, documents, and maps represents a frequent activity in a context of use where an end-user interacts on a large vertical display close to bystanders, such as a meeting in a corporate environment or a family display at home. In these contexts, mid-air gesture interaction is suitable for a large variety of end-users, provided that gestures are consistently mapped to similar functions across media types. We present Lui (Large User Interface), a ready-to-deploy and to-use application for browsing multimedia objects by consistent mid-air gesture interaction on a large display that is customizable by mapping new gesture classes to functions in real-time. The method followed to design the gesture interaction and to develop the application consists of four stages: (1) a contextual gesture elicitation study (23 participants × 18 referents = 414 proposed gestures) is conducted with the various media types to determine a consensus set satisfying consistency, (2) the continuous integration of this consensus set with gesture recognizers into a pipeline software architecture, (3) a comparative testing of these recognizers on the consensus set to configure the pipeline with the most efficient ones, and (4) an evaluation of the interface regarding its global quality and specific to the implemented gestures.

The field of generative artificial intelligence has seen significant advancements in recent years with the advent of large language models, which have shown impressive results in software engineering tasks but not yet in engineering user interfaces. Thus, we raise a specific research question: would an LLM-based system be able to search for relevant GUI layouts? To address this question, we conducted a controlled study evaluating how Instigator, an LLM-based system for searching GUI layouts of web pages by generative pre-trained training, would return GUI layouts that are relevant to a given instruction and what would be the user experience of (N =34) practitioners interacting with Instigator. Our results identify a very high similarity and a moderate correlation between the rankings of the GUI layouts generated by Instigator and the rankings of the practitioners with respect to their relevance to a given design instruction. We highlight the results obtained through thirteen UEQ+ scales that characterize the user experience of the practitioner with Instigator, which we use to discuss perspectives for improving such future tools.

Radar sensing technologies offer several advantages over other gesture input modalities, such as the ability to reliably sense human movements, a reasonable deployment cost, insensitivity to ambient conditions such as light, temperature, and the ability to preserve anonymity. These advantages come at the price of high processing complexity mainly due to the spatio-temporal variations of gesture articulation performed by different people. Deep learning methods, such as CNN-LSTM and 3D CNN-LSTM, have a high potential to recognize radar-based gestures but usually require hundreds or thousands of labeled training samples and high processing power. Asking a lot of people to acquire a lot of gestures is particularly tedious and tiring to the point of being unrealistic. To overcome these challenges, we propose FORTE, a hand gesture recognition with few samples based on an optimized CNN architecture working on pre-processed raw data. Using a k=5-fold cross-validation, we define and compare three alternative CNNs for recognizing hand gestures acquired in a semi-mobile context of use with a portable radar attached to a smartphone. The best CNN reaches an accuracy of 94.96% with a precision of 95.92% and a recall of 96.03% for a dataset composed of solely 5 participants producing 2 samples for 20 classes covering 1 pointing, 2 pantomimic, 3 iconic, and 14 semaphoric gestures. We suggest some implications for designing radar-based gestures and we discuss the limitations of this approach.

Virtual reality applications offer the promise to immerse end users in a synthetic environment where several actions could be observed, simulated, and reproduced, before transferring them to reality, which makes them particularly appropriate for training. Yet, when the training requires complex handling of information, the tasks become cognitively intensive, and developing adequate applications becomes challenging. To address this challenge, we define a method for developing head-mounted-display-based virtual reality applications for modular training tasks, composed of a training model with parameters, a step-wise approach for supporting this development, and a software framework enacting the application of this approach. Authoring such applications is expected to become more flexible and provide personalization facilities. To evaluate the impact of this method, we define a case study concerning an application for training school teachers who deal with a variety of situations in a classroom for an experiment involving N= 7 participants for a set of tasks. Pre-study and post-study acceptances reveal the impact of the software framework and a workload evaluation is conducted using the NASA TLX questionnaire.

User interface aesthetics, a particular sub-characteristic of the ISO 25010 software quality model, is correlated to the perceived or actual usability of a graphical user interface, its user experience, and trust. While many measures, such as balance, symmetry, proportion, alignment, regularity, and simplicity, can be computed, no consensus exists today on which measure to adopt, which formula to compute for each measure, and which interpretation to give for each computed formula. To accommodate these variations and to make the assessment explicit and interpretable, we present a quality model-based approach for measuring aesthetics that defines a quality function in terms of the formula of the measures, their weights, their composition, and their overall computation. This quality model is transformed into a configuration used by GRACE, a web application developed for this purpose. When the measures, their formula, their weights, or the quality function change, the quality model changes, which is re-computed to compare it with any other model, thus making the measurement process explicit and interpretable. We apply this approach to a large dataset "Lab in the Wild'' to investigate the correlations between aesthetic measures and perceived visual complexity and colorfulness. We discuss limitations through threats to validity.

Microwave radars bring many benefits to mid-air gesture sensing due to their large field of view and independence from environmental conditions, such as ambient light and occlusion. However, radar signals are highly dimensional and usually require complex deep learning approaches. To understand this landscape, we report results from a systematic literature review of ( N =118) scientific papers on radar sensing, unveiling a large variety of radar technology of different operating frequencies and bandwidths and antenna configurations but also various gesture recognition techniques. Although highly accurate, these techniques require a large amount of training data that depend on the type of radar. Therefore, the training results cannot be easily transferred to other radars. To address this aspect, we introduce a new gesture recognition pipeline that implements advanced full-wave electromagnetic modeling and inversion to retrieve physical characteristics of gestures that are radar independent, i.e., independent of the source, antennas, and radar-hand interactions. Inversion of radar signals further reduces the size of the dataset by several orders of magnitude, while preserving the essential information. This approach is compatible with conventional gesture recognizers, such as those based on template matching, which only need a few training examples to deliver high recognition accuracy rates. To evaluate our gesture recognition pipeline, we conducted user-dependent and user-independent evaluations on a dataset of 16 gesture types collected with the Walabot, a low-cost off-the-shelf array radar. We contrast these results with those obtained for the same gesture types collected with an ultra-wideband radar made of a vector network analyzer with a single horn antenna and with a computer vision sensor, respectively. Based on our findings, we suggest some design implications to support future development in radar-based gesture recognition.

The body of knowledge accumulated by gesture elicitation studies (GES), although useful, large, and extensive, is also heterogeneous, scattered in the scientific literature across different venues and fields of research, and difficult to generalize to other contexts of use represented by different gesture types, sensing devices, applications, and user categories. To address such aspects, we introduce RepliGES, a conceptual space that supports (1) replications of gesture elicitation studies to confirm, extend, and complete previous findings, (2) reuse of previously elicited gesture sets to enable new discoveries, and (3) extension and generalization of previous findings with new methods of analysis and for new user populations towards consolidated knowledge of user-defined gestures. Based on RepliGES, we introduce GEStory, an interactive design space and visual tool, to structure, visualize and identify user-defined gestures from a number of 216 published gesture elicitation studies.

We show how two recently introduced visual tools, RepliGES and GEStory, can be used conjointly to inform possible replications of Gesture Elicitation Studies (GES) with a case study centered on gestures that can be sensed with radars. Starting from a GES identified in GEStory, we employ the dimensions of the RepliGES space to enumerate eight possible ways to replicate that study towards gaining new insights into end user’s preferences for gesture-based interaction for applications that use radar sensors.

Microwave radar sensors in human-computer interactions have several advantages compared to wearable and image-based sensors, such as privacy preservation, high reliability regardless of the ambient and lighting conditions, and larger field of view. However, the raw signals produced by such radars are high-dimension and relatively complex to interpret. Advanced data processing, including machine learning techniques, is therefore necessary for gesture recognition. While these approaches can reach high gesture recognition accuracy, using artificial neural networks requires a significant amount of gesture templates for training and calibration is radar-specific. To address these challenges, we present a novel data processing pipeline for hand gesture recognition that combines advanced full-wave electromagnetic modelling and inversion with machine learning. In particular, the physical model accounts for the radar source, radar antennas, radar-target interactions and target itself, i.e.,, the hand in our case. To make this processing feasible, the hand is emulated by an equivalent infinite planar reflector, for which analytical Green’s functions exist. The apparent dielectric permittivity, which depends on the hand size, electric properties, and orientation, determines the wave reflection amplitude based on the distance from the hand to the radar. Through full-wave inversion of the radar data, the physical distance as well as this apparent permittivity are retrieved, thereby reducing by several orders of magnitude the dimension of the radar dataset, while keeping the essential information. Finally, the estimated distance and apparent permittivity as a function of gesture time are used to train the machine learning algorithm for gesture recognition. This physically-based dimension reduction enables the use of simple gesture recognition algorithms, such as template-matching recognizers, that can be trained in real time and provide competitive accuracy with only a few samples. We evaluate significant stages of our pipeline on a dataset of 16 gesture classes, with 5 templates per class, recorded with the Walabot, a lightweight, off-the-shelf array radar. We also compare these results with an ultra wideband radar made of a single horn antenna and lightweight vector network analyzer, and a Leap Motion Controller.

Eye movement analysis is a popular method to evaluate whether a user interface meets the users' requirements and abilities. However, with current tools, setting up a usability evaluation with an eye-tracker is resource-consuming, since the areas of interest are defined manually, exhaustively and redefined each time the user interface changes. This process is also error-prone, since eye movement data must be finely synchronised with user interface changes. These issues become more serious when the user interface layout changes dynamically in response to user actions. In addition, current tools do not allow easy integration into interactive applications, and opportunistic code must be written to link these tools to user interfaces. To address these shortcomings and to leverage the capabilities of eye-tracking, we present UsyBus, a communication framework for autonomous, tight coupling among reusable agents. These agents are responsible for collecting data from eye-trackers, analyzing eye movements, and managing communication with other modules of an interactive application. UsyBus allows multiple heterogeneous eye-trackers as input, provides multiple configurable outputs depending on the data to be exploited. Modules exchange data based on the UsyBus communication framework, thus creating a customizable multi-agent architecture. UsyBus application domains range from usability evaluation to gaze interaction applications design. Two case studies, composed of reusable modules from our portfolio, exemplify the implementation of the UsyBus framework.

Finger-based gesture input becomes a major interaction modality for surface computing. Due to the low precision of the finger and the variation in gesture production, multistroke gestures are still challenging to recognize in various setups. In this paper, we present µV, a multistroke gesture recognizer that addresses the properties of articulation, rotation, scaling, and translation invariance by combining $P+'s cloud-matching for articulation invariance with !FTL's local shape distance for RST-invariance. We evaluate µV against five competitive recognizers on MMG, an existing gesture set, and on two new versions for smartphones and tablets, MMG+ and RMMG+, a randomly rotated version on both platforms. µV is significantly more accurate than its predecessors when rotation invariance is required and not significantly inferior when it is not. µV is also significantly faster than others with many samples and not significantly slower with few samples.

The "Software as a Service" (SaaS) model of cloud computing popularized online multiuser collaborative software. Two famous examples of this class of software are Office 365 from Microsoft and Google Workspace. Cloud technology removes the need to install and update the software on end users' computers and provides the necessary underlying infrastructure for online collaboration. However, to provide a good end-user experience, cloud services require an infrastructure able to scale up to the task and allow low-latency interactions with a variety of users worldwide. This is a limiting factor for actors that do not possess such infrastructure. Unlike cloud computing which forgets the computational and interactional capabilities of end users' devices, the edge computing paradigm promises to exploit them as much as possible. To investigate the potential of edge computing over cloud computing, this paper presents a method for engineering interactive collaborative software supported by edge devices for the replacement of cloud computing resources. Our method is able to handle user interface aspects such as connection, execution, migration, and disconnection differently depending on the available technology. We exemplify our approach by developing a distributed Pictionary game deployed in two scenarios: a nonshared scenario where each participant interacts only with their own device and a shared scenario where participants also share a common device, including a TV. After a theoretical comparative study of edge vs. cloud computing, an experiment compares the two implementations to determine their effect on the end user's perceived experience and latency vs. real latency.

This paper presents SnappView, an open-source software development kit that facilitates end-user review of graphical user interfaces for mobile applications and streamlines their input into a continuous design life cycle. SnappView structures this user interface review process into four cumulative stages: (1) a developer creates a mobile application project with user interface code instrumented by only a few instructions governing SnappView and deploys the resulting application on an application store; (2) any tester, such as an end-user, a designer, a reviewer, while interacting with the instrumented user interface, shakes the mobile device to freeze and capture its screen and to provide insightful multimodal feedback such as textual comments, critics, suggestions, drawings by stroke gestures, voice or video records, with a level of importance; (3) the screenshot is captured with the application, browser, and status data and sent with the feedback to SnappView server; and (4) a designer then reviews collected and aggregated feedback data and passes them to the developer to address raised usability problems. Another cycle then initiates an iterative design. This paper presents the motivations and process for performing mobile application review based on SnappView. Based on this process, we deployed on the AppStore "WeTwo", a real-world mobile application to find various personal activities over a one-month period with 420 active users. This application served for a user experience evaluation conducted with N1=14 developers to reveal the advantages and shortcomings of the toolkit from a development point of view. The same application was also used in a usability evaluation conducted with N2=22 participants to reveal the advantages and shortcomings from an end-user viewpoint.

Despite the tremendous progress made for recognizing gestures acquired by various devices, such as the Leap Motion Controller, developing a gestural user interface based on such devices still induces a significant programming and software engineering effort before obtaining a running interactive application. To facilitate this development, we present QuantumLeap, a framework for engineering gestural user interfaces based on the Leap Motion Controller. Its pipeline software architecture can be parameterized to define a workflow among modules for acquiring gestures from the Leap Motion Controller, for segmenting them, recognizing them, and managing their mapping to functions of the application. To demonstrate its practical usage, we implement two gesture-based applications: an image viewer that allows healthcare workers to browse DICOM medical images of their patients without any hygiene issues commonly associated with touch user interfaces and a large-scale application for managing multimedia contents on wall screens. To evaluate the usability of QuantumLeap, seven participants took part in an experiment in which they used QuantumLeap to add a gestural interface to an existing application.

This paper presents theoretical and empirical results about user-defined gesture preferences for squeezable objects by focusing on a particular object: a deformable cushion. We start with a theoretical analysis of potential gestures for this squeezable object by defining a multi-dimension taxonomy of squeeze gestures composed of 82 gesture classes. We then empirically analyze the results of a gesture elicitation study resulting in a set of N=32 participants X 21 referents = 672 elicited gestures, further classified into 26 gesture classes. We also contribute to the practice of gesture elicitation studies by explaining why we started from a theoretical analysis (by systematically exploring a design space of potential squeeze gestures) to end up with an empirical analysis (by conducting a gesture elicitation study afterward): the intersection of the results from these sources confirm or disconfirm consensus gestures. Based on these findings, we extract from the taxonomy a subset of recommended gestures that give rise to design implications for gesture interaction with squeezable objects.

Arm-and-hand tracking by technological means allows gathering data that can be elaborated for determining gesture meaning. To this aim, machine learning (ML) algorithms have been mostly investigated looking for a balance between the highest recognition rate and the lowest recognition time. However, this balance comes mainly from statistical models, which are challenging to interpret. In contrast, we present$\mu C^{1}$and$\mu C^{2}$, two geometric model-based approaches to gesture recognition which support the visualization and geometrical interpretation of the recognition process. We compare$\mu C^{1}$and$\mu C^{2}$with respect to two classical ML algorithms, k-nearest neighbor (k-NN) and support vector machine (SVM), and two state-of-the-art (SotA) deep learning (DL) models, bidirectional long short-term memory (BiLSTM) and gated recurrent unit (GRU), on an experimental dataset of ten gesture classes from the Italian Sign Language (LIS), each repeated 100 times by five inexperienced non-native signers, and gathered with wearable technology (a sensory glove and inertial measurement units). As a result, we achieve a compromise between high recognition rates ($>90\%$) and low recognition times ($ < 0.1 {\mathrm{ s}}$) that is adequate for human–computer interaction. Moreover, we elaborate on the algorithms’ geometric interpretation based on geometric algebra, which supports some understanding of the recognition process.

The visual quality of graphical user interfaces can be estimated by software measurement, which consists of measuring visual design formulas on a dataset of interfaces and interpreting them for improving their overall quality. When performed manually, this process becomes very tedious and error prone, especially for large datasets. When performed with existing software, this process is accelerated, but tied to a particular set of measures with their own interpretation, making them inflexible. To overcome these shortcomings, GUIMETRICS improves this process by automatically collecting screenshots in various platform configurations and resolutions and automatically computing and interpreting measures on-demand. The cloud-based architecture of GUIMETRICS can be extended with external modules for computing any visual measure, even in different programming languages, thus making it more flexible.

Personal drones are more and more present in our lives and acting as “flying cameras” is one of their most prominent applications. In this work, we conduct a synopsis of the scientific literature on human-drone interaction to identify system functions and corresponding commands for controlling drone-based aerial photography and video, from which we compile a dictionary of interactions. We also discuss opportunities for more research at the intersection of drone computing, augmented vision, and personal photography.

Adapting the user interface (UI) to the changing context of use is intended to support the interaction effectiveness and sustain UI usability. However, designing and/or processing UIs adaptation at design time does not encompass real situation requirements. Adaptation should have a cross-cutting and low-cost impact on software patterning and appearance with regard to the situation and the ambient-context. To meet this requirement, we present TADAP proposal for run-time adaptive and adaptable UI based user feedbacks and machine learning. It allows a task-driven adaptation of the user interface (UI) at runtime by mixed-initiative. The particularity of TADAP is the utilization of Machine Learning potential to support context-aware runtime adaptation within model-driven UI. Further, TADAP allows the UI adaptation by mixed-initiative (User and System) considering the user preferences (implicit and explicit) during an interaction. Such a mixed-initiative runtime UI-adaptation tool provides recommendations on how to personalize the UI. Further, it has the ability to track real-time users’ interventions and learn their preferences. Diverse tests were performed and showed TADAP as a promising initiative for intelligent model-driven UI adaptation.

We introduce “4E,” a new design approach of reconfigurable displays that can change their form factors by capitalizing on four quality properties inspired by applied material: extensibility, extendability, expandability, and extractability. This approach is applicable to both fixed and portable displays. We define and exemplify each property, highlighting the key differences in how reconfigurable displays can change their form factors to accommodate more screen real estate for more users, applications, and functionality. To demonstrate the 4E approach, we conduct a targeted literature review and introduce E3Screen, a prototype that enhances any flat screen (e.g., of a tablet, laptop, monitor) with two slidable, rotatable, and foldable lateral displays. We report results from a controlled experiment with N=103 participants, conducted to collect, analyze, and understand end users’ preferences for display configurations permitted by the extendability, expandability, and extractability of E3Screen. Our results structure future research and development in reconfigurable displays and multi-display collaborative workspaces.

With the continuously increasing number and variety of devices, the study of visual design of their Graphical User Interfaces grows in importance and scope, particularly for new devices, including smartphones, tablets, and large screens. Conducting a visual design experiment typically requires defining and building a GUI dataset with different resolutions for different devices, computing visual design measures for the various configurations, and analyzing their results. This workflow is very time- and resource-consuming, therefore limiting its reproducibility. To address this problem, we present UiLab, a cloud-based workbench that parameterizes the settings for conducting an experiment on visual design of Graphical User Interfaces, for facilitating the design of such experiments by automating some workflow stages, and for fostering their reproduction by automating their deployment. Based on requirements elicited for UiLab, we define its conceptual model to delineate the borders of services of the software architecture to support the new workflow. We exemplify it by demonstrating a system walkthrough and we assess its impact on experiment reproducibility in terms of design and development time saved with respect to a classical workflow. Finally, we discuss potential benefits brought by this workbench with respect to reproducing experiments in GUI visual design and existing shortcomings to initiate future avenues. We publicly release UiLab source code on a GitHub repository.

Intra-platform plasticity regularly assumes that the display of a computing platform remains fixed and rigid during interactions with the platform in contrast to reconfigurable displays, which can change form depending on the context of use. In this paper, we present a model-based approach for designing and deploying graphical user interfaces that support intra-platform plasticity for reconfigurable displays. We instantiate the model for E3Screen, a new device that expands a conventional laptop with two slidable, rotatable, and foldable lateral displays, enabling slidable user interfaces. Based on a UML class diagram as a domain model and a SCRUD list as a task model, we define an abstract user interface as interaction units with a corresponding master-detail design pattern. We then map the abstract user interface to a concrete user interface by applying rules for the reconfiguration, concrete interaction, unit allocation, and widget selection and implement it in JavaScript. In a first experiment, we determine display configurations most preferred by users, which we organize in the form of a state-transition diagram. In a second experiment, we address reconfiguration rules and widget selection rules. A third experiment provides insights into the impact of the lateral displays on a visual search task.

Abstract Adapting the user interface of a software system to the requirements of the context of use continues to be a major challenge, particularly when users become more demanding in terms of adaptation quality. A considerable number of methods have, over the past three decades, provided some form of modelling with which to support user interface adaptation. There is, however, a crucial issue as regards in analysing the concepts, the underlying knowledge, and the user experience afforded by these methods as regards comparing their benefits and shortcomings. These methods are so numerous that positioning a new method in the state of the art is challenging. This paper, therefore, defines a conceptual reference framework for intelligent user interface adaptation containing a set of conceptual adaptation properties that are useful for model-based user interface adaptation. The objective of this set of properties is to understand any method, to compare various methods and to generate new ideas for adaptation. We also analyse the opportunities that machine learning techniques could provide for data processing and analysis in this context, and identify some open challenges in order to guarantee an appropriate user experience for end-users. The relevant literature and our experience in research and industrial collaboration have been used as the basis on which to propose future directions in which these challenges can be addressed.

Gesture elicitation studies represent a popular and resourceful method in HCI to inform the design of intuitive gesture commands, reflective of end-users' behavior, for controlling all kinds of interactive devices, applications, and systems. In the last ten years, an impressive body of work has been published on this topic, disseminating useful design knowledge regarding users' preferences for finger, hand, wrist, arm, head, leg, foot, and whole-body gestures. In this paper, we deliver a systematic literature review of this large body of work by summarizing the characteristics and findings ofN=216gesture elicitation studies subsuming 5,458 participants, 3,625 referents, and 148,340 elicited gestures. We highlight the descriptive, comparative, and generative virtues of our examination to provide practitioners with an effective method to (i) understand how new gesture elicitation studies position in the literature; (ii) compare studies from different authors; and (iii) identify opportunities for new research. We make our large corpus of papers accessible online as a Zotero group library at https://www.zotero.org/groups/2132650/gesture_elicitation_studies.

Visual interfaces are important in human motion to capture it, to visualize it, and to facilitate motion-based interactive systems. This workshop aims at providing a platform for researchers, designers and users to discuss the challenges related to the design of visual interfaces for motion-based interaction, in terms of visualization (e.g. graphical user interface, multimodal feedback, evaluation) and processing (e.g., data collection, treatment, interpretation, recognition) of human movement (e.g., motor skills, amplitude of movements, limitations). We will share experiences, lessons learned and elaborate tools for developing all the possible applications going forward.

We address in this work visual menus for smartglasses by proposing a wide range of design options, such as shape, location, orientation, and presentation modalities, which we compile in a design space with eight dimensions. From this design space, we select a subset of fourteen 2-D menus, for which we collect users’ preferences during a vignette experiment with N=251 participants. We report numerical measures of absolute, relative, and aggregate preference for smartglasses menus, and employ a particular Thurstone’s pairwise comparison technique, the Bradley-Terry model, to evaluate menu designs. Our results highlight key variables influencing users’ preferences regarding the visual appearance of smartglasses menus, which we use to discuss opportunities for future work.

The area of Engineering Interactive Computing Systems (EICS) is defined as the intersection between Software Engineering (SE) and Human-Computer Interaction (HCI). In this paper, we provide an overview of what EICS is and how it is positioned with respect to other venues in HCI, such as CHI, UIST, and IUI, highlighting its legacy and paying homage to past scientific events from which EICS emerged. We also take this opportunity to deliver a comparative analysis of the past, present, and perhaps future research and development questions raised in EICS by analyzing the EICS papers published from its first edition to this one, based on global and chronological word clouds, phrase nets, and topic modelling.

Touch-surface multi-stroke gestures, as well as freehand drawings, are typically acquired by devices and sensors as a suite of timestamped points on a plane. This Cartesian coordinate system, although useful for computation like complexity analysis, gesture classification and recognition, becomes complex and inefficient when gestures need to be visualized and directly manipulated for editing. To address these challenges, a new mathematical representation of these gestures via a Bézier curve is defined to initiate a model-based approach for gesture direct manipulation (e.g., cut, copy, paste, translate, scale, rotate, deform, crop, compose, decompose). SketchADoodle, an Android-based mobile application for drawing, gesturing, demonstrates how the pseudo-code of the Bézier-based operations are engineered for real-time direct manipulation. We release the programming code for further development of gesture-based user interfaces based on Bézier curves.

Whilst new patents and announcements advertise the technical availability of foldable displays, which are capable to be folded to some extent, there is still a lack of fundamental and applied understanding of how to model, to design, and to prototype graphical user interfaces for these devices before actually implementing them. Without waiting for their off-the-shelf availability and without being tied to any physical foldable mechanism, Flecto defines a model, an associated notation, and a supporting software for prototyping graphical user interfaces running on foldable displays, such as foldable smartphone or assemblies of foldable surfaces. For this purpose, we use an extended notation of the Yoshizawa-Randlett diagramming system, used to describe the folds of origami models, to characterize a foldable display and define possible interactive actions based on its folding operations. A guiding method for rapidly prototyping foldable user interfaces is devised and supported by Flecto, a design environment where foldable user interfaces are simulated in 3D environment instead of in physical reality. We report on a case study to demonstrate Flecto in action and we gather the feedback from users on Flecto, using Microsoft Product Reaction Cards.

While end users can acquire full 3D gestures with many input devices, they often capture only 3D trajectories, which are 3D uni-path, uni-stroke single-point gestures performed in thin air. Such trajectories with their $(x,y,z)$ coordinates could be interpreted as three 2D stroke gestures projected on three planes,\ie, $XY$, $YZ$, and $ZX$, thus making them admissible for established 2D stroke gesture recognizers. To investigate whether 3D trajectories could be effectively and efficiently recognized, four 2D stroke gesture recognizers, \ie, \$P, \$P+, \$Q, and Rubine, are extended to the third dimension: $\$P^3$, $\$P+^3$, $\$Q^3$, and Rubine-Sheng, an extension of Rubine for 3D with more features. Two new variations are also introduced: $\F for flexible cloud matching and FreeHandUni for uni-path recognition. Rubine3D, another extension of Rubine for 3D which projects the 3D gesture on three orthogonal planes, is also included. These seven recognizers are compared against three challenging datasets containing 3D trajectories, \ie, SHREC2019 and 3DTCGS, in a user-independent scenario, and 3DMadLabSD with its four domains, in both user-dependent and user-independent scenarios, with varying number of templates and sampling. Individual recognition rates and execution times per dataset and aggregated ones on all datasets show a highly significant difference of $\$P+^3$ over its competitors. The potential effects of the dataset, the number of templates, and the sampling are also studied.

Graphical Adaptive Menus are Graphical User Interface menus whose predicted items of immediate use can be automatically rendered in a prediction window. Rendering this prediction window is a key question for adaptivity to enable the end-user to efficiently differentiate predicted items from normal ones and to consequently select appropriate items. Adaptivity for graphical menus has been investigated more for normal screens, such as desktops, than for small screens, such as smartphones, where real estate imposes severe rendering constraints. To address this question, this article defines and explores a design space where graphical adaptive menus are structured based on Bertin’s eight visual variables (i.e., position, size, shape, value, color, orientation, texture, and motion) and their combination by comparing their rendering for small screens with respect to normal screens. Based on this design space, previously introduced graphical adaptive menus are revisited in terms of four stability properties (i.e., spatial, physical, format, and temporal), and new menu designs are introduced and discussed for both normal and small screens. The resulting set of graphical adaptive menu has been subject to a preference analysis from which a particular design emerged: the cloud menu, where predicted items are arranged in an adaptive tag cloud. We investigate empirically the effect of the cloud menu on the item selection time and the error rate with respect to a static menu and an adaptive linear menu. This article then suggests a set of usability guidelines for designers and practitioners to design graphical adaptive menus in general and cloud menus in particular.

Welcome to RCIS, the “Research Challenges in Information Science” community and its annual conference! In this short article, we introduce newcomers to this field and to its community with an overview of what RCIS is and how it positions with respect to other venues existing in the domain, like CAiSE, MODEL, and ER, highlighting its legacy. We also take this opportunity to enumerate and define the eight major RCIS areas of scientific contributions and exemplify them with some representative papers published during this 13thedition or before, which we hope to guide researchers and practitioners towards making their future RCIS submissions successful and impactful in the RCIS community.

Welcome to EICS, the Engineering Interactive Computing Systems community, PACMHCI/EICS journal, and annual conference! In this short article, we introduce newcomers to the field and to our community with an overview of what EICS is and how it positions with respect to other venues in Human-Computer Interaction, such as CHI, UIST, and IUI, highlighting its legacy and paying homage to past scientific events from which EICS emerged. We also take this opportunity to enumerate and exemplify scientific contributions to the field of Engineering Interactive Computing Systems, which we hope to guide researchers and practitioners towards making their future PACMHCI/EICS submissions successful and impactful in the EICS community.

A gesture elicitation study, as originally defined, consists of gathering a sample of participants in a room, instructing them to produce gestures they would use for a particular set of tasks, materialized through a representation called referent, and asking them to fill in a series of tests, questionnaires, and feedback forms. Until now, this procedure is conducted manually in a single, physical, and synchronous setup. To relax the constraints imposed by this manual procedure and to support stakeholders in defining and conducting such studies in multiple contexts of use, this paper presents Gelicit, a cloud computing platform that supports gesture elicitation studies distributed in time and space structured into six stages: (1) define a study: a designer defines a set of tasks with their referents for eliciting gestures and specifies an experimental protocol by parameterizing its settings; (2) conduct a study: any participant receiving the invitation to join the study conducts the experiment anywhere, anytime, anyhow, by eliciting gestures and filling forms; (3) classify gestures: an experimenter classifies elicited gestures according to selected criteria and a vocabulary; (4) measure gestures: an experimenter computes gesture measures, like agreement, frequency, to understand their configuration; (5) discuss gestures: a designer discusses resulting gestures with the participants to reach a consensus; (6) export gestures: the consensus set of gestures resulting from the discussion is exported to be used with a gesture recognizer. The paper discusses Gelicit advantages and limitations with respect to three main contributions: as a conceptual model for gesture management, as a method for distributed gesture elicitation based on this model, and as a cloud computing platform supporting this distributed elicitation. We illustrate Gelicit through a study for eliciting 2D gestures executing Internet of Things tasks on a smartphone.

This paper presents MoCaDiX, a method for designing cross-device graphical user interfaces of an information system based on its UML class diagram, structured as a four-step process: (1) a UML class diagram of the information system is created in a model editor, (2) how the classes, attributes, methods, and relationships of this class diagram are presented across devices is then decided based on user interface patterns with their own parametrization, (3) based on these parameters, a Concrete User Interface model is generated in QuiXML, a lightweight fit-to-purpose User Interface Description Language, and (4) based on this model, HTML5 cross-device user interfaces are semi-automatically generated for four configurations: single/multipledevice single/multiple-display on a smartphone, a tablet, and a desktop. From the practitioners' viewpoint, a first experiment investigates effectiveness, efficiency, and subjective satisfaction of three intermediate and three expert designers, using MoCaDiX on a representative class diagram. From the end user's viewpoint, a second experiment compares subjective satisfaction and preference of twenty end users assessing layout strategies for interfaces generated on two devices.

We introduce AB4Web, a web-based engine that implements a balanced randomized version of the multivariate A/B testing, specifically designed for practitioners to readily compare end-users' preferences for user interface alternatives, such as menu layouts, widgets, controls, forms, or visual input commands. AB4Web automatically generates a balanced set of randomized pairs from a pool of user interface design alternatives, presents them to participants, collects their preferences, and reports results from the perspective of four quantitative measures: the number of presentations, the preference percentage, the latent score of preference, and the matrix of preferences. In this paper, we exemplify the AB4Web tester with a user study for which N=108 participants expressed their preferences regarding the visual design of 49 distinct graphical adaptive menus, with a total number of 5,400 preference votes. We compare the results obtained from our quantitative measures with four alternative methods: Condorcet, de Borda count starting at one and zero, and the Dowdall scoring system. We plan to release AB4Web as a public tool for practitioners to create their own A/B testing experiments.

We present empirical results about users' gesture preferences for smart rings by analyzing 672 gestures from 24 participants. We report an overall low consensus (mean .112, maximum .225 on the unit scale) between participants' gesture proposals, and we point to the challenges of designing highly-generalizable ring gestures across users. We also contribute to the practice of gesture elicitation studies by discussing how a priori conditions (e.g., participants' traits, such as creativity and motor skills), commitment and behavior during the experiment (e.g., their thinking times), but also a posteriori aspects (the experimenter's choice of criteria to group gestures into categories) affect agreement. We offer design guidelines for ring gestures informed by our empirical observations, and present a collection of gestures reflective of our participants' mental models for effecting commands using smart rings.

Nearest neighbor classifiers recognize stroke gestures by computing a (dis)similarity between a candidate gesture and a training set based on points, which may require normalization, resampling, and rotation to a reference before processing. To eliminate this expensive preprocessing, this paper introduces a vector-between-vectors recognition where a gesture is defined by a vector based on geometric algebra and performs recognition by computing a novel Local Shape Distance (LSD) between vectors. We mathematically prove the LSD position, scale, and rotation invariance, thus eliminating the preprocessing. To demonstrate the viability of this approach, we instantiate LSD for n=2 to compare !FTL, a 2D stroke-gesture recognizer with respect to $1 and $P, two state-of-the-art gesture recognizers, on a gesture set typically used for benchmarking. !FTL benefits from a recognition rate similar to $P, but a significant smaller execution time and a lower algorithmic complexity.

This paper presents Cloud Menus, a split adaptive menu for small screens where the predicted menu items are arranged in a circular tag cloud with a location consistent with their corresponding position in the static menu and a font size depending on their prediction level. This layout results from a 3-step design process: (i) defining an initial design space on Bertin's 8 visual variables and 4 quality properties, (ii) identifying the most preferred layout based on agreement rate, and (iii) implementing it into Cloud Menus, a new widget for Android with circular layout. An empirical study suggests that cloud menus reduce item selection time and error rate when prediction is correct without penalizing it when prediction is incorrect, compared to two baselines: a non-adaptive static menu and an adaptive linear menu. From this study, design guidelines for cloud menus are elaborated.

This paper presents a design space for engineering adaptive user interfaces throughout the user interface development life cycle in order to describe any adaptation technique, adaptable or adaptive, to compare two or more techniques, and to generate new, perhaps unprecedented, techniques. Grounded in the theory of psychological perception, this design space structures the adaptation life cycle into two regulation loops between the user and the system: a perception-decision-action (PDA) loop for both the system and the user, and a learning-prediction-adaptation (LPA) for supporting the adaptation, this last being particularly expressive for adaptivity. This PDA-LPA design space enables defining properties for assessing the quality of these loops between the system and the end-user. This design space of is instantiated on two advanced adaptive user interfaces: adaptive user interfaces based on machine learning and adaptive layouts. This design space provides new insights for considering adaptivity design options.

This paper presents a model-based approach for designing Polymodal Menus, a new type of multimodal adaptive menu for small screen graphical user interfaces where item selection and adaptivity are responsive to more than one interaction modality: a menu item can be selected graphically, tactilely, vocally, gesturally, or any combination of them. The prediction window containing the most predicted menu items by assignment, equivalence, or redundancy is made equally adaptive. For this purpose, an adaptive menu model maintains the most predictable menu items according to various prediction methods. This model is exploited throughout various steps defined on a new Adaptivity Design Space based on a Perception-Decision-Action cycle com-ing from cognitive psychology. A user experiment compares four conditions of Polymodal Menus (graphical, vocal, gestural, and mixed) in terms of menu selection time, error rate, user subjective satisfaction and user preference, when item prediction has a low or high level of accuracy. Polymodal Menus offer alternative input/output modalities to select menu items in various contexts of use, especially when graphical modality is constrained.

Information systems become more accessible as a service offered to anybody, anywhere, at any time, via almost any device and computing platform. The continuous growth and the heterogeneity of these devices induce diverse user experiences depending on the device and challenge designers to creating methods and tools for engineering usable, yet accessible, information systems. Instead of repeating a similar development life cycle, design patterns concentrate design solutions with embedded usability and accessibility. Once a pattern is selected, the developer is responsible for adequately program the pattern code, which is a tedious and error-prone task. In order to address these challenges, this paper presents UIPLML (User Interface Pattern Language Markup Language), a XML-compliant markup language for defining user interface patterns for multiple contexts of use, e.g., for different users carrying out a task on different devices in different environments. A meta-model with new expressiveness enables multi-facet pattern matching. To validate it, four UIPLML pattern databases have been created: a base of 237 entries for multi-platform systems, a base of 42 entries for context-aware interfaces, a base of 10 entries for culturally-aware interfaces, and a base of 52 entries for accessibility. The master/detail pattern is in particular supported by a software for generative pattern-based approach where application parameters and contextual data govern automated user interface XML creation which, in turns, generates code for multi-platform information systems.

An effective User Interface (UI) is a key success factor for interactive systems. Hence, particular attention should be paid to the UI design during the Requirement Engineering process (RE). Several RE tools have been proposed in order to support the UI design. However, these tools have limitations in terms of requirements completeness, requirements quality analysis and UI generation from requirements. In this paper, we present a new RE toolkit called QualIHM, that deals with the limitations of the existing RE. The toolkit supports the description of requirements in different formats. In addition, QualiHM facilitates the UI design by transforming requirement formats from one to another, generating the UI code and providing feedback about the aesthetic of the UI.

The graphical user interface (GUI) of an interactive system is nowadays the most frequently used interaction modality. While the contents are of high importance, the Look and Feel is an equally essential factor determining the GUI quality that is impacted by several determinants such as but not limited to aesthetics, pleasurability, fun, etc. Therefore, GUIs aesthetics is a potential element to focus on in order to facilitate communication between device and user. On that basis, one question that comes up is: “Is it possible to evaluate the quality of a GUI by estimating its aesthetics through a series of measurable geometric metrics?”. This paper suggests possible directions to address the previous question by, first, introducing a simplifying model of GUIs aesthetics that captures aesthetics aspects and regions-related metrics. In a second phase, a methodology for the evaluation of GUIs aesthetics is defined based on the underlying model. The paper finally puts forwards a model-based implementation of the aforementioned methodology in the form of a web service tool for metrics-based evaluation of GUIs and discuss the results of a survey on users aesthetics perceptions.

Designing accessible web sites and more generally Internet-connected devices remains a challenging task nowadays. A number of guidelines (such as the WCAG2) are now widely available and recognised. To better cope with the quickly evolving technological landscape, these guidelines are also being formulated in technology-neutral terms. However this is still leaving the user dimension largely implicit, which makes it difficult to understand exactly which kind of user a given website is hindering. This paper describes how to capture and use rational links between guidelines and user capabilities/impairments by combining a set of complementary models (user, task, user interface, guidelines). The process of building those accessibility rationales relies upon available user and guidelines ontologies and also on obstacle identification and resolution techniques borrowed from the requirements engineering domain. This resulting enriched guidance enables a number of interesting new scenarios to better help web developers, analyse guidelines or make comparisons between guidelines.

In order to address challenges posed by different users conducting their interactive tasks on heterogeneous platforms and devices in various environments, this paper provides a computational framework to support the adaptation of the user interface of interactive systems. This framework consists of: a meta-model for understanding fundamental concepts required by adaptation, a reference framework for characterizing seven dimensions for conducting adaptation based on the meta-model, and a design space for consistently assessing the adaptation coverage. In this way, development phases are considered with a standard approach, a unified terminology, and an extensive catalog of techniques.

This paper addresses the problem of modernizing graphical user interfaces of interactive applications by re-engineering their resource files in four phases: resource decompilation, which extracts resource files from the executable code of an interactive application; modeling the source user interface, which transforms extracted resources into a resource model; resource to user interface transformation, which transforms the resource model into a Concrete User Interface model, and forward engineering, which offers two alternatives: after editing the user interface model, a new interface could be generated or recompiled into a resource to be incorporated back. The paper motivates and details this re-engineering approach by focusing on methods and algorithms implemented in UsiResourcer, a software tool that reverse engineers MS Windows resource files into a Concrete User Interface Model for further process. A discussion on the generalization of the approach is also provided.

User interface design consists of a collaborative activity where various stakeholders can all sketch the future interactive system at different levels of fidelity on different devices and computing platforms. User interfaces sketches are also intended to support prototypes on multiple computing platforms and operating systems that all impose their own capabilities and constraints. In order to support the needs for user interface design by sketching, this paper describes Gambit, a multi-platform system that provides a light-weight approach for prototyping graphical user interfaces by sketching with HTML5. The paper reports on an experiment for the most preferred platform/devices for three primary sketching functions by designers and developers in a Gambit-supported session.

User interface development life cycle often involve several different views of the user interface over time either at the same level of abstraction or at different levels of abstraction. The relationship between these different views is often supported by tiling coordinated windows containing these related views simultaneously, thus leaving the developer with the responsibility to effectively and efficiently link the corresponding elements of these different views. This paper attempts to overcome the shortcomings posed by the coordinated visualization of multiple views by providing UsiView, a user interface rendering engine in which one single window ensures an animated transition between these different user interface views dynamically an internal view, an external view, and a conceptual view. Examples include the following cases: an authoring environment ensures an animated transition between an internal view (e.g., HTML5) and its external view (e. g., a web page), an Integrated Development Environment ensures an animated transition between its conceptual view and its external view; a model-driven engineering environment ensures an animated transition between the conceptual view at different levels of abstraction, e.g., from task to abstract user interface to concrete user interface until final user interface. The paper discusses the potential advantages of using animated transitions between user interface views during the development life cycle.

Several algorithms have been developed for pen-based gesture recognition. Yet, their integration in streamline engineering of interactive systems is bound to several shortcomings: they are hard to compare to each other, determining which one is the most suitable in which situation is a research problem, their performance largely vary depending on contextual parameters that are hard to predict, their fine-tuning in a real interactive application is a challenge. In order to address these shortcomings, we developed UsiGesture, an engineering method and a software support platform that accommodates multiple algorithms for pen-based gesture recognition in order to integrate them in a straightforward way into interactive computing systems. The method is aimed at providing designers and developers with support for the following steps: defining a dataset that is appropriate for an interactive system (e.g., made of commands, symbols, characters), determining the most suitable gesture recognition algorithm depending on contextual variables (e.g., user, platform, environment), fine-tuning the parameters of this algorithm by multi-criteria optimization (e.g., system speed and recognition rate vs. human distinguishability and perception), and incorporation of the fine-tuned algorithm in an integrated development environment for engineering interactive systems.

This paper introduces, motivates, defines, and exemplifies the concept of distribution graph as a way for modelling and developing Distributed User Interfaces of interactive systems. A distribution graph consists of a state chart model enriched as follows: states represent individual states of entities involved in the distribution as well as a collective representation of their synchronization; transitions are represented by event-condition-actions where the action part consists of a distribution script. A distribution script expresses the distribution behaviour based on distribution primitives. These primitives are basic operations that manipulate parts or wholes of user interface for distribution at run-time. These primitives are themselves implemented on top of an environment for distributed computing that is implemented for four major computing platforms (i.e., Microsoft Windows, Mac OS X, Linux, and Mobile Linux). Thanks to the capabilities provided by this environment, the user interfaces belonging to these distributed systems can be run indifferently on any of these computing platforms. This paper defines the new concepts introduced for this purpose, i.e., distribution primitive, distribution script, and distribution graph, and demonstrates how they can effectively support distributed user interfaces.

Animated transitions are widely used in many different domains of human activity, ranging from cartoons and movies to computer science for powerfully conveying a message more effectively and efficiently about a phenomenon of interest. This paper reviews a series of techniques for defining, analyzing, and exploiting animated transitions in different types of interactive information systems. A general conceptual model is provided that explicitly links a model of an interactive information system, its model elements and relationships to animated transitions in order to adequately reflect any change of the model into animated transitions. Two instantiations of this conceptual framework are discussed: animated transitions for representing adaptation of the graphical user interface of an interactive system, along with its implementation; and animated transitions between user interface views during development life cycle.

This article aims to investigate user preferences for extra-user interfaces (extra-UI), formerly known as meta-user interfaces. These are special user interfaces that allow the user to control or personalize the application's user interface. They are named “extra-user interfaces” because the application does not need it to work. Their main goal is the support of systems with several contexts, displays, devices, or platforms. The purposes and features offered by all the Extra-UI vary from one application to another. To create coherence between them, we defined a catalogue of 14 distribution primitives that are typically provided and classified into 4 categories: simple primitives, basic primitives, advanced primitives, and management operations. Based on this catalogue, a comparative analysis of the state of the art was conducted to identify which interaction styles have been properly used and to discuss the rationale behind these usages. From this analysis, we set up and conducted a comparative evaluation of user preferences by 14 participants testing 6 selected distribution primitives in 4 different interaction styles. The research outcomes exemplified that there were significant differences on user preferences between interaction styles with regards to experience level and primitive type.

Nowadays several Computer-Aided Software Engineering environments exploit Model-Driven Engineering (MDE) techniques in order to generate a single user interface for a given computing platform or multi-platform user interfaces for several computing platforms simultaneously. Therefore, there is a need to assess the usability of those generated user interfaces, either taken in isolation or compared to each other. This paper describes an MDE approach that generates multi-platform graphical user interfaces (e.g., desktop, web) that will be subject to an exploratory controlled experiment. The usability of user interfaces generated for the two mentioned platforms and used on multiple display devices (i.e., standard size, large, and small screens) has been examined in terms of satisfaction, effectiveness and efficiency. An experiment with a factorial design for repeated measures was conducted for 31 participants, i.e., postgraduate students and professors selected by convenience sampling. The data were collected with the help of questionnaires and forms and were analyzed using parametric and non-parametric tests such as ANOVA with repeated measures and Friedman's test, respectively. Efficiency was significantly better in large screens than in small ones as well as in the desktop platform rather than in the web platform, with a confidence level of 95%. The experiment also suggests that satisfaction tends to be better in standard size screens than in small ones. The results suggest that the tested MDE approach should incorporate enhancements in its multi-device/platform user interface generation process in order to improve its generated usability.

Given its current state of the art, model-based UI development (MBDUI) is able to fulfill the major requirements of desktop and mobile applications, such as form-based user interfaces that adapt to the actual context of use. More recent research deals with the development of 3D interactive multimodal environments. Though user-centered design is more and more driving the design of these environments, less attention is devoted to the development processes than to interactive tools supporting isolated phases in the realization process. In this paper we present an attempt to structure an approach to support 3DUIs development by introducing a MBDUI compliant method. The development method is articulated on three axes: models and their specification language, approach, and tools that support the method based on the underlying models.

Fusion engines are fundamental components of multimodal inter-active systems, to interpret input streams whose meaning can vary according to the context, task, user and time. Other surveys have considered multimodal interactive systems; we focus more closely on the design, specification, construction and evaluation of fusion engines. We first introduce some terminology and set out the major challenges that fusion engines propose to solve. A history of past work in the field of fusion engines is then presented using the BRETAM model. These approaches to fusion are then classified. The classification considers the types of application, the fusion principles and the temporal aspects. Finally, the challenges for future work in the field of fusion engines are set out. These include software frameworks, quantitative evaluation, machine learning and adaptation.

This research aims to propose a multi-modal fusion framework for high-level data fusion between two or more modalities. It takes as input low level features extracted from different system devices, analyses and identifies intrinsic meanings in these data. Extracted meanings are mutually compared to identify complementarities, ambiguities and inconsistencies to better understand the user intention when interacting with the system. The whole fusion life cycle will be described and evaluated in an office environment scenario, where two co-workers interact by voice and movements, which might show their intentions. The fusion in this case is focusing on combining modalities for capturing a context to enhance the user experience.

Over the last few decades, several usability knowledge-based systems have been developed to provide user interface designers with usability knowledge (e.g. heuristics, usability guidelines, standards). Such systems are intended to assist designers during the design process, and to improve the usability of the user interface being designed. However, the assumption that such systems actually improve the usability of the resulting user interface remains to be demonstrated: virtually no systems have been empirically tested by designers who create products. In order to confirm this assumption, we conducted an experimental study in which professional web designers had to create webpages, either using a knowledge-based system – MetroWeb – or without it. This study was intended to determine the influence of MetroWeb on the professional web designers' cognitive activity, and to find out whether MetroWeb actually assists them to develop a user-centred design. The results show that the web designers did not very often use MetroWeb while designing webpages; however, rather surprisingly, the designers who did use MetroWeb more often exhibited a user-centred activity than those working without MetroWeb. We conclude this paper by discussing these findings, and suggesting future possible ways of research intended to assist designers to adopt a user-centred approach to their activity.

To overcome the shortcomings posed by audio rendering of web pages for blind users, this paper implements an interaction technique where web pages are parsed so as to automatically generate a virtual reality scene that is augmented with a haptic feedback. All elements of a web page are transformed into a corresponding "hapget" (haptically-enhanced widget), a three dimensional widget exhibiting a behavior that is consistent with their web counterpart and having haptic extension governed by usability guidelines for haptic interaction. A set of implemented hapgets is described and used in some examples. All hapgets introduced an extension to UsiXML, a XML-compliant User Interface Description Language that fosters model-driven engineering of user interfaces. In this way, it possible to render any UsiXML-compatible user interface thanks to the interaction technique described, and not only web pages.

A pluripotent information system is an open and distributed information system that (i) automatically adapts at runtime to changing operating conditions, and (ii) satisfies both the requirements anticipated at development time, and those unanticipated before but relevant at runtime. Engineering pluripotency into an information system therefore responds to two recurring critical issues: (i) the need for adaptability given the uncertainty in a systempsilas operating environment, and (ii) the difficulty to fully anticipate and account for all possible stakeholderspsila requirements at development time and respond to the change of requirements at runtime. We draw on our grouppsilas research efforts over the last two years to show and discuss how pluripotency can be engineered into information systems.

AudioCubes is a novel tangible user interface allowing any person interested by sound design such as sound creators, and music trainers to intuitively explore and create dynamically changing sound. A new sound is created by manipulating distributed cube tangible user interface that can be coupled wirelessly by locating them in the interaction range of each other on a table. At any time, a sound processing network combines operational properties of AudioCubes, such as location on a plane or in space, movement, arrangement with other cubes, and layout. Sound algorithm parameters and the configuration of the sound processing network can be changed simultaneously, allowing a fast and convenient exploration of sound creation space that creates a new interaction technique for creating sounds.

This paper presents a series of new algorithms for paginating interaction spaces (i.e.; windows, dialog boxes, web pages...) based on a multi-layer specification in a user interface description language. We first describe how an interaction space can be split using information from the presentation layer (Concrete User Interface). We then demonstrate how information from higher levels of abstraction (Abstract User Interface, Task model) can be used to produce a pagination that is more meaningful from the task's viewpoint than other techniques. The pagination relies on a set of explicit splitting rules that can be applied as the first step in a graceful degradation. These splitting rules are implemented as an interface builder plug-in which automatically generates code under the designer's control.

This paper addresses the problem of the graceful degradation of user interfaces where an initial interface is transferred to a smaller platform. It presents a technique for pagination of interaction spaces (e.g., windows, dialog boxes, web pages) based on a multi-layer specification in the user interface description language UsiXML. We first describe how an interaction space can be split using information from the presentation layer (Concrete User Interface). We then show how information from higher abstraction levels (Abstract user Interface, Task model) can be used to refine the process. This technique belongs to a collection of transformation rules that have been developed to adapt a user interface to smaller, more constrained displays.

From a topological model of a working environment, MigriXML automatically generates a virtual reality environment for controlling the run-time migration of a graphical user interface from one computing platform to another one (e.g., from a desktop to a pocket computer), from one interaction surface to another (e.g., from a laptop to a wall screen) at run-time. For this purpose, any user interface subject to migration is described in USer Interface eXtensible Markup Language regarding its look & feel as well as the platforms and the surfaces involved in the migration. Each interface, in part or in whole, can be attached to a platform or a surface, detached from it, and migrated across platforms or interaction surfaces. Instead of communicating data and code during the migration, the description of the user interface of concern is wirelessly passed from one platform to another one to be regenerated on the target platform. To ensure a continuous control of the run-time migration, MigriXML automatically generates a world model representing the context of use where the source/target platforms/interaction surfaces are represented. Finally, migrating a user interface becomes as natural as its direct manipulation from one platform to another exactly in the same way as it is done on a single platform.

A transformational approach for developing multimodal web user interfaces is presented that progressively moves from a task model and a domain model to a final user interface. This approach consists of three steps: deriving one or many abstract user interfaces from a task model and a domain model, deriving one or many concrete user interfaces from each abstract one, and producing the code of the corresponding final user interfaces. To ensure these steps, transformations are encoded as graph transformations performed on the involved models expressed in their graph equivalent. For each step, a graph grammar gathers relevant graph transformations for accomplishing the sub-steps. The final user interface is multimodal as it involves graphical (keyboard, mouse) and vocal interaction. The approach outlined in the paper is illustrated throughout a running example for a graphical interface, a vocal interface, and two multimodal interfaces with graphical and vocal predominances, respectively.

Compared to other hospital units, the emergency department presents some distinguishing characteristics of its own. Emergency health-care delivery is a collaborative process involving the contribution of several individuals who accomplish their tasks while working autonomously under pressure and sometimes with limited resources. Effective computerization of the emergency department information system presents a real challenge due to the complexity of the scenario. Current computerized support suffers from several problems, including inadequate data models, clumsy user interfaces, and poor integration with other clinical information systems. To tackle such complexity, we propose an approach combining three points of view, namely the transactions (in and out of the department), the (mono and multi) user interfaces and data management. Unlike current systems, we pay particular attention to the user-friendliness and versatility of our system. This means that intuitive user interfaces have been conceived and specific software modeling methodologies have been applied to provide our system with the flexibility and adaptability necessary for the individual and group coordinated tasks. Our approach has been implemented by prototyping a web-based, multiplatform, multiuser, and versatile clinical information system built upon multitier software architecture, using the Java programming language.

Generative Programming (GP) is a computing paradigm allowing automatic creation of entire software families utilizing the configuration of elementary and reusable components. GP can be projected on different technologies, e.g. C++-templates, Java-Beans, Aspect-Oriented Programming (AOP), or Frame technology. This paper focuses on Frame Technology, which aids the possible implementation and completion of software components. The purpose of this paper is to introduce the GP paradigm in the area of GUI application generation. It demonstrates how automatically customized executable applications with GUI parts can be generated from an abstract specification.

User interfaces to information systems can be considered systematic as they consist of two types of tasks performed on classes of a domain model: basic tasks performed on one class at a time (such as insert, delete, modify, sort, list, print) and complex tasks performed on parts or whole of one or several classes (e.g., tasks involving various attributes of different classes with constraints between and establishing relationships between). This paper presents how a wizard tool can produce user interfaces to such tasks according to a model-driven approach based on a domain model of the information system. This process consists of seven steps: database selection, data source selection, building the opening procedure, data source selection for control widgets, building the closing procedure, setting the size of the widgets, and laying them out. The wizard generates code for Visual Basic and eMbedded Visual Basic, thus enabling to obtain support for both stationary and mobile tasks simultaneously, while maintaining consistency.

Re-engineering transforms a final user interface into a logical representation that is manipulable enough to allow\nforward engineering to port a UI from one computing platform to another with maximum flexibility and minimal effort. Re-engineering is used to adapt a UI to another context. This adaptation is governed by two main tasks:\nthe adaptation of the code itself to the new computing plating platform and the redesign of the UI to better suit\nthe new constraints of the target platform (interaction capabilities, screen size,…). To support this process, we\nhave developed a reverse engineering tool that allows a flexible recovery of the presentation model from Web\nsites, adapting the reverse engineering to the target platforms, and a forward engineering tool that converts this\npresentation model into any final executable UI, in particular expressed in VRML, WML, ...

This IUI'04 workshop is an opportunity to identify and articulate the key research challenges for the design and engineering of mixed reality systems. By clarifying and systematizing these challenges, we can improve our own understanding of the current state of the field, stimulate research activity, especially collaboration, and help establish the design of MR systems as a distinct research area.

This paper introduces and describes the notion of graceful degradation as a method for supporting the design of user interfaces for multiplatform systems when the capabilities of each platform are very different. The approach is based on a set of transformational rules applied to a single user interface designed for the less constraint platform. A major concern of the graceful degradation approach is to guarantee a maximal continuity between the platform specific versions of the user interface. In order to guarantee the continuity property, a priority ordering between rules is proposed. That ordering permits to apply first the rules with a minimal impact on the multiplatform system continuity.

Wold (Wizard fOr Leveraging the Development of multi-platform user interfaces) helps designers to produce running user interfaces to data bases of information systems simultaneously for multiple computing platforms. This software consists of a wizard application guiding designers step by step according to a mixed-initiative approach of production rules structured in a decision tree for choosing appropriate design options covering user interfaces to be produced. The main goal of this software is to speed up the development life cycle according to a transformational approach, spiral life cycle, derivation of user interfaces from data bases structure and queries, intelligent layout derived from data base structure and query. User interfaces are structured and described according to characteristics that remain independent of computing platforms.

The migration of a user interface (UI) is the action of transferring a UI from one device to another, for example from a desktop computer to a handheld device. A UI is said to be migratable if it has the ability to migrate. This paper describes how the QTk toolkit has been extended to provide a migratable UI and the application programming interface (API) provided to the developers. Basically, an indirection layer has been introduced between the application and the actual representation of the UI. The migration of a UI is achieved by firstly creating a clone of the state of the site displaying the UI, secondly by changing the indirection to point to this clone. The API provides a way to specify if (the entirety of) a window can be migrated or not at construction time. A migratable window returns a universal reference that can be given to any site with whom a network connection is possible. This reference can be used by a receiver widget to migrate the window there. Interestingly, a migratable window can itself contain a receiver widget configured to display the content of another migratable window: all windows are transparently migrated. Also a window (stationary or migratable) may contain one or more receiver widgets : it is possible to dynamically compose a Ul from several different UIs.

This paper describes a framework that serves as a reference for classifying user interfaces supporting multiple targets, or multiple contexts of use in the field of context-aware computing. In this framework, a context of use is decomposed into three facets: the end users of the interactive system, the hardware and software computing platform with which the users have to carry out their interactive tasks and the physical environment where they are working. Therefore, a context-sensitive user interface is a user interface that exhibits some capability to be aware of the context (context awareness) and to react to changes of this context. This paper attempts to provide a unified understanding of context-sensitive user interfaces rather than a prescription of various ways or methods of tackling different steps of development. Rather, the framework structures the development life cycle into four levels of abstraction: task and concepts, abstract user interface, concrete user interface and final user interface. These levels are structured with a relationship of reification going from an abstract level to a concrete one and a relationship of abstraction going from a concrete level to an abstract one. Most methods and tools can be more clearly understood and compared relative to each other against the levels of this framework. In addition, the framework expresses when, where and how a change of context is considered and supported in the context-sensitive user interface thanks to a relationship of translation. In the field of multi-target user interfaces is also introduced, defined, and exemplified the notion of plastic user interfaces. These user interfaces support some adaptation to changes of the context of use while preserving a predefined set of usability properties.

Mobile computing poses a series of unique challenges for user interface design and development: user interfaces must now accommodate the capabilities of various access devices and be suitable for different contexts of use, while preserving consistency and usability. We propose a set of techniques that will aid UI designers who are working in the domain of mobile computing. These techniques will allow designers to build UIs across several platforms, while respecting the unique constraints posed by each platform. In addition, these techniques will help designers to recognize and accommodate the unique contexts in which mobile computing occurs. Central to our approach is the development of a user-interface model that serves to isolate those features that are common to the various contexts of use, and to specify how the user-interface should adjust when the context changes. We claim that without some abstract description of the UI, it is likely that the design and the development of user-interfaces for mobile computing will be very time consuming, error-prone or even doomed to failure.

Several tools for working with guidelines already exist, both as commercial products as well as within research and development. As these tools frequently manipulate guidelines during many development steps of a user interface of an interactive application, they can overthrow any approach followed to develop this application. They also raise the fundamental question of to what extent can we trust these tools. To answer this question, we introduce five development milestones through which we must pass to produce a high quality tool for working with guidelines:\n1. An initial unstructured but comprehensive set of guidelines is formed by collecting, gathering, merging, compiling guidelines from all available world-wide ergonomic sources.\n2. The initial set is sorted and classified within a single organising framework.\n3. A methodology, paying particular attention to finding and applying relevant guidelines is developed for grounding interactive applications on the organised set of guidelines.\n4. The structured guidelines and the supporting methodology are given computational representations for manipulation by computer-based tools.\n5. The methodology developed in (3) is further modified to optimise the effectiveness of computer-assisted user interface design.\nIn this paper, we define these milestones and their associated goals, specify a general procedure and discuss some problems raised at each milestone. We then deliver an analytic synthesis of various experiences acquired to solve these problems and we discuss the validity of these experiences from the point of view of completeness, consistency and correctness. From these experiences, we finally draw some lessons useful for any future usage and development of a tool for working with guidelines. (C) 1999 Elsevier Science B.V. All rights reserved.