The cube is an object often used for tangible interaction across many fields of application, notably by manipulating cubes or performing gestures with or on them. Although some system- or designer-defined gestures exist for tangible cubes, user-defined gestures for them remain unexplored. To bridge this gap, we conducted three gesture elicitation studies, each using an identical experimental protocol on a different sample of 30 participants proposing a gesture for 13, 16, and 20 referents controlling a smart home appliance, respectively. Based on the total of 1,440 gesture proposals, we established a classification of user-defined gestures for designing tangible interaction with cubes in smart home control structured into seven major categories, which are further broken down into 55 subcategories. This classification is abstracted from properties such as the number of fingers and hands used to manipulate a cube, the repetition of gestures, their combination, and the time of articulation. A Poisson law estimates the number of participants to be recruited to obtain a desired number of gesture categories.

User interface requirements elicitation is an early and critical stage in the software development life cycle: requirements are elicited from stakeholders, and user interface elements that should satisfy these requirements are also created, shared, and edited. To effectively capture the diverse perspectives of stakeholders on these requirements, crowd elicitation is a promising approach to leverage the collective knowledge of a time- and space-distributed group of stakeholders to manage user interface requirements and their associated user interface elements. Annotating these elements manually in freeform digital ink leads to results that are difficult to interpret and time-consuming to exploit by designers and developers. Instead, we propose that stakeholders annotate user interface elements by commonly-agreed gestures that will be automatically recognized to populate 10 annotation metrics. To this end, we present Crowditivity , a publicly available web application that effectively harnesses crowd elicitation in three related activities: (1) collecting and managing user interface functional and non-functional requirements according to an adapted Volere Shell template; (2) assigning user interface elements of a mockup or prototype to requirements to satisfy them; and (3) enabling stakeholders to annotate these elements by ink-based gestures. A 2D multi-stroke gesture classifier is specifically tailored to recognize gesture classes corresponding to 18 annotation types, that are empirically validated through a gesture elicitation study ( n 1 = 26) confirmed by an identification study ( n 2 = 59). Through an evaluation of a case study for a web application, we investigated how eight representative designers valued the Crowditivity platform. We finally define and illustrate a set of ten metrics to quantify the output of Crowditivity in terms of annotations and annotators resulting from the gesture recognition.

2D-3D gesture interaction most often associates a recognized gesture with a single command, such as “Swipe right” to navigate to the next screen. When gestures are performed one after another in a gesture sequence, each gesture still needs to be associated with one separate command at a time. Increasing the number of such gestures to realize more commands is not favorable to the end user, who memorizes and uses only a limited number of gestures. Instead of relying on gesture sequences, we can expand the range of executable commands to perform more complex tasks by gesture composition. To this end, we define an instruction as an action applied to a device in an environment, together with its valued parameters, such as “Increase the brightness of the bedroom lamp to 10“. Elementary gestures, preferably congruent and hierarchical, are composed into a compound gesture. The instructions can be combined into macro-instructions to summarize gesture composition. To put these definitions in practice, we developed Emeriti , a framework for the recognition of elementary and compound gestures in both 2D (uni- or multistroke gestures recognized on an interactive surface) and 3D (point trajectories in space) by composition. We demonstrate the effectiveness and usefulness of turning gesture sequences into gesture composition using (macro-)instructions in an application simulating a smart home in which every device in every room can be controlled by compound gestures with parameters. This makes it possible to multiply the number of possible commands for more complex tasks without considerably increasing the number of gestures.

Effective decision-making often relies on timely insights from complex visual data. While Information Visualization (InfoVis) dashboards can support this process, they rarely adapt to users’ cognitive state, and less so in real time. We present SYMBIOTIK, an intelligent, context-aware adaptive visualization system that leverages neurophysiological signals to estimate mental workload (MWL) and dynamically adapt visual dashboards using reinforcement learning (RL). Through a user study with 120 participants and three visualization types, we demonstrate that our approach improves task performance and engagement. SYMBIOTIK offers a scalable, real-time adaptation architecture, and a validated methodology for neuroadaptive user interfaces.

The ‘keyword method’ is an effective technique for learning vocabulary of a foreign language. It involves creating a memorable visual link between what a word means and what its pronunciation in a foreign language sounds like in the learner’s native language. However, these memorable visual links remain implicit in the people’s mind and are not easy to remember for a large number of words. To enhance the memorisation and recall of the vocabulary, we developed an application that combines the keyword method with text-to-image generators to externalise the memorable visual links into visuals. These visuals represent additional stimuli during the memorisation process. To explore the effectiveness of this approach we first run a pilot study to investigate how difficult it is to externalise the descriptions of mental visualisations of memorable links, by asking participants to write them down. We used these descriptions as prompts for text-to-image generator (DALL-E 2) to convert them into images and asked participants to select their favourites. Next, we compared different text-to-image generators (DALL-E 2, Midjourney, Stable and Latent Diffusion) to evaluate the perceived quality of the generated images by each. Despite heterogeneous results, participants mostly preferred images generated by DALL-E 2, which was used also for the final study. In this study, we investigated whether providing such images enhances the retention of vocabulary being learned, compared to the keyword method alone. Our results indicate that people did not encounter difficulties describing their visualisations of memorable links and that providing corresponding images significantly increases memory retention.

Handwriting movements can be leveraged as a unique form of behavioral biometrics, to verify whether a real user is operating a device or application. This task can be framed as a “reverse Turing test” in which a computer has to detect if an input instance has been generated by a human or artificially. To tackle this task, we study ten public datasets of handwritten symbols (isolated characters, digits, gestures, pointing traces, and signatures) that are artificially reproduced using seven different synthesizers, including, among others, the Kinematic Theory (ΣΛ model), generative adversarial networks, Transformers, and Diffusion models. We train a shallow recurrent neural network that achieves excellent performance (98.3% Area Under the ROC Curve (AUC) score and 1.4% equal error rate on average across all synthesizers and datasets) using nonfeaturized trajectory data as input. In few-shot settings, we show that our classifier achieves such an excellent performance when trained on just 10% of the data, as evaluated on the remaining 90% of the data as a test set. We further challenge our classifier in out-of-domain settings, and observe very competitive results as well. Our work has implications for computerized systems that need to verify human presence, and adds an additional layer of security to keep attackers at bay.

This article explores video-viewing behavior when videos are wrapped in interactive content in the case of iOtok, a 13-episodes web documentary series. The interaction and viewing data were collected over a period of one year, providing a dataset of more than 12,200 total video views by 6000 users. Standard metrics (video views, percentage viewed, number of sessions) show higher active participation for registered users compared to unregistered users. Results also indicate that serialization over multiple weeks is an effective strategy for audience building over a long period of time without negatively affecting video views. In viewing behavior analysis, we focused on three perspectives: (i) regularity (watching on a weekly basis or not), (ii) intensity (number of videos per session), and (iii) order of watching. We performed a perspective based and combined perspectives analysis involving manual coding techniques, rule-based, and k-means clustering algorithms to reveal different user profiles (intermittent, exemplary, detached, enthusiastic users, and nibblers) and highlight further viewing behavior differences (e.g., post-series users binge-watched more than concurrent users during first 13 weeks while the series was weekly released). We discuss how these results can be used to inform the design and promotion of future web documentaries.

Gesture recognition with miniaturised radar sensors has received increasing attention as a novel interaction medium. The practical use of radar technology, however, often requires sensing through materials. Yet, it is still not well understood how the internal structure of materials impacts recognition performance. To tackle this challenge, we collected a large dataset of 14,090 radar recordings for 6 paradigmatic gesture classes sensed through a variety of everyday materials, performed by humans (6 materials) and a robot system (75 materials). Next, we developed a hybrid CNN+LSTM deep learning model and derived a robust indirect method to measure signal distortions, which we used to compile a comprehensive catalogue of materials for radar-based interaction. Among other findings, our experiments show that it is possible to estimate how different materials would affect gesture recognition performance of arbitrary classifiers by selecting just 3 reference materials. Our catalogue, software, models, data collection platform, and labeled datasets are publicly available.