The cube is an object often used for tangible interaction across many fields of application, notably by manipulating cubes or performing gestures with or on them. Although some system- or designer-defined gestures exist for tangible cubes, user-defined gestures for them remain unexplored. To bridge this gap, we conducted three gesture elicitation studies, each using an identical experimental protocol on a different sample of 30 participants proposing a gesture for 13, 16, and 20 referents controlling a smart home appliance, respectively. Based on the total of 1,440 gesture proposals, we established a classification of user-defined gestures for designing tangible interaction with cubes in smart home control structured into seven major categories, which are further broken down into 55 subcategories. This classification is abstracted from properties such as the number of fingers and hands used to manipulate a cube, the repetition of gestures, their combination, and the time of articulation. A Poisson law estimates the number of participants to be recruited to obtain a desired number of gesture categories.

The wide range of algorithms for recognizing two-dimensional stroke gestures raises a crucial question: which recognizer should be selected depending on the gesture set and its context of use? Since the context of use expresses a category of users performing interactive tasks on a device in a physical environment, many contextual conditions are feasible. Recognizers have been tested in only a few of them, making them difficult to compare and select. A recognizer suitable for one context of use does not necessarily translate to another. Recognizer variables, such as the number of resampling points, the number of templates, and the number of users, vary from one recognizer to another, making them challenging to specify for practitioners and heterogeneous to compare for researchers. To address these issues, this article performs comparative tests of 16 recognizers (15 state-of-the-art recognizers and one for rotation invariance) in 11 gesture sets (9 reference sets and 2 for multi-device rotation invariance) against their accuracy, speed, and algorithmic complexity, in user-(in)dependent scenarios based on the variables mentioned above. In particular, the impact of two contextual dimensions, i.e., the visual impairments of end users and input devices, on accuracy is further studied. The comparative testing results in an empirically validated decision table for the researcher to select a recognizer depending on contextual conditions and a flow chart for the practitioner to be guided. We break down our design implications into two categories: gesture acquisition, recognition, and gesture set design. We also publicly release Gester , a JavaScript software for conducting comparative testing of recognizers under various contextual conditions, allowing the practitioner to test a recognizer in a project and the researcher to benchmark recognizers.

The performance of state-of-the-art stroke gesture recognizers has been extensively tested against accuracy and speed in the user-dependent or user-independent procedure on a wide variety of gesture sets. These tests were carried out under heterogeneous experimental conditions, making the results non-comparable from one source to another, and calling for a consistent assessment that ensures reproducibility of the results. When a new gesture set should be incorporated in a gesture-based user interface and/or when a new recognizer is made available, this gesture set needs to be tested with existing and new recognizers to determine which recognizer offers the best performance in a given context of use. To this end, this technical note presents Gester , a software that automates the performance testing of stroke gesture recognizers in terms of accuracy (through three recognition rates) and speed (through four execution times). It exports data for visualizing confusion matrices and confusion wheels. We elaborate on typical use cases supported by Gester , which include the two traditional testing procedures, i.e., user-dependent and user-independent scenarios, and provide two new ones, i.e., dataset-dependent and dataset-independent, to test the performance of recognizers under new conditions. Then, we illustrate a series of activities.

Although many advancements have been made on myoelectric pattern-recognition, the control of poly-articulated upper-limb prostheses remains insufficiently robust. Electrode-shift, sweat or fatigue degrade the performance of classifiers over time, resulting in unfruitful device usage and frequent re-calibration. To tackle this issue, here we introduce two models − µP6 and µP8 − that combine Geometric Algebra with nearest-neighbor classification. We aim at reducing both the necessary training data and training time and, unlike most current state-of-the-art algorithms, we exploit an alternative geometric representation (and visualization) of the EMG signal as different polygons for different types of gestures, facilitating the explanation of the decision-making process to a layman. Moreover, we explore four abstention strategies to reduce the number of misclassifications. We perform an offline analysis on two datasets, alongside two other standard models: nonlinear logistic regression (NLR) and linear discriminant analysis (LDA). Even with few training data, the proposed algorithms achieve high F1-scores (>0.95), significantly higher or non-significantly different from the values obtained with NLR and LDA, while maintaining relatively low abstentions rates and training times (<2 ms). The proposed algorithms allow to reduce the amount of training data and training times without compromising recognition rates. The proposed algorithms may contribute for a faster prosthesis re-calibration procedure while allowing to re-gain high recognition rates. Furthermore, the decision-making process is explainable and interpretable, potentially improving user trust and acceptance.

Finger-based gesture input becomes a major interaction modality for surface computing. Due to the low precision of the finger and the variation in gesture production, multistroke gestures are still challenging to recognize in various setups. In this paper, we present µV, a multistroke gesture recognizer that addresses the properties of articulation, rotation, scaling, and translation invariance by combining $P+'s cloud-matching for articulation invariance with !FTL's local shape distance for RST-invariance. We evaluate µV against five competitive recognizers on MMG, an existing gesture set, and on two new versions for smartphones and tablets, MMG+ and RMMG+, a randomly rotated version on both platforms. µV is significantly more accurate than its predecessors when rotation invariance is required and not significantly inferior when it is not. µV is also significantly faster than others with many samples and not significantly slower with few samples.

Despite the tremendous progress made for recognizing gestures acquired by various devices, such as the Leap Motion Controller, developing a gestural user interface based on such devices still induces a significant programming and software engineering effort before obtaining a running interactive application. To facilitate this development, we present QuantumLeap, a framework for engineering gestural user interfaces based on the Leap Motion Controller. Its pipeline software architecture can be parameterized to define a workflow among modules for acquiring gestures from the Leap Motion Controller, for segmenting them, recognizing them, and managing their mapping to functions of the application. To demonstrate its practical usage, we implement two gesture-based applications: an image viewer that allows healthcare workers to browse DICOM medical images of their patients without any hygiene issues commonly associated with touch user interfaces and a large-scale application for managing multimedia contents on wall screens. To evaluate the usability of QuantumLeap, seven participants took part in an experiment in which they used QuantumLeap to add a gestural interface to an existing application.

Arm-and-hand tracking by technological means allows gathering data that can be elaborated for determining gesture meaning. To this aim, machine learning (ML) algorithms have been mostly investigated looking for a balance between the highest recognition rate and the lowest recognition time. However, this balance comes mainly from statistical models, which are challenging to interpret. In contrast, we present$\mu C^{1}$and$\mu C^{2}$, two geometric model-based approaches to gesture recognition which support the visualization and geometrical interpretation of the recognition process. We compare$\mu C^{1}$and$\mu C^{2}$with respect to two classical ML algorithms, k-nearest neighbor (k-NN) and support vector machine (SVM), and two state-of-the-art (SotA) deep learning (DL) models, bidirectional long short-term memory (BiLSTM) and gated recurrent unit (GRU), on an experimental dataset of ten gesture classes from the Italian Sign Language (LIS), each repeated 100 times by five inexperienced non-native signers, and gathered with wearable technology (a sensory glove and inertial measurement units). As a result, we achieve a compromise between high recognition rates ($>90\%$) and low recognition times ($ < 0.1 {\mathrm{ s}}$) that is adequate for human–computer interaction. Moreover, we elaborate on the algorithms’ geometric interpretation based on geometric algebra, which supports some understanding of the recognition process.

While end users can acquire full 3D gestures with many input devices, they often capture only 3D trajectories, which are 3D uni-path, uni-stroke single-point gestures performed in thin air. Such trajectories with their $(x,y,z)$ coordinates could be interpreted as three 2D stroke gestures projected on three planes,\ie, $XY$, $YZ$, and $ZX$, thus making them admissible for established 2D stroke gesture recognizers. To investigate whether 3D trajectories could be effectively and efficiently recognized, four 2D stroke gesture recognizers, \ie, \$P, \$P+, \$Q, and Rubine, are extended to the third dimension: $\$P^3$, $\$P+^3$, $\$Q^3$, and Rubine-Sheng, an extension of Rubine for 3D with more features. Two new variations are also introduced: $\F for flexible cloud matching and FreeHandUni for uni-path recognition. Rubine3D, another extension of Rubine for 3D which projects the 3D gesture on three orthogonal planes, is also included. These seven recognizers are compared against three challenging datasets containing 3D trajectories, \ie, SHREC2019 and 3DTCGS, in a user-independent scenario, and 3DMadLabSD with its four domains, in both user-dependent and user-independent scenarios, with varying number of templates and sampling. Individual recognition rates and execution times per dataset and aggregated ones on all datasets show a highly significant difference of $\$P+^3$ over its competitors. The potential effects of the dataset, the number of templates, and the sampling are also studied.

Nearest neighbor classifiers recognize stroke gestures by computing a (dis)similarity between a candidate gesture and a training set based on points, which may require normalization, resampling, and rotation to a reference before processing. To eliminate this expensive preprocessing, this paper introduces a vector-between-vectors recognition where a gesture is defined by a vector based on geometric algebra and performs recognition by computing a novel Local Shape Distance (LSD) between vectors. We mathematically prove the LSD position, scale, and rotation invariance, thus eliminating the preprocessing. To demonstrate the viability of this approach, we instantiate LSD for n=2 to compare !FTL, a 2D stroke-gesture recognizer with respect to $1 and $P, two state-of-the-art gesture recognizers, on a gesture set typically used for benchmarking. !FTL benefits from a recognition rate similar to $P, but a significant smaller execution time and a lower algorithmic complexity.