Pointing transfer functions define the mapping between input devices and onscreen cursor movement. Despite being used by millions daily, only marginal improvements in pointing performance have been achieved by tuning transfer functions since the introduction of acceleration-based gains. We present TFTune, a reinforcement learning-based approach for improving pointing by automatically tuning personalized transfer functions. We show that TFTune-generated functions outperform operating system defaults, improving movement times by 7% on macOS when using a trackpad (7 minutes of tuning) and 8% on participants’ personal Windows computers with hardware (i.e. mice and monitors) of varying characteristics (after just 1 minute of tuning). Further, we show that TFTune generalizes beyond traditional pointing devices, providing 16% improvement for a muscle-computer interface (2 minutes of tuning). TFTune demonstrates an initial approach for scalable and meaningful performance improvements in input–output mappings, opening a new direction for exploring the use of machine learning for improving fundamental computer inputs.

Athletes of all levels can find drills repetitive, tedious, and unengaging. Sport training games address this, aiming to make practicing techniques fun. However, there is little information about what feedback athletes need to support skill acquisition and improve engagement. To address this, we develop a game for training the optimal shooting arc technique for basketball free throws, and compare several scoring mechanics based on: 1) adherence to technique, 2) results (successfully making a shot), and 3) a combination of technique and results feedback. We find that technique feedback is essential to adopting a new technique, while results feedback improves engagement. We also find that combining technique and results feedback is effective, but can be overly complex for novices. Our results allow us to make recommendations to designers and researchers on how they can effectively scaffold skill development and engagement in sport training games.

Biometric footstep recognition, based on a person’s unique pressure patterns under their feet during walking, is an emerging field with growing applications in security and safety. However, progress in this area has been limited by the lack of large, diverse datasets necessary to address critical challenges such as generalization to new users and robustness to shifts in factors like footwear or walking speed. The recent release of the UNB StepUP-P150 dataset, the largest and most comprehensive collection of high-resolution footstep pressure recordings to date, opens new opportunities for addressing these challenges through deep learning. To mark this milestone, the First International StepUP Competition for Biometric Footstep Recognition was launched. Competitors were tasked with developing robust recognition models using the StepUP-P150 dataset that were then evaluated on a separate, dedicated test set designed to assess verification performance under challenging variations, given limited and relatively homogeneous reference data. The competition attracted global participation, with 23 registered teams from academia and industry. The top-performing team, Saeid UCC, achieved the best equal error rate (EER) of 10.77% using a generative reward machine (GRM) optimization strategy. Overall, the competition showcased strong solutions, but persistent challenges in generalizing to unfamiliar footwear highlight a critical area for future work.

With growing adoption of blockchain in established and emerging applications, there is an increasing need to support efficient ad hoc querying of authenticated historical data. This is especially true in fields such as healthcare to meet the rigorous security and regulatory requirements of ever-expanding digital health platforms. Existing blockchain systems, however, offer little or no support for querying capabilities over historical data. Although a full blockchain archive node can be used to maintain historical records of all executed transactions on the chain, it is not scalable when dealing with large volumes of data. Moreover, such ‘offline’ historical data lack tamper evidence support. To address these issues, we introduce an authenticated index structure called Authenticated Multi-Version Skip List (AMVSL), designed to support a rich set of query features over historical blockchain data. We further present three range queries: SVRK, MVRK, and MVAK, which offer querying over a range of keys and a range of versions. Our experimental evaluation of two healthcare-inspired examples demonstrates that AMVSL efficiently supports these queries and can achieve performance that is several orders of magnitude faster than existing authenticated data structures.

Athletes often learn suboptimal techniques that place a ceiling on their performance or put them at risk of injury. Adopting a new technique can lead to a short-term dip in performance while learning it, which can be demotivating and cause an athlete to revert to their previous, suboptimal technique. To address the challenge of demotivation in adopting new techniques, we explore technique-oriented sport training games, which aim to improve sport skills by capturing behaviour and providing feedback that motivates adopting a new technique. As of yet, previous work has provided little information on how to design sports training games, and some early research suggests that immersive games may distract players, preventing them from being able to learn a new technique effectively. We designed a study comparing a baseline non-game training system with three versions of a running game, ranging from a simplistic text-based game to a 3D audiovisual game with motion feedback. We find that the games with more immersive elements were as effective as the baseline system for adopting a new technique but were preferred by players and improved their intrinsic motivation. We propose design considerations from these findings and provide new directions for researching effective sport training games.

The control performance of myoelectric prostheses would not only depend on the feature extraction and classification algorithms but also on interactions of dynamic window-based hyperparameters (WBHP) used to construct input signals. However, the relationship between these hyperparameters and how they influence the performance of the convolutional neural networks (CNNs) during motor intent decoding has not been studied. Therefore, we investigated the impact of various combinations of WBHP (window length and overlap) employed for the construction of raw two-dimensional (2-D) surface electromyogram (sEMG) signals on the performance of CNNs when used for motion intent decoding. Moreover, we examined the relationship between the window length of the 2-D sEMG and three commonly used CNN kernel sizes. To ensure high confidence in the findings, we implemented three CNNs, which are variants of the existing models, and a newly proposed CNN model. Experimental analysis was conducted using three distinct benchmark databases, two from upper limb amputees and one from able-bodied subjects. The results demonstrate that the performance of the CNNs improved as the overlap between consecutively generated 2-D signals increased, with 75% overlap yielding the optimal improvement by 12.62% accuracy and 39.60% F1-score compared to no overlap. Moreover, the CNNs performance was better for kernel size of seven than three and five across the databases. For the first time, we have established with multiple evidence that WBHP would substantially impact the decoding outcome and computational complexity of deep neural networks, and we anticipate that this may spur positive advancement in myoelectric control and related fields.

Electromyography (EMG) has been explored as an HCI input modality following a long history of success for prosthesis control. While EMG has the potential to address a range of hands-free interaction needs, it has yet to be widely accepted outside of prosthetics due to a perceived lack of robustness and intuitiveness. To understand how EMG input systems can be better designed, we sampled the ACM digital library to identify limitations in the approaches taken. Leveraging these works in combination with our research group’s extensive interdisciplinary experience in this field, four themes emerged (1) interaction design, (2) model design, (3) system evaluation, and (4) reproducibility. Using these themes, we provide a step-by-step framework for designing EMG-based input systems to strengthen the foundation on which EMG-based interactions are built. Additionally, we provide a call-to-action for researchers to unlock the hidden potential of EMG as a widely applicable and highly usable input modality.

While there has been renewed interest in the use of myoelectric control for general-purpose applications, the burden of training and maintaining robust models still limits its real-world viability. Online unsupervised adaptation has been proposed to solve this issue by updating the model using predicted pseudo-labels in real time during regular device use. Until now, however, these unsupervised strategies have been limited as they rely on the very classifier outputs they are adapting, making them ill-suited when there is a drastic shift in the input space (e.g., after donning and doffing a device) or there is insufficient training data. In such situations, leveraging context (i.e., task-specific information that can help understand or assess a circumstance) could provide additional guidance for adaptation and improve its robustness. Although difficult to extract in traditional prosthesis control use cases without additional sensors, context may be more readily available in other general-purpose applications, such as in human-computer interaction. In this study, we explore leveraging context, both positive (i.e., reinforcing correct actions) and negative (i.e., correcting poor actions), for conditioning pseudo-label predictions within an adaptive gamified target acquisition setting. The results show that leveraging this additional con-text significantly outperforms the current state-of-the-art high-confidence unsupervised adaptation (p<0.05) using both offline and online performance metrics. This pilot work contributes novel findings and contextual approaches that do not rely on additional sensors, and thus outlines a promising direction of study for myoelectric control as a reliable and effective interaction technique.

Post-processing techniques have been shown to improve the quality of the decision stream generated by classifiers used in pattern-recognition-based myoelectric control. However, these techniques have largely been tested individually and on well-behaved, stationary data, failing to fully evaluate their trade-offs between smoothing and latency during dynamic use. Correspondingly, in this work, we survey and compare 8 different post-processing and decision stream improvement schemes in the context of continuous and dynamic class transitions: majority vote, Bayesian fusion, onset locking, outlier detection, confidence-based rejection, confidence scaling, prior adjustment, and adaptive windowing. We then propose two new temporally aware post-processing schemes that use changes in the decision and confidence streams to better reject uncertain decisions. Our decision-change informed rejection (DCIR) approach outperforms existing schemes during both steady-state and transitions based on error rates and decision stream volatility whether using conventional or deep classifiers. These results suggest that added robustness can be gained by appropriately leveraging temporal context in myoelectric control.

Gait has been shown to be a highly unique and repeatable behavioural biometric, and integrated pressure-based sensing modalities provide a convenient and robust environment for authentication. Many studies in this emerging field, and biometrics in general, however, have used techniques for model development and validation that can yield optimistically biased and unrealistic performance estimates. In this study, the bias that can result from information leakage during training was demonstrated in two aspects of model evaluation: 1) the order of processing steps in machine learning pipelines, and 2) the use of a posteriori performance measures, which are based on test samples that were used to determine certain model parameters. Additionally, the drawbacks of 3) using single-number metrics for comparing models or for final performance estimates was also demonstrated. Ultimately, methods to avoid these pitfalls are recommended.

The role of feature extraction in electromyogram (EMG) based pattern recognition has recently been emphasized with several publications promoting deep learning (DL) solutions that outperform traditional methods. It has been shown that the ability of DL models to extract temporal, spatial, and spatio–temporal information provides significant enhancements to the performance and generalizability of myoelectric control. Despite these advancements, it can be argued that DL models are computationally very expensive, requiring long training times, increased training data, and high computational resources, yielding solutions that may not yet be feasible for clinical translation given the available technology. The aim of this paper is, therefore, to leverage the benefits of spatio–temporal DL concepts into a computationally feasible and accurate traditional feature extraction method. Specifically, the proposed novel method extracts a set of well-known time-domain features into a matrix representation, convolves them with predetermined fixed filters, and temporally evolves the resulting features over a short and long-term basis to extract the EMG temporal dynamics. The proposed method, based on Fixed Spatio–Temporal Convolutions, offers significant reductions in the computational costs, while demonstrating a solution that can compete with, and even outperform, recent DL models. Experimental tests were performed on sparse-and high-density EMG (HD-EMG) signals databases, across a total of 44 subjects performing a maximum of 53 movements. Despite the simplification compared to deep approaches, our results show that the proposed solution significantly reduces the classification error rates by 3% to 10% in comparison to recent DL models, while being efficient for real-time implementations.

Despite a historical focus on prosthetics, the incorporation of electromyography (EMG) sensors into less obtrusive wearable designs has recently gained attention as a potential human–computer interaction scheme for general consumer use. Because consumers are more used to wrist-worn devices, this article presents a comprehensive and systematic investigation of the feasibility of hand gesture recognition using EMG signals recorded at the wrist. A direct comparison of signal and information quality is conducted between concurrently recorded wrist and forearm signals. Both signals were collected simultaneously from 21 subjects while they performed a selection of 17 different single-finger gestures, multifinger gestures, and wrist gestures. Wrist EMG signals yielded consistently higher ($p< 0.05$) signal quality metrics than forearm signals for gestures that involved fine finger movements, while maintaining comparable quality for wrist gestures. Similarly, the performance of both individual state-of-the-art EMG features and a standard feature set was found to be significantly better when using wrist signals for single and multifinger gestures, and comparable for wrist gestures. Classifiers trained and tested using wrist EMG signals achieved average accuracy levels of 92.1% for single-finger gestures, 91.2% for multifinger gestures, and 94.7% for the conventional wrist gestures. In conclusion, this article clearly demonstrates the feasibility of using wrist EMG signals for hand gesture recognition. Results highlight not only the promise of this approach, but also the viability of incorporating prior knowledge from the prosthetics field in the design of wrist-based EMG pattern recognition systems.

Efficient storage and transmission of electromyogram (EMG) data are important for emerging applications such as telemedicine and big data, as a vital tool for further advancement of the field. However, due to limitations in internet speed and hardware resources, transmission and storage of EMG data are challenging. As a solution, this work proposes a new method for EMG data compression using deep convolutional autoencoders (CAE). Eight-channel EMG data from 10 subjects, and high-density EMG data from 18 subjects, were investigated for compression. The CAE architecture was designed to extract an abstract data representation that is heavily compressed, but from which the salient information for classification can be effectively reconstructed. The proposed method attained efficient compression; for CR = 1600, the average PRDN (percentage RMS difference normalized) was 31.5% and the wrist motions classification accuracy (CA) reduced roughly 5%. The CAE substantially outperformed the state-of-the-art high-efficiency video coding and a well-known wavelet-thresholding compression technique. Moreover, by reducing the bit-resolution of the CAE's compressed data from 24 bits to 6 bits, an additional 4-fold compression was achieved without significant degradation of the reconstruction performance. Furthermore, the CAE's inter-subject performance was promising; e.g., for CR = 1600, the PRDN for the inter-subject case was only 2.6% less than that of the within-subject performance. The powerful EMG compression performance with remarkable reconstruction results reflects the CAEs potential as an automatic end-to-end approach with the ability to learn the complete encoding and decoding process. Furthermore, the excellent inter-subject performance demonstrates the generalizability and usability of the proposed approach.

The growth of IoT technology, increasing prevalence of embedded devices, and advancements in biomedical technology have led to the emergence of numerous wearable health monitoring devices (WHMDs) in clinical settings and in the community. The majority of these devices are Bluetooth Low Energy (BLE) enabled. Though the advantages offered by BLE-enabled WHMDs in tracking, diagnosing, and intervening with patients are substantial, the risk of cyberattacks on these devices is likely to increase with device complexity and new communication protocols. Furthermore, vendors face risk and financial tradeoffs between speed to market and ensuring device security in all situations. Previous research has explored the security and privacy of such devices by manually testing popular BLE-enabled WHMDs in the market and generally discussed categories of possible attacks, while mostly focused on IP devices. In this work, we propose a new semi-automated framework that can be used to identify and discover both known and unknown vulnerabilities in WHMDs. To demonstrate its implementation, we validate it with a number of commercially available BLE-enabled enabled wearable devices. Our results show that the devices are vulnerable to a number of attacks, including eavesdropping, data manipulation, and denial of service attacks. The proposed framework could therefore be used to evaluate potential devices before adoption into a secure network or, ideally, during the design and implementation of new devices.

Peripheral breathing guides – tools designed to influence breathing while completing another primary task – have been proposed to provide physiological benefits during information work. While research has shown that guides can influence breathing rates under ideal conditions, there is little evidence that they can lead to underlying markers of physiological benefit under interrupted work conditions. Further, even if guides are effective during work tasks, it is unclear how personal and workplace factors affect peoples' willingness to adopt them for everyday use. In this paper, we present the results of a comparative, mixed-methods study of five different peripheral breathing guides. Our findings show that peripheral breathing guides are viable and can provide physiological markers of benefit during interrupted work. Further, we show that guides are effective – even when use is intermittent due to workplace distractions. Finally, we contribute guidelines to support the design of breathing guides for everyday information work.

Automatic speech recognition interfaces are becoming increasingly pervasive in daily life as a means of interacting with and controlling electronic devices. Current speech interfaces, however, are infeasible for a variety of users and use cases, such as patients who suffer from locked-in syndrome or those who need privacy. In these cases, an interface that works based on envisioned speech, the idea of imagining what one wants to say, could be of benefit. Consequently, in this work, we propose an imagined speech Brain-Computer-Interface (BCI) using Electroencephalogram (EEG) signals. EEG signals are processed using a deep spatio-temporal learning architecture with 1D Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM), respectively. LSTM units are implemented in a many-to-many fashion to produce a time series of imagined speech outputs. Using this series, the performance of the system is boosted using majority vote (MV) post-processing to further improve results. The performance is evaluated on two publicly available datasets; one to test the performance of the tuned model, and another to test its generalization to a new dataset. The proposed architecture outperforms previous results with improvements of up to 23.7%.

With the emergence of wearable technology, IoT, and Edge computing, the nature of healthcare is rapidly shifting towards digital health aided by these ICT technologies. At the same time, consumer devices, such as smart, wearable fitness watches are gaining market share as a way to monitor physical activity and wellness. Despite these advances, and their ability to capture longitudinal behavioural patterns, these devices have yet to be fully leveraged within the healthcare system. If the user-generated data from such devices could be collected without com-promising an individual’s privacy, these insights could comprise part of a more holistic and preventative healthcare solution. In this article, we propose an Edge-assisted data analytics frame-work that uses Federated Learning to re-train local machine learning models using user-generated data. This framework could leverage pre-trained models to extract user-customized insights while preserving privacy and Cloud resources. We also identify some potential application scenarios and discuss research challenges to be explored within the proposed framework.

The Timed-Up-and-Go (TUG) test is a simple clinical tool commonly used to quickly assess the mobility of patients. Researchers have endeavored to automate the test using sensors or motion tracking systems to improve its accuracy and to extract more resolved information about its sub-phases. While some approaches have shown promise, they often require the donning of sensors or the use of specialized hardware, such as the now discontinued Microsoft Kinect, which combines video information with depth sensors (RGBD). In this work, we leverage recent advances in computer vision to automate the TUG test using a regular RGB video camera without the need for custom hardware or additional depth sensors. Thirty healthy participants were recorded using a Kinect V2 and a standard video feed while performing multiple trials of 3 and 1.5 meter versions of the TUG test. A Mask Regional Convolutional Neural Net (R-CNN) algorithm and a Deep Multitask Architecture for Human Sensing (DMHS) were then used together to extract global 3D poses of the participants. The timing of transitions between the six key movement phases of the TUG test were then extracted using heuristic features extracted from the time series of these 3D poses. The proposed video-based vTUG system yielded the same error as the standard Kinect-based system for all six key transitions points, and average errors of less than 0.15 seconds from a multi-observer hand labeled ground truth. This work describes a novel method of video-based automation of the TUG test using a single standard camera, removing the need for specialized equipment and facilitating the extraction of additional meaningful information for clinical use.

Rejection of movements based on the confidence in the classification decision has previously been demonstrated to improve the usability of pattern recognition based myoelectric control. To this point, however, the optimal rejection threshold has been determined heuristically, and it is not known how different thresholds affect the tradeoff between error mitigation and false rejections in real-time closed-loop control. To answer this question, 24 able-bodied subjects completed a real-time Fitts' law-style virtual cursor control task using a support vector machine classifier. It was found that rejection improved information throughput at all thresholds, with the best performance coming at thresholds between 0.60 and 0.75. Two fundamental types of error were defined and identified: operator error (identifiable, repeatable behaviors, directly attributable to the user), and systemic error (other errors attributable to misclassification or noise). The incidence of both operator and systemic errors were found to decrease as rejection threshold increased. Moreover, while the incidence of all error types correlated strongly with path efficiency, only systemic errors correlated strongly with throughput and trial completion rate. Interestingly, more experienced users were found to commit as many errors as novice users, despite performing better in the Fitts' task, suggesting that there is more to usability than error prevention alone. Nevertheless, these results demonstrate the usability gains possible with rejection across a range of thresholds for both novice and experienced users alike.

A myoelectric prosthesis (myo) is a dexterous artificial limb controlled by muscle contractions. Learning to use a myo can be challenging, so extensive training is often required to use a myo prosthesis effectively. Signal visualizations and simple muscle-controlled games are currently used to help patients train their muscles, but are boring and frustrating. Furthermore, current training systems require expensive medical equipment and clinician oversight, restricting training to infrequent clinical visits. To address these limitations, we developed a new game that promotes fun and success, and shows the viability of a low-cost myoelectric input device. We adapted a user-centered design (UCD) process to receive feedback from patients, clinicians, and family members as we iteratively addressed challenges to improve our game. Through this work, we introduce a free and open myo training game, provide new information about the design of myo training games, and reflect on an adapted UCD process for the practical iterative development of therapeutic games.