Wearable e-textile interfaces require gesture recognition capabilities but face severe constraints in power consumption, computational capacity, and form factor that make traditional deep learning impractical. While lightweight architectures like MobileNet improve efficiency, they still demand thousands of parameters, limiting deployment on textile-integrated platforms. We introduce a convexified attention mechanism for wearable applications that dynamically weights features while preserving convexity through nonexpansive simplex projection and convex loss functions. Unlike conventional attention mechanisms using non-convex softmax operations, our approach employs Euclidean projection onto the probability simplex combined with multi-class hinge loss, ensuring global convergence guarantees. Implemented on a textile-based capacitive sensor with four connection points, our approach achieves 100.00% accuracy on tap gestures and 100.00% on swipe gestures—consistent across 10-fold cross-validation and held-out test evaluation—while requiring only 120–360 parameters, a 97% reduction compared to conventional approaches. With sub-millisecond inference times (290–296 μ s) and minimal storage requirements (< 7KB), our method enables gesture interfaces directly within e-textiles without external processing. Our evaluation, conducted in controlled laboratory conditions with a single-user dataset, demonstrates feasibility for basic gesture interactions. Real-world deployment would require validation across multiple users, environmental conditions, and more complex gesture vocabularies. These results demonstrate how convex optimization can enable efficient on-device machine learning for textile interfaces.

Recent studies of driver distraction have reported a number of detrimental effects of in-vehicle interaction on driver performance. This paper examines and predicts the potential effects of such interaction on other vehicles around the driver's vehicle. Specifically, the paper describes how computational cognitive models can be used to predict the complex interactions among several vehicles driving in a line when one or more of the vehicles' drivers are performing a secondary task (phone dialing). The results of simulating two distinct car-following scenarios illustrate that in-vehicle interaction by one driver can have significant downstream effects on other drivers, especially with respect to speed deviations relative to a lead vehicle. This work generalizes recent work developing computational evaluation tools for user interfaces in complex domains, and further serves as an example of how user interaction in some domains can have broader effects on the community at large.

Journal Article Shared Input Multimodal Mobile Interfaces: Interaction Modality Effects on Menu Selection in Single-Task and Dual-Task Environments Get access Shengdong Zhao, Shengdong Zhao * 1Department of Computer Science, National University of Singapore, 13 Computing Drive, Computing 2, #01-04, Singapore 117417 *Corresponding author: [email protected] Search for other works by this author on: Oxford Academic Google Scholar Duncan P. Brumby, Duncan P. Brumby 2UCL Interaction Centre, University College London, Gower Street, London WC1E 6BT, UK Search for other works by this author on: Oxford Academic Google Scholar Mark Chignell, Mark Chignell 3Knowledge Media Design Institute (KMDI), University of Toronto, 27 King's College Circle, Toronto, Ont., Canada M5S 1A1 Search for other works by this author on: Oxford Academic Google Scholar Dario Salvucci, Dario Salvucci 4Drexel University, 3141 Chestnut Street, Philadelphia, PA 19104, USA Search for other works by this author on: Oxford Academic Google Scholar Sahil Goyal Sahil Goyal 5National University of Singapore, 13 Computing Drive, Computing 2, #01-04, Singapore 117417 Search for other works by this author on: Oxford Academic Google Scholar Interacting with Computers, Volume 25, Issue 5, September 2013, Pages 386–403, https://doi.org/10.1093/iwc/iws021 Published: 06 February 2013 Article history Received: 29 December 2011 Revision received: 11 October 2012 Accepted: 13 November 2012 Published: 06 February 2013

Connected vehicles offer great potential for new sources of information, but may also introduce new sources of distraction. This paper compares three methods to quantify distraction, and focuses on one method: computational models of driver behavior. An integration of a saliency map and the Distract-R prototyping and evaluation system is proposed as a potential model. The saliency map captures the bottom-up influences of visual attention and this influence is integrated with top-down influences captured by Distract-R. The combined model will assess the effect of coordinating salient visual features and drivers' expectations, and in using both together, generate more robust predictions of performance.

Previous studies of multitasking have highlighted the importance of cognitive load in interruptibility by showing that forced interruptions are least disruptive when cognitive load is low, and also that users prefer to address interruptions at low-load points when given a choice. We present an empirical study that uses a ringing-phone scenario to examine how users manage deferrable interruptions in the presence of varying time constraints. We found that while cognitive load did influence multitasking as expected, the time constraints placed on the user also had a significant impact. In particular, we observed three distinct strategies for addressing interruption: the expected strategy of switching at low-load points, but also two other strategies of continuing on after a low-load point or giving up at a high-load point. The presence of the latter two strategies strongly suggests that users can adapt their multitasking behavior with respect to the time constraints of the interrupting task.

Theoretical accounts of task resumption after interruption have almost exclusively argued for resumption as a primarily memory-based process. In contrast, for many task domains, resumption can more accurately be represented in terms of a process of reconstruction-perceptual re-encoding of the information necessary to perform the task. This paper discusses a theoretical, computational framework in which one can represent these reconstruction processes and account for aspects of performance, such as measures of resumption lag. The paper also describes computational models of two sample task domains that illustrate the sometimes complex relationship between reconstruction and more general human cognitive, perceptual, and motor processes.

Recent research has found that forced interruptions at points of higher mental workload are more disruptive than at points of lower workload. This paper investigates a complementary idea: when users experience deferrable interruptions at points of higher workload, they may tend to defer processing of the interruption until times of lower workload. In an experiment, users performed a mail-browser primary task while being occasionally interrupted by a secondary chat task, evenly distributed between points of higher and lower workload. Analysis showed that 94% of the time, users switched to the interrupting task during periods of lower workload, versus only 6% during periods of higher workload. The results suggest that when interruptions can be deferred, users have a strong tendency to ''monotask'' until primary-task mental workload has been minimized.

A common problem in many areas of behavioral research is the analysis of the large volume of data recorded during the execution of the tasks being studied. Recent work has proposed the use of an automated method based on canonical sets to identify the most representative patterns in a large data set, and described an initial experiment in identifying canonical web-browsing patterns. However, there is a significant limitation to the method: it requires the similarity matrix to be symmetric, and thus can only be used for problems that can be modeled as unoriented topologies. In this paper we propose a novel enhancement to the method to support oriented topologies by allowing the similarity matrix to be nonsymmetric. We demonstrate the power of this new technique by applying the new method to find canonical lane changes in a driving simulator experiment.

While many computer tutoring systems have long been delivered as desktop applications, these systems have only recently begun to appear on mobile devices. In this work we apply principles of mobile human computer interaction and mobile learning to the design and development of an intelligent tutoring system delivered on a personal digital assistant. We developed a proof-of-concept mobile tutor for business math and tested it in a course designed for first-year college students. Our experiences with the tutor suggest that a tutor based on principles of both mobile learning and mobile human computer interaction can provide a mobile tutoring system capable of providing support to students consistent with mobile device usage patterns.

We investigate how people adapt their strategy for interleaving multiple concurrent tasks to varying objectives. A study was conducted in which participants drove a simulated vehicle and occasionally dialed a telephone number on a mobile phone. Experimental instructions and feedback encouraged participants to focus on either driving or dialing. Results show that participants adapted their task interleaving strategies to meet the required task objective, but in a manner that was nonetheless intricately shaped by internal psychological constraints. In particular, participants tended to steer in between dialing chunks of digits even when extreme vehicle drift implied that more reactive strategies would have generated better lane keeping. To better understand why drivers interleaved tasks at chunk boundaries, a modeling analysis was conducted to derive performance predictions for a range of dialing strategies. The analysis supported the idea that interleaving at chunk boundaries efficiently traded the time given up to dialing with the maintenance of a central lane position. We discuss the implications of this work in terms of contributions to understanding how cognitive constraints shape strategy adaptations in dynamic multitask environments.

While the collection of behavioral protocols has been common practice in human-computer interaction research for many years, the analysis of large protocol data sets is often extremely tedious and time-consuming, and automated analysis methods have been slow to develop. This paper proposes an automated method of protocol analysis to find canonical behaviors --- a small subset of protocols that is most representative of the full data set, providing a reasonable "big picture" view of the data with as few protocols as possible. The automated method takes advantage of recent algorithmic developments in computational vision, modifying them to allow for distance measures between behavioral protocols. The paper includes an application of the method to web-browsing protocols, showing how the canonical behaviors found by the method match well to sets of behaviors identified by expert human coders.

Multitasking in user behavior can be represented along a continuum in terms of the time spent on one task before switching to another. In this paper, we present a theory of behavior along the multitasking continuum, from concurrent tasks with rapid switching to sequential tasks with longer time between switching. Our theory unifies several theoretical effects - the ACT-R cognitive architecture, the threaded cognition theory of concurrent multitasking, and the memory-for-goals theory of interruption and resumption - to better understand and predict multitasking behavior. We outline the theory and discuss how it accounts for numerous phenomena in the recent empirical literature.

As driver distraction from in-vehicle devices becomes an increasingly critical issue, researchers have aimed to establish better scientific understanding of distraction along with better engineering tools to build less distracting devices. This article presents a new system, Distract-R, that allows designers to rapidly prototype and evaluate new in-vehicle interfaces. The core engine of the system relies on a rigorous cognitive model of driver behavior which, when integrated with models of task behavior on the prototyped interfaces, generate predictions of driver performance and distraction. Distract-R allows a designer to prototype basic interfaces, demonstrate possible tasks on these interfaces, specify relevant driver characteristics and driving scenarios, and finally simulate, visualize, and analyze the resulting behavior as generated by the cognitive model. The article includes three modeling studies that demonstrate the system's ability to account for various aspects of driver performance for several types of in-vehicle interfaces. More generally, Distract-R illustrates how cognitive models can be used as internal simulation engines for design tools intended for nonmodelers, with the ultimate goal of helping to understand and predict user behavior in multitasking environments.

The paper describes an approach to modeling the strategic variations in performing secondary tasks while driving. In contrast to previous efforts that are based on simulation of a cognitive architecture interacting with a task environment, we take an approach that develops a cognitive constraint model of the interaction between the driver and the task environment in order to make inferences about dual-task performance. Analyses of driving performance data reveal that a set of simple equations can be used to accurately model changes in the lateral position of the vehicle within the lane. The model quantifies how the vehicle's deviation from lane center increases during periods of inattention, and how the vehicle returns to lane center during periods of active steering. We demonstrate the benefits of the approach by modeling the dialing of a cellular phone while driving, where drivers balance the speed in performing the dial task with accuracy (or safety) in keeping the vehicle centered in the roadway. In particular, we show how understanding, rather than simulating, the constraints imposed by the task environment can help to explain the costs and benefits of a range of strategies for interleaving dialing and steering. We show how particular strategies are sensitive to a combination of internal constraints (including switch costs) and the trade-off between the amount of time allocated to secondary task and the risk of extreme lane deviation.

Portable music players such as Apple's iPod have become ubiquitous in many environments, but one environment in particular has elicited new safety concerns and challenges -- in-vehicle use while driving. We present the first study of portable music-player interaction while driving, examining the effects of iPod interaction by drivers navigating a typical roadway in a driving simulator. Results showed that selecting media on the iPod had a significant effect on driver performance as measured by lateral deviation from lane center; the effect was comparable to previously reported effects of dialing a cellular phone. In addition, selecting media and watching videos had a significant effect on car-following speed, resulting in speed reductions that presumably compensated for impaired lateral performance. Given that iPod interaction has become increasingly common while driving, these results serve as a first step toward understanding the potential effects of portable music-player interaction on driver behavior and performance.

As driver distraction from in-vehicle devices increasingly becomes a concern on our roadways, researchers have searched for better scientific understanding of distraction along with better engineering tools to build less distracting devices. This paper presents a new system, Distract-R, that allows designers to rapidly prototype and evaluate new in-vehicle interfaces. The core engine of the system relies on a rigorous cognitive model of driver performance, which the system integrates with models of behavior on the prototyped interfaces to generate predictions of distraction. Distract-R allows a designer to prototype basic interfaces, demonstrate possible tasks on these interfaces, specify relevant driver characteristics and driving scenarios, and finally simulate, visualize, and analyze the resulting behavior as generated by the cognitive model. The paper includes two sample studies that demonstrate the system's ability to account for effects of input modality and driver age on performance.

Although engineering models of user behavior have enjoyed a rich history in HCI, they have yet to have a widespread impact due to the complexities of the modeling process. In this paper we describe a development system in which designers generate predictive cognitive models of user behavior simply by demonstrating tasks on HTML mock-ups of new interfaces. Keystroke-Level Models are produced automatically using new rules for placing mental operators, then implemented in the ACT-R cognitive architecture. They interact with the mock-up through integrated perceptual and motor modules, generating behavior that is automatically quantified and easily examined. Using a query-entry user interface as an example [19], we demonstrate that this new system enables more rapid development of predictive models, with more accurate results, than previously published models of these tasks.

Cognitive modeling has evolved into a powerful tool for understanding and predicting user behavior. Higher-level modeling frameworks such as GOMS and its variants facilitate fast and easy model development but are sometimes limited in their ability to model detailed user behavior. Lower-level cognitive architectures such as EPIC, ACT-R, and Soar allow for greater precision and direct interaction with real-world systems but require significant modeling training and expertise. In this paper we present a modeling framework, ACT-Simple, that aims to combine the advantages of both approaches to cognitive modeling. ACT-Simple embodies a "compilation" approach in which a simple description language is compiled down to a core lower-level architecture (namely ACT-R). We present theoretical justification and empirical validation of the usefulness of the approach and framework.

When designing and evaluating in-car user interfaces for drivers, it is essential to determine what effects these interfaces may have on driver behavior and performance. This paper describes a novel approach to predicting effects of in-car interfaces by modeling behavior in a cognitive architecture. A cognitive architecture is a theoretical frame-work for building computational models of cognition and performance. The proposed approach centers on integrating a user model for the interface with an existing driver model that accounts for basic aspects of driver behavior (e.g., steering and speed control). By running the integrated model and having it interact with the interface while driving, we can generate a priori predictions of the effects of interface use on driver performance. The paper illustrates the approach by comparing four representative dialing interfaces for an in-car, hands-free cellular phone. It also presents an empirical study that validates several of the qualitative and quantitative predictions of the model.

This article describes and evaluates a class of methods for performing automated analysis of eye-movement protocols. Although eye movements have become increasingly popular as a tool for investigating user behavior, they can be extremely difficult and tedious to analyze. In this article we propose an approach to automating eye-movement protocol analysis by means of tracing-relating observed eye movements to the sequential predictions of a process model. We present three tracing methods that provide fast and robust analysis and alleviate the equipment noise and individual variability prevalent in typical eye-movement protocols. We also describe three applications of the tracing methods that demonstrate how the methods facilitate the use of eye movements in the study of user behavior and the inference of user intentions.

We discuss a novel type of interface, the intelligent gaze-added interface, and describe the design and evaluation of a sample gaze-added operating-system interface. Gaze-added interfaces, like current gaze-based systems, allow users to execute commands using their eyes. However, while most gaze-based systems replace the functionality of other inputs with that of gaze, gaze-added interfaces simply add gaze functionality that the user can employ if and when desired. Intelligent gaze-added interfaces utilize a probabilistic algorithm and user model to interpret gaze focus and alleviate typical problems with eye-taking data. We extended a standard WIMP operating-system interface into a new interface, IGO, that incorporates intelligent gaze-added input. In a user study, we found that users quickly adapted to the new interface and utilized gaze effectively both alone and with other inputs.

This paper describes EyeTracer, an interactive environment for manipulating, viewing, and analyzing eye-movement protocols. EyeTracer augments the typical functionality of such systems by incorporating model-based tracing algorithms that interpret protocols with respect to the predictions of a cognitive process model. These algorithms provide robust strategy classification and fixation assignment that help to alleviate common difficulties with eye-movement data, such as equipment noise and individual variability. Using the tracing algorithms for analysis and visualization, EyeTracer facilitates both exploratory analysis for initial understanding of behavior and confirmatory analysis for model evaluation and refinement.

The process of fixation identification—separating and labeling fixations and saccades in eye-tracking protocols—is an essential part of eye-movement data analysis and can have a dramatic impact on higher-level analyses. However, algorithms for performing fixation identification are often described informally and rarely compared in a meaningful way. In this paper we propose a taxonomy of fixation identification algorithms that classifies algorithms in terms of how they utilize spatial and temporal information in eye-tracking protocols. Using this taxonomy, we describe five algorithms that are representative of different classes in the taxonomy and are based on commonly employed techniques. We then evaluate and compare these algorithms with respect to a number of qualitative characteristics. The results of these comparisons offer interesting implications for the use of the various algorithms in future work.

While current eye-based interfaces offer enormous potential for efficient human-computer interaction, they also manifest the difficulty of inferring intent from user eye movements. This paper describes how fixation tracing facilitates the interpretation of eye movements and improves the flexibility and usability of eye-based interfaces. Fixation tracing uses hidden Markov models to map user actions to the sequential predictions of a cognitive process model. In a study of eye typing, results show that fixation tracing generates significantly more accurate interpretations than simpler methods and allows for more flexibility in designing usable interfaces. Implications for future research in eye-based interfaces and multimodal interfaces are discussed.