We present empirical results from a study conducted with eleven users with upper-body motor impairments who imagined input devices and corresponding gestures to operate them for performing common tasks in interactive systems. We report a strong preference for embodying devices (80%), primarily through the hands, rather than holding them, and identify ten device archetypes, among which smartphones (36.4%) and remote controls (27.3%) were most prevalent. We also observed a diversity of gestures to operate imagined devices involving both unimanual and bimanual input with little consensus (.069) across participants, which we analyzed in relation to their self-reported motor impairments. Based on these findings, we propose design recommendations for accessible interactions involving imagined input devices, structured through the lenses of ability-based and ability-mediating design. We also outline future work opportunities for imagination-powered accessible computing, in which users’ imagination plays a central role.

The podcasting landscape is undergoing a significant transformation due to the integration of generative AI, which is reconfiguring creators’ roles and how podcasts are experienced by listeners. In this paper, we present findings from a corpus of scientific articles identified in the ACM Digital Library and IEEE Xplore, which examine the intersection of podcasting and AI. Based on our analysis, two pillars emerge for AI integration into podcasting: (i) AI as a productivity tool, supporting human creators through smart editing and analytics, and (ii) AI as an autonomous content generator, enabled by agent-based processing, zero-shot audio generation, and high personalization in podcast delivery and consumption. These two pillars provide an account of how scientific contributions position AI within the podcasting landscape, and surface a transition from “podcasting then,” as predominantly handcrafted and human-produced media, toward “podcasting now,” as an increasingly AI-driven and personalizable audio experience.

Figure 1: Examples of UI designs generated with the text prompt "Design a [application type] interface," showcasing a settings menu in a fitness app (left), a search results screen in a recipe app (middle), and a dashboard in a financial app (right).Note the strong compliance with the Web Content Accessibility Guidelines (WCAG), yet the highly uniform design themes.

Figure 1: Photograph captured during one of our missions at the Mars Desert Research Station, depicting an environment characterized by extreme constraints, isolation, and confinement, with harsh conditions that can affect physical and psychological well-being.Note the deserted landscape, the strong sunlight, and the bulky equipment.In this setting, closely mimicking a Mars analog environment, we conducted two user experience studies exploring various pragmatic and hedonic dimensions of interaction with computer systems, with a baseline represented by user experience measured in a conventional environment.

Haptic feedback, commonly experienced as vibrotactile cues on mobile devices, increases user performance and enhances user experience, but research addressing users with upper-body motor impairments remains scarce.In particular, variations in how touch input is performed across different motor abilities raise questions about the optimal location, on-device or on-body, for vibrotactile feedback to maximize its effectiveness.In this work, we apply design thinking to explore alternative approaches for delivering haptic feedback at locations distant from the on-screen touch point, such as the user's hand, wrist, forearm, or even the other arm.To inform our design explorations, we leverage empirical findings from a dataset of touch gestures performed on mobile devices by users with various upper-body motor impairments.We present future research opportunities at the intersection of haptic technology, wearable devices, accessible computing, and touchscreen input.

Vibrotactile feedback is known to have a positive impact on the user performance and experience of touchscreen input. Unfortunately, not all touchscreen devices have built-in haptics capabilities, public displays being a notable example, while the design options for vibrotactile feedback on conventional mobile devices, such as smartphones and tablets, are still limited. In this work, we propose leveraging commodity wearable devices, such as smartwatches, rings, and armbands, to enable vibrotactile feedback as closely as possible to the touch point when the touchscreen does not natively support haptics, and to complement existing haptics capabilities when the touchscreen already features them. To this end, we introduce Wear + Touch, a set of four interaction techniques designed to augment touchscreen input with vibrations delivered through wearable devices, from substituting to complementing, extending, and expanding on-screen vibrotactile feedback. We demonstrate Wear + Touch with several practical implementations involving a large touchscreen display, smartphone, tablet, smartwatch, smart ring, and armband as well as a custom wearable device that we designed for vibrations delivered simultaneously on the user’s index finger, wrist, and forearm. We also report empirical results from two controlled experiments involving a total of 40 participants, conducted to understand the multi-faceted user experience of vibrotactile feedback delivered through the Wear + Touch techniques. In the first experiment, we reveal a favorable user experience of vibrations delivered on the index finger, wrist, and forearm, in this order, when the touchscreen lacks haptics capabilities. In the second experiment, we reveal benefits of reinforcing and expanding on-screen vibrations with vibrotactile feedback on the index finger, wrist, and arm, respectively. We use our findings to suggest future work opportunities for combined vibrotactile feedback involving both touchscreens and wearable devices.

The evolving landscape of distributed user interfaces requires the prototyping stage also be distributed between users, tasks, platforms, and environments. To create a cohesive distribution of the user interface elements in such ecosystems, paired sketching has emerged as a collaborative design method that leverages multiple stakeholders’ strengths, including designers, developers, and end users, working in pairs. In the context of developer experience applied to paired sketching for distributed user interfaces, we decomposed a workflow into four disciplines according to the Software and Systems Process Engineering Meta-Model (SPEM) notation. First, we defined a protocol to deploy paired sketching of distributed user interfaces, supported by UbiSketch , a collaborative software environment tailored featuring sketch recognition and whiteboarding. Second, to evaluate paired sketching for engineering interactive systems, we conducted an experiment involving five pairs of stakeholders who sketched a distributed user interface for inside-the-vehicule interaction distributed on four platforms: smartphone, tablet, pen display, and tabletop. Empirical results from questionnaires, reactivity, intention, perceived satisfaction, and free comments, suggest a preference order in which the tabletop is ranked first, followed by the tablet, smartphone, and pen display. Based on these results, we discuss the potential of paired sketching for distributed user interfaces.

Isolated, Confined, and Extreme (ICE) environments, such as encountered in space exploration missions, pose unique physical and psychological challenges that influence user interactions with computer systems, yet remain considerably less documented compared to conventional settings. To investigate the impact of such environments on mobile interaction, we conducted an experiment involving a crew of analog astronauts operating a drone via a handheld controller in both a conventional Earth-based setting and an ICE environment represented by the extreme landscape of the Mars Desert Research Station. Our findings reveal how the user experience of mobile interaction evolves over multiple evaluation sessions conducted over a two-week period in the ICE environment, for which we analyze both pragmatic and hedonic dimensions, such as perceived efficiency, adaptability, novelty, usefulness, and trust. Based on our findings, we outline a set of implications for the design of mobile interaction intersecting space research through the distinctive lens of astronaut-drone interaction.

Decades ago, the natural interaction design paradigm has brought about a revolutionary shift in perspective regarding how interactions with computer systems could and should take place. By promoting intuition, familiarity, and context matching, natural interactions leverage users’ best capabilities to engage in effective dialogue with their computer interlocutors. Natural interaction has prevailed and transpires today in the way we use gestures, voice, or eye gaze to interact with mobile and wearable devices, communicate with personal assistants, play video games through whole-body movements, and engage with public ambient displays.

We explore the chair as a referential frame for facilitating hand gesture input to control interactive systems. First, we conduct a Systematic Literature Review on the topic of interactions supported by chairs, and uncover little research on harnessing everyday chairs for input, limited to chair rotation and tilting movements. Subsequently, to understand end users’ preferences for gestures performed on the chair’s surface (i.e., on-chair gestures) and in the space around the chair (i.e., from-chair gestures), we conduct an elicitation study involving 54 participants, 3 widespread chair variations—armchair, office-chair, and stool,—and 15 referents encompassing common actions, digital content types, and navigation commands for interactive systems. Our findings reveal a preference for unimanual gestures implemented with strokes, hand poses, and touch input, with specific nuances and kinematic profiles according to the chair type. Based on our findings, we propose a range of implications for interactive systems leveraging on-chair and from-chair gestures.

AI techniques have always been central to multimodal interaction, being leveraged for sensing user intention, action, and emotion and for generating adaptive feedback. In this process, AI has played the role of enabling technology toward making interactions with computer systems and computerized environments more efficient, accessible, and natural through the prism of multimodality. In this work, we argue that AI can play yet another key role in multimodal interaction by implementing distinctive modalities complementing those accessible by humans on their own. The result is multimodal interaction through human-AI integration, where multimodality is distributed across users and AI embodiments of various form factors, from robotic assistants to smart wearables to ambient devices. The new concept of “AI as modality,” both contrasting and complementing the established “AI as tool,” is particularly relevant for recent interaction paradigms involving human-computer integration and human augmentation. In this paper, we outline the vision of leveraging AI for delivering distinctive modalities toward the next generation of multimodal interactive experiences.

We report results from a systematic literature review on the topic of reward systems for older people engaging with serious games, involving an analysis of 121 scientific articles identified from multiple bibliographic databases. Our analysis, employing various taxonomies of game type, genre and challenges, reward types and features, and interaction devices and feedback modalities, reveals preponderant attention on visual feedback, sports games, and time pressure/reaction challenges, but limited scientific research on unlocking mechanisms, achievement, item-granting rewards, haptic feedback modalities, or multi-player scenarios in serious games designed for older adults. These findings reveal reward mechanisms as a distinct type of interactive media featuring some, but ignoring many design possibilities. Based on our findings, we propose several recommendations for future research on designing reward mechanisms in serious games that adapt to users’ varying abilities, favor a diversity of input and output modalities, and expand to multi-player and multi-genre setups.

Modern in-vehicle infotainment systems connect to drivers’ personal digital devices, such as smartphones, to facilitate calls, notifications delivery, and access to online media and resources while driving. This communications process facilitates distributed user interfaces (DUIs) and allows casual in-vehicle interactions. However, such functionalities of modern vehicles still lack the extensive capabilities of state-of-the-art DUIs, which can offer a variety of interaction opportunities through features such as migration, redirection, adaptation, and granularity, among others. In this work, we explore DUIs in the specific context of smart vehicles, and introduce V2DUIs (Vehicle-to-DUIs) as a new addition to the V2X (Vehicle-to-Everything) family of smart vehicle concepts.

We introduce ChairMX, on-chair input technology for novel interactive media consumption experiences through the chair. ChairMX is the end result of an elaborate four-stage research and development process involving a systematic literature review of the extent in which chairs have been used in interactive systems, involvement of potential end users to elicit their preferences for on-chair input, engineering on-chair gesture input recognition technology with wearables, and exploration of application opportunities. By leveraging a ring-like form factor wearable, ChairMX is a readily deployable technology, working for uninstrumented chairs everywhere and enabling a variety of use case scenarios involving interactive media consumption, from the living room to public environments.

We address in this work the topic of broadcasting one’s visual reality, captured by the video cameras embedded in mobile and wearable devices, to a remote audience. We discuss several designs of such life broadcasting systems, which we position at the intersection of lifelogging, Alternate Reality, and Cross-Reality technology. To this end, we introduce the “Alternate Reality Broadcast-Time” matrix for the broadcasting and consumption of alternate realities, in which designs of systems that implement sharing and consumption of personal visual realities can be positioned, characterized, and compared. These design options range from simple video streaming over the web using conventional video protocols to mediated and augmented reality, to audio narration and vibrotactile rendering of concepts automatically detected from video captured by wearable cameras. To demonstrate the usefulness of our broadcast-time matrix, we describe three prototypes implementing lifelogging, concept recognition from video, augmented and mediated vision, and vibrotactile feedback. Our contributions open the way toward new applications that blend lifelogging, consumption of multimedia alternate realities, and XR technology to empower users with richer opportunities for self-expression and new means to connect with the followers of events in their lives.

We present empirical results from a gesture elicitation study conducted with eleven wheelchair users that proposed on-body, in-air, and on-wheelchair gestures to effect twenty-one referents representing common actions, types of digital content, and navigation commands for interactive systems. We report a large preference for on-body (47.6%) and in-air (40.7%) compared to on-wheelchair (11.7%) gestures, mostly represented by touch input on different parts of the body and hand poses performed in mid-air with one hand. Following an agreement analysis that revealed low consensus (≤ 5.5%) between users, although high perceived gesture ease, goodness, and social acceptability within users, we examine our participants’ gesture characteristics in relation to their self-reported motor impairments, e.g., low strength, rapid fatigue, etc. We highlight the need for personalized gesture sets, tailored to and reflective of both users’ preferences and specific motor abilities, an implication that we examine through the lenses of ability-based design.

We present “fingerhints,” on-finger kinesthetic feedback represented by hyper-extension movements of the index finger, bypassing user agency, for notifications delivery. To this end, we designed a custom-made finger-augmentation device, which leverages mechanical force to deliver fingerhints as programmable hyper-extensions of the index finger. We evaluate fingerhints with 21 participants, and report good usability, low technology creepiness, and moderate to high social acceptability. In a second study with 11 new participants, we evaluate the wearable comfort of our fingerhints device against four commercial finger- and hand-augmentation devices. Finally, we present insights from the experience of one participant, who wore our device for eight hours during their daily life. We discuss the user experience of fingerhints in relation to our participants’ personality traits, finger dexterity levels, and general attitudes toward notifications, and present implications for interactive systems leveraging on-finger kinesthetic feedback for on-body computing.

We examine gestures performed with a class of input devices with distinctive quality properties in the wearables landscape, which we call “index-Finger Augmentation Devices” (iFADs). We introduce a four-level taxonomy to characterize the diversity of iFAD gestures, evaluate iFAD gesture articulation on a dataset of 6,369 gestures collected from 20 participants, and compute recognition accuracy rates. Our findings show that iFAD gestures are fast (1.84s on average), easy to articulate (1.52 average rating on a difficulty scale from 1 to 5), and socially acceptable (81% willingness to use them in public places). We compare iFAD gestures with gestures performed using other devices (styli, touchscreens, game controllers) from several public datasets (39,263 gestures, 277 participants), and report that iFAD gestures are two times faster than whole-body gestures and as fast as stylus and finger strokes performed on touchscreens.

The consumption of digital audiovisual media is a mainstay of many people’s lives. However, people with accessibility needs often have issues accessing this content. With a view to addressing this inequality, there exists a wide range of interventions that researchers have explored to bridge this accessibility gap. Despite this work, our understanding of the capability of these interventions is poor. In this paper, we address this through a systematic review of the literature, creating a dataset of and analysing N = 181 scientific papers. We have found that certain areas have accrued a disproportionate amount of attention from the research community – for example, blind and visually impaired and d/Deaf and hard of hearing people account for of papers (N = 170). We describe challenges researchers have addressed, end-user communities of focus, and interventions examined. We conclude by evaluating gaps in the literature and areas that could use more focus on in the future.

We introduce GearWheels, a software tool for studies about gesture input with wearables, including smartwatches, rings, and glasses. GearWheels features an event-based asynchronous software architecture design implemented exclusively with web standards, communications protocols, and data formats, which makes it flexible to support many wearables via HTTP and WebSocket communications. GearWheels differentiates from prior software tools for gesture acquisition, elicitation, recognition, and analysis with its web-based, wearable-oriented, experiment-centered architecture design. We demonstrate GearWheels with a device affixed to the index finger, wrist, and the temple of a pair of glasses to illustrate touch stroke-gesture and motion-gesture input acquisition. We also perform a technical evaluation of GearWheels in the form of a simulation experiment, and report the request-response time performance of the software components of GearWheels with off-the-shelf wearables. We release GearWheels as open source software to assist researchers and practitioners in implementing studies about gesture input with wearables.

We introduce VIREO, a web-based software tool for graphical authoring of vibrotactile feedback for mobile and wearable applications. VIREO enables flexible specification of vibrotactile patterns with model-based and free-draw input, and is compatible with devices that run JavaScript, either natively or in a web browser. We demonstrate VIREO with applications developed for smartphones, smartwatches, armbands, and smartglasses, and we present the results of a usability evaluation study with sixteen participants represented by coders with various programming experience. We discuss our contributions in the context of the results of a Systematic Literature Review conducted on the topic of software tools, editors, and platforms developed in the scientific community for authoring vibrotactile feedback. Given that one finding of our review is the little availability of such contributions, we release VIREO as a free resource on the web for researchers and practitioners to author and integrate vibrotactile feedback in mobile and wearable applications.

Microwave radars bring many benefits to mid-air gesture sensing due to their large field of view and independence from environmental conditions, such as ambient light and occlusion. However, radar signals are highly dimensional and usually require complex deep learning approaches. To understand this landscape, we report results from a systematic literature review of ( N =118) scientific papers on radar sensing, unveiling a large variety of radar technology of different operating frequencies and bandwidths and antenna configurations but also various gesture recognition techniques. Although highly accurate, these techniques require a large amount of training data that depend on the type of radar. Therefore, the training results cannot be easily transferred to other radars. To address this aspect, we introduce a new gesture recognition pipeline that implements advanced full-wave electromagnetic modeling and inversion to retrieve physical characteristics of gestures that are radar independent, i.e., independent of the source, antennas, and radar-hand interactions. Inversion of radar signals further reduces the size of the dataset by several orders of magnitude, while preserving the essential information. This approach is compatible with conventional gesture recognizers, such as those based on template matching, which only need a few training examples to deliver high recognition accuracy rates. To evaluate our gesture recognition pipeline, we conducted user-dependent and user-independent evaluations on a dataset of 16 gesture types collected with the Walabot, a low-cost off-the-shelf array radar. We contrast these results with those obtained for the same gesture types collected with an ultra-wideband radar made of a vector network analyzer with a single horn antenna and with a computer vision sensor, respectively. Based on our findings, we suggest some design implications to support future development in radar-based gesture recognition.

The body of knowledge accumulated by gesture elicitation studies (GES), although useful, large, and extensive, is also heterogeneous, scattered in the scientific literature across different venues and fields of research, and difficult to generalize to other contexts of use represented by different gesture types, sensing devices, applications, and user categories. To address such aspects, we introduce RepliGES, a conceptual space that supports (1) replications of gesture elicitation studies to confirm, extend, and complete previous findings, (2) reuse of previously elicited gesture sets to enable new discoveries, and (3) extension and generalization of previous findings with new methods of analysis and for new user populations towards consolidated knowledge of user-defined gestures. Based on RepliGES, we introduce GEStory, an interactive design space and visual tool, to structure, visualize and identify user-defined gestures from a number of 216 published gesture elicitation studies.

We show how two recently introduced visual tools, RepliGES and GEStory, can be used conjointly to inform possible replications of Gesture Elicitation Studies (GES) with a case study centered on gestures that can be sensed with radars. Starting from a GES identified in GEStory, we employ the dimensions of the RepliGES space to enumerate eight possible ways to replicate that study towards gaining new insights into end user’s preferences for gesture-based interaction for applications that use radar sensors.

We examine touchscreen stroke-gestures and mid-air motion-gestures articulated by users with upper-body motor impairments with devices worn on the wrist, finger, and head. We analyze users’ gesture input performance in terms of production time, articulation consistency, and kinematic measures, and contrast the performance of users with upper-body motor impairments with that of a control group of users without impairments. Our results, from two datasets of 7,290 stroke-gestures and 3,809 motion-gestures collected from 28 participants, reveal that users with upper-body motor impairments take twice as much time to produce stroke-gestures on wearable touchscreens compared to users without impairments, but articulate motion-gestures equally fast and with similar acceleration. We interpret our findings in the context of ability-based design and propose ten implications for accessible gesture input with upper-body wearables for users with upper-body motor impairments.

We introduce “Sensorimotor Realities,” a new concept in the XR landscape and corresponding technology-agnostic framework for computer-mediated perception and motor action. Sensorimotor Realities capitalize on the heterogeneity of human sensorimotor abilities to support conceptualization, characterization, and design of computer technology that leverages existing abilities in new, computer-mediated worlds. We introduce a conceptual space for Sensorimotor Realities with six dimensions, discuss examples of interactive systems in this space, and show how Sensorimotor Realities are distinct in nature and goal from Augmented, Mixed, Virtual, and Mediated Reality. We capitalize on Sensorimotor Realities to propose “ability-mediating design,” an approach to designing accessible interactive computer systems that complements ability-based design. We discuss how Sensorimotor Realities offer new opportunities for research and development at the intersection of XR, wearable computing, ambient intelligence, and accessible computing, and draw a research roadmap with three major directions of scientific investigation.

Augmented Reality Television (ARTV) can take many forms, from AR content displayed outside the TV frame to video-projected TV screens to social TV watching in VR to immersive holograms in the living room. While the user experience (UX) of individual forms of ARTV has been documented before, “journeys” as transitions between such forms have not. In this work, we examine the UX of watching TV when switching between various levels of augmentation. Our findings from an experiment with fourteen participants reveal an UX characterized by high perceived usability, captivation, and involvement with a low to medium workload and a moderate feeling of dissociation from the physical world. We interpret our results in the context of Garrett’s established five-plane model of UX—strategy, scope, structure, skeleton, and surface—and propose a sixth plane, “switch,” which separates conceptually the design of user journeys in ARTV from the specifics of the other UX planes.

We address gesture input for TV control, for which we examine mid-air free-hand interactions that can be detected via radar sensing. We adopt a scenario-based design approach to explore possible locations from the living room where to integrate radar sensors, e.g., in the TV set, the couch armrest, or the user’s smartphone, and we contribute a four-level taxonomy of locations relative to the TV set, the user, personal robot assistants, and the living room environment, respectively. We also present preliminary results about an interactive system using a 15-antenna ultra-wideband 3D radar, for which we implemented a dictionary of six directional swipe gestures for the control of dichotomous TV system functions.

We examine accessible interactions for wheelchair users and public displays with three studies. In a first study, we conduct a Systematic Literature Review, from which we report very few scientific papers on this topic and a preponderant focus on touch input. In a second study, we conduct a Systematic Video Review using YouTube as a data source, and unveil accessibility challenges for public displays and several input modalities alternative to direct touch. In a third study, we conduct semi-structured interviews with eleven wheelchair users to understand their experience interacting with public displays and to collect their preferences for more accessible input modalities. Based on our findings, we propose the “assisted interaction” phase to extend Vogel and Balakrishnan’s four-phase interaction model with public displays, and the “ability” dimension for cross-device interaction design to support, via users’ personal mobile devices, independent use of interactive public displays.

We clarify fundamental aspects of end-user elicitation, enabling such studies to be run and analyzed with confidence, correctness, and scientific rigor. To this end, our contributions are multifold. We introduce a formal model of end-user elicitation in HCI and identify three types of agreement analysis:expert,codebook, andcomputer. We show that agreement is a mathematicaltolerance relationgenerating a tolerance space over the set of elicited proposals. We review current measures of agreement and show that all can be computed from anagreement graph. In response to recent criticisms, we show that chance agreement represents an issue solely for inter-rater reliability studies and not for end-user elicitation, where it is opposed bychance disagreement. We conduct extensive simulations of 16 statistical tests for agreement rates, and report Type I errors and power. Based on our findings, we provide recommendations for practitioners and introduce a five-level hierarchy for elicitation studies.

We conduct a systematic literature review on wearable interactions for users with motor impairments and report results from a meta-analysis of 57 scientific articles identified in the ACM DL and IEEE Xplore databases. Our findings show limited research conducted on accessible wearable interactions (e.g., just four papers addressing smartwatch input), a disproportionate interest for hand gestures compared to other input modalities for wearable devices, and low numbers of participants with motor impairments involved in user studies about wearable interactions (a median of 6.0 and average of 8.2 participants per study). We compile an inventory of 92 finger, hand, head, shoulder, eye gaze, and foot gesture commands for smartwatches, smartglasses, headsets, earsets, fitness trackers, data gloves, and armband wearable devices extracted from the scientific literature that we surveyed. Based on our findings, we propose four directions for future research on accessible wearable interactions for users with motor impairments.

Immersive experiences – enabled by technologies such as VR, AR, 360° video and other highly immersive multimedia applications – have the potential to make interacting with various activities more inclusive for many people. This can be achieved by applying the principles of inclusive design. This panel will discuss the current challenges in designing inclusive immersive technologies, and how they should be addressed.

We present XR4ISL, an XR system designed to support psychology experiments examining Implicit Social Learning, a fundamental phenomenon that guides human behavior, cognition, and emotion. We discuss XR4ISL with reference to MR4ISL, a previous system designed for Mixed Reality only, and reflect on differences between Mixed and Virtual Reality for psychology experiments.

We employ the ISO/IEC 25010 standard for systems and software quality requirements and evaluation to present a proposal of a software architecture design for gesture input with two types of wearables: smartwatches, which have already entered mainstream, and smartglasses, forecast to become a growing market due to increased adoption of augmented reality applications. We present the technical details of our software architecture and describe practical implementation aspects that employ web standards.

As Augmented Reality television gains momentum, it is important to understand whether cultural differences among viewers favor different expectations and preferences for immersion in such new television environments. A previous study documented the preferences of 172 participants from various European countries for twenty application scenarios for ARTV, such as virtual objects coming out of the TV screen into the room. In this work, we conduct an empirical generalization of this previous study to understand potential cultural differences in users’ preferences for and expectations of ARTV. To this end, we report insights from data collected from a sample of 147 participants from China, which we compare against the preferences expressed by the participants from Europe from the original study. Our findings reveal similarities, but also differences in terms of expectations of ARTV across the two cultural groups. We draw implications for future research on culturally-aware augmentations of the television watching experience.

Personal drones are more and more present in our lives and acting as “flying cameras” is one of their most prominent applications. In this work, we conduct a synopsis of the scientific literature on human-drone interaction to identify system functions and corresponding commands for controlling drone-based aerial photography and video, from which we compile a dictionary of interactions. We also discuss opportunities for more research at the intersection of drone computing, augmented vision, and personal photography.

We conduct an examination of the accessibility challenges experienced by people with upper body motor impairments when interacting with television. We report findings from a study with N=41 people with motor impairments (spinal cord injury, cerebral palsy, muscular dystrophy) and document their challenges and coping strategies for using the TV remote control, but also their television watching experience and expectations of suitable assistive technology for television. Our results show that, despite several accessible remote control products available on the market, the majority of our participants preferred to DIY and hack, and to adopt coping strategies to be able to use conventional TV remote controls. We contrast their experience against that of a control group with N=41 people without impairments. We reflect about the DIY culture and people with motor impairments, and we propose future work directions to increase the accessibility of interactions with television.

We conduct an overview of the landscape of scientific research falling at the intersection of television, immersive media, and accessible interactive technology. To this end, we consider for our analysis a number of 449 papers published at IMX, TVX, and EuroITV between 2003 and 2020, of which we report a total of 19 papers (4.23%) addressing users with disabilities and only 9 (2.00%) actually involving people with disabilities as participants in user studies. We analyze the topics and research contributions of these papers, and draw conclusions about the extent to which accessibility research has been present in the IMX/TVX community.

Despite an exciting area with many promises for innovations in wearable interactive systems, research on interaction techniques for smart rings lacks structured knowledge and readily-available resources for designers to systematically attain such innovations. In this work, we conduct a systematic literature review of ring-based gesture input, from which we extract key results and a large set of gesture commands for ring, ring-like, and ring-ready devices. We use these findings to deliver GestuRING, our web-based tool to support design of ring-based gesture input. GestuRING features a searchable gesture-to-function dictionary of 579 records with downloadable numerical data files and an associated YouTube video library. These resources are meant to assist the community in attaining further innovations in ring-based gesture input for interactive systems.

We introduce “4E,” a new design approach of reconfigurable displays that can change their form factors by capitalizing on four quality properties inspired by applied material: extensibility, extendability, expandability, and extractability. This approach is applicable to both fixed and portable displays. We define and exemplify each property, highlighting the key differences in how reconfigurable displays can change their form factors to accommodate more screen real estate for more users, applications, and functionality. To demonstrate the 4E approach, we conduct a targeted literature review and introduce E3Screen, a prototype that enhances any flat screen (e.g., of a tablet, laptop, monitor) with two slidable, rotatable, and foldable lateral displays. We report results from a controlled experiment with N=103 participants, conducted to collect, analyze, and understand end users’ preferences for display configurations permitted by the extendability, expandability, and extractability of E3Screen. Our results structure future research and development in reconfigurable displays and multi-display collaborative workspaces.

Abstract Software architecture and applications for the connected car can process and share a variety of digital content, among which high-definition video and augmented reality (AR) content, toward enhanced driving assistance, navigation and infotainment systems and services. However, several technical challenges need to be overcome to make such systems and services viable and efficient, including dealing effectively with a variety of types of systems, devices and platforms, either installed inside the vehicle or represented by the personal mobile and wearable devices of the drivers and passengers. In this paper, we outline these technical challenges and propose a software solution in the form of an event-based middleware layer by modeling the smart, connected car as a specific type of a smart environment. We employ an adapted version of Euphoria, a recently introduced software architecture for general-purpose smart environments, to implement asynchronous communications among heterogeneous input/output devices inside the vehicle. We also adapt Euphoria to fit into the four-layer infrastructure model of the connected car. We conduct a technical evaluation of the request-response time performance achieved with the Euphoria middleware for streaming digital content from 1 Mbps (360p@30fps) to 32 Mbps (4K@30fps) on various devices, either integrated in the vehicle, not integrated but used inside the vehicle and devices outside the vehicle (the control condition). Our results show effective live streaming achieved for 2K content at 30fps with the 600 Mbps network (i.e., the connected car) and for 4K content at 30fps with the 1.7 Gbps network envisioned for hyper-connected vehicles. These results open up opportunities for high-definition video and AR applications in the automotive industry.

Smart eyewear and Augmented Reality technology have been examined closely in the scientific community to provide vision rehabilitation to people with visual impairments as well as augmented vision to people with and without visual impairments for various application scenarios and contexts of use. However, current systems lack flexibility in the configuration and customization of the features and functionalities they present to their users, as we show in this paper by means of a thorough literature review and categorization of prior work on augmented and mediated vision for smart eyewear devices. To address the flexibility aspect that has been missing in prior work, we introduce FlexiSee, an application for smart eyewear devices, such as see-through Augmented Reality glasses and Head-Mounted Mixed Reality Displays, specifically designed to enable flexible configuration, customization, and control of both augmented and mediated vision. FlexiSee achieves this desiderata by implementing visual filters (e.g., color correction, edge highlighting, contrast adjustment, and others) that are coupled with a web-based interface, readily accessible from smartphones, tablets, smartwatches, and other devices with web browsers, where authorized users can specify and apply custom parameters for the visual filters implemented by FlexiSee. We also introduce FlexiSee-DS, a three-dimensional design space for FlexiSee-like applications, that includes mediation & augmentation, user categories, and control design dimensions to specify a variety of FlexiSee-like systems. We show how the dimensions of FlexiSee-DS were applied to inform the design of our FlexiSee system, and we highlight and focus on the distinction between primary users and vision monitors and assistants, where the latter two categories represent new types of users for augmented and mediated vision that have various degrees of control, from their remote locations, over the visual reality delivered to and perceived by the primary users of smart eyewear devices. We conduct a user study to understand the perception of vision monitors and assistants regarding our new FlexiSee concept and system, and we report empirical results about usability aspects (e.g., we found an average SUS score of 75.3 and high ratings for the perceived usefulness of FlexiSee) as well as user feedback and suggestions to inform further developments of FlexiSee-like systems and applications.

Intra-platform plasticity regularly assumes that the display of a computing platform remains fixed and rigid during interactions with the platform in contrast to reconfigurable displays, which can change form depending on the context of use. In this paper, we present a model-based approach for designing and deploying graphical user interfaces that support intra-platform plasticity for reconfigurable displays. We instantiate the model for E3Screen, a new device that expands a conventional laptop with two slidable, rotatable, and foldable lateral displays, enabling slidable user interfaces. Based on a UML class diagram as a domain model and a SCRUD list as a task model, we define an abstract user interface as interaction units with a corresponding master-detail design pattern. We then map the abstract user interface to a concrete user interface by applying rules for the reconfiguration, concrete interaction, unit allocation, and widget selection and implement it in JavaScript. In a first experiment, we determine display configurations most preferred by users, which we organize in the form of a state-transition diagram. In a second experiment, we address reconfiguration rules and widget selection rules. A third experiment provides insights into the impact of the lateral displays on a visual search task.

Gesture elicitation studies represent a popular and resourceful method in HCI to inform the design of intuitive gesture commands, reflective of end-users' behavior, for controlling all kinds of interactive devices, applications, and systems. In the last ten years, an impressive body of work has been published on this topic, disseminating useful design knowledge regarding users' preferences for finger, hand, wrist, arm, head, leg, foot, and whole-body gestures. In this paper, we deliver a systematic literature review of this large body of work by summarizing the characteristics and findings ofN=216gesture elicitation studies subsuming 5,458 participants, 3,625 referents, and 148,340 elicited gestures. We highlight the descriptive, comparative, and generative virtues of our examination to provide practitioners with an effective method to (i) understand how new gesture elicitation studies position in the literature; (ii) compare studies from different authors; and (iii) identify opportunities for new research. We make our large corpus of papers accessible online as a Zotero group library at https://www.zotero.org/groups/2132650/gesture_elicitation_studies.

We present a concept, hardware prototype, and wearable system designed to deliver vibrotactile feedback at finger, wrist, and forearm level for everyday situations of inattentional blindness. Objects and phenomena automatically detected from the video stream acquired by a pair of glasses with embedded video camera are matched against a list of concepts of interest and converted into vibrotactile patterns delivered to the user on their finger, wrist, or forearm according to the location of the wearable device. Our demonstrations employ an off-the-shelf smart ring (Ring Zero) and armband (Myo) and a custom bracelet that we designed for this work.

We discuss engineering aspects for shifting from “do you see what I see?” applications that stream the user’s field of view to remote viewers toward “do you control what I see?” features in which remote viewers are given the opportunity and tool to control the primary user’s field of view. To this end, we present two applications for (1) smartglasses with embedded video camera for live video streaming and (2) the HoloLens HMD that presents users with mediated versions of the visual world controlled by remote viewers.

With this position paper, we want to draw the attention of the community toward theoretical work and practical opportunities that have been overlooked so far regarding concepts, principles, and design knowledge for the user experience of Augmented and Mixed Reality content, I/O devices, interactions, applications, and systems. Despite considerable innovations in commercial products and research prototypes enabling Augmented and Mixed Reality worlds, how to design great user experiences in such worlds has been overall neglected at core, while the information and knowledge currently available to practitioners cover usability aspects mostly. Therefore, we advocate for theoretical foundations of the user experience in Augmented and Mixed Reality and propose several directions for more scientific research in this regard.

We address in this work visual menus for smartglasses by proposing a wide range of design options, such as shape, location, orientation, and presentation modalities, which we compile in a design space with eight dimensions. From this design space, we select a subset of fourteen 2-D menus, for which we collect users’ preferences during a vignette experiment with N=251 participants. We report numerical measures of absolute, relative, and aggregate preference for smartglasses menus, and employ a particular Thurstone’s pairwise comparison technique, the Bradley-Terry model, to evaluate menu designs. Our results highlight key variables influencing users’ preferences regarding the visual appearance of smartglasses menus, which we use to discuss opportunities for future work.

Connected vehicles collect and share data by communicating with road infrastructure, with each other, the web, IoT systems, and with their occupants’ personal devices. Part of this data is presented to drivers via a multitude of interactive devices and systems. Thus, one challenge that arises in such a complex environment is effective and safe operation of the various interactive systems, e.g., the in-vehicle infotainment (IVI). In this paper, we present a synopsis of input modalities from the literature of automotive user interfaces (AutoUIs) for media consumption inside connected vehicles.

We examine the concept and characteristics of “Augmented Reality Television” (ARTV) using a four-step investigation method consisting of (1) an analysis of commonly-accepted perspectives on Augmented and Mixed Reality systems, (2) a literature survey of previous work on ARTV, (3) relevant connections with other areas of scientific investigation from TVX/IMX, such as Ambient Media, Interactive TV, and 3-D TV, and (4) by proposing a conceptual framework for ARTV called the “Augmented Reality Television Continuum.” Our work comes at a moment when the excitement and hype about the potential of AR for home entertainment has overlooked rigorous analysis and clear-cut examinations of the concepts involved, which should be the hallmark of any exact science. With this work, our goal is to draw the community’s attention toward fundamentals and first principles of ARTV and to tease out its salient qualities on solid foundations.

We introduce the dissimilarity-consensus method, a new approach to computing objective measures of consensus between users' gesture preferences to support data analysis in end-user gesture elicitation studies. Our method models and quantifies the relationship between users' consensus over gesture articulation and numerical measures of gesture dissimilarity, e.g., Dynamic Time Warping or Hausdorff distances, by employing growth curves and logistic functions. We exemplify our method on 1,312 whole-body gestures elicited from 30 children, ages 3 to 6 years, and we report the first empirical results in the literature on the consensus between whole-body gestures produced by children this young. We provide C# and R software implementations of our method and make our gesture dataset publicly available.

We examine the articulation characteristics of stroke-gestures produced by people with upper body motor impairments on touchscreens as well as the accuracy rates of popular classification techniques, such as the $-family, to recognize those gestures. Our results on a dataset of 9,681 gestures collected from 70 participants reveal that stroke-gestures produced by people with motor impairments are recognized less accurately than the same gesture types produced by people without impairments, yet still accurately enough (93.0%) for practical purposes; are similar in terms of geometrical criteria to the gestures produced by people without impairments; but take considerably more time to produce (3.4s vs. 1.7s) and exhibit lower consistency (-49.7%). We outline a research roadmap for accessible gesture input on touchscreens for users with upper body motor impairments, and we make our large gesture dataset publicly available in the community.

We examine users' preferences for Augmented Reality for television and report findings from an exploratory study with 172 participants we conducted to understand the perceived value of twenty distinct AR-TV scenarios. We connect our findings to participants' overall perceptions of and experience with AR technology as well as to their self-reported television watching behavior. Our results reveal high perceived value for AR-TV scenarios involving interactive content, wall-sized and room-sized video projections, multiple perspectives of the same movie scene, and for virtual objects coming out of the TV screen into the room. Other scenarios, such as live video of remote friends watching the same broadcast or multiple virtual channels displayed around the TV screen were rated less valuable.

We introduce "Life-Tags," a wearable, smartglasses-based system for abstracting life in the form of clouds of tags and concepts automatically extracted from snapshots of the visual reality recorded by wearable video cameras. Life-Tags summarizes users' life experiences using word clouds, highlighting the "executive summary" of what the visual experience felt like for the smartglasses user during some period of time, such as a specific day, week, month, or the last hour. In this paper, we focus on (i) design criteria and principles of operation for Life-Tags, such as its first-person, eye-level perspective for recording life, passive logging mode, and privacy-oriented operation, as well as on (ii) technical and engineering aspects for implementing Life-Tags, such as the block architecture diagram highlighting devices, software modules, third-party services, and dataflows. We also conduct a technical evaluation of Life-Tags and report results from a controlled experiment that generated 21,600 full HD snapshots from six indoor and outdoor scenarios, representative of everyday life activities, such as walking, eating, traveling, etc., with a total of 180 minutes of recorded life to abstract with tag clouds. Our experimental results and Life-Tags prototype inform design and engineering of future life abstracting systems based on smartglasses and wearable video cameras to ensure effective generation of rich clouds of concepts, reflective of the visual experience of the smartglasses user.

Welcome to EICS, the Engineering Interactive Computing Systems community, PACMHCI/EICS journal, and annual conference! In this short article, we introduce newcomers to the field and to our community with an overview of what EICS is and how it positions with respect to other venues in Human-Computer Interaction, such as CHI, UIST, and IUI, highlighting its legacy and paying homage to past scientific events from which EICS emerged. We also take this opportunity to enumerate and exemplify scientific contributions to the field of Engineering Interactive Computing Systems, which we hope to guide researchers and practitioners towards making their future PACMHCI/EICS submissions successful and impactful in the EICS community.

We present SAPIENS, a software architecture designed to support engineering of interactive systems featuring peripheral interaction in the context of smart environments. SAPIENS introduces dedicated components for user and device tracking, attention detection, priority management for devices, tasks, and notifications, context-awareness inference, user interruptibility prediction, and device interchangeability that can be instantiated at will according to the needs of the application. To implement these components effectively, SAPIENS employs event-based processing by reusing the core engine of a recently introduced software architecture, Euphoria (Schipor et al., 2019), that was specifically designed for engineering interactions in smart environments with heterogeneous I/O devices, and relies entirely on web standards, protocols, and open data-interchange formats, such as JavaScript, WebSockets, HTTP, and JSON. This inheritance makes SAPIENS flexible and adaptable to support implementation of diverse application scenarios for peripheral interaction and for a wide variety of smart environments, devices, platforms, data formats, and contexts of use. We present our design criteria for SAPIENS regarding (1) event handling techniques, (2) quality, (3) contextual, and (4) attention-related properties, and describe its components and dataflows that make SAPIENS a specialized software architecture for peripheral interaction scenarios. We also demonstrate SAPIENS with a practical application, inspired and adapted from Bakker's (2013) classical example for peripheral interaction, for which we provide an online simulation tool that researchers and practitioners can readily use to consult actual JavaScript code implementing the inner logic of selected components of our architecture as well as to observe live JSON messages exchanged by the various components of SAPIENS.

We introduce AB4Web, a web-based engine that implements a balanced randomized version of the multivariate A/B testing, specifically designed for practitioners to readily compare end-users' preferences for user interface alternatives, such as menu layouts, widgets, controls, forms, or visual input commands. AB4Web automatically generates a balanced set of randomized pairs from a pool of user interface design alternatives, presents them to participants, collects their preferences, and reports results from the perspective of four quantitative measures: the number of presentations, the preference percentage, the latent score of preference, and the matrix of preferences. In this paper, we exemplify the AB4Web tester with a user study for which N=108 participants expressed their preferences regarding the visual design of 49 distinct graphical adaptive menus, with a total number of 5,400 preference votes. We compare the results obtained from our quantitative measures with four alternative methods: Condorcet, de Borda count starting at one and zero, and the Dowdall scoring system. We plan to release AB4Web as a public tool for practitioners to create their own A/B testing experiments.

We present empirical results about users' gesture preferences for smart rings by analyzing 672 gestures from 24 participants. We report an overall low consensus (mean .112, maximum .225 on the unit scale) between participants' gesture proposals, and we point to the challenges of designing highly-generalizable ring gestures across users. We also contribute to the practice of gesture elicitation studies by discussing how a priori conditions (e.g., participants' traits, such as creativity and motor skills), commitment and behavior during the experiment (e.g., their thinking times), but also a posteriori aspects (the experimenter's choice of criteria to group gestures into categories) affect agreement. We offer design guidelines for ring gestures informed by our empirical observations, and present a collection of gestures reflective of our participants' mental models for effecting commands using smart rings.

We introduce KeyTime, a new technique and accompanying software for predicting the production times of users' stroke gestures articulated on touchscreens. KeyTime employs the principles and concepts of the Kinematic Theory, such as lognormal modeling of stroke gestures' velocity profiles, to estimate gesture production times significantly more accurately than existing approaches. Our experimental results obtained on several public datasets show that KeyTime predicts user-independent production times that correlate r=.99 with groundtruth from just one example of a gesture articulation, while delivering an average error in the predicted time magnitude that is 3 to 6 times smaller than that delivered by CLC, the best prediction technique up to date. Moreover, KeyTime reports a wide range of useful statistics, such as the trimmed mean, median, standard deviation, and confidence intervals, providing practitioners with unprecedented levels of accuracy and sophistication to characterize their users' a priori time performance with stroke gesture input.

We introduce GATO, a human performance analysis technique grounded in the Kinematic Theory that delivers accurate predictions for the expected user production time of stroke gestures of all kinds: unistrokes, multistrokes, multitouch, or combinations thereof. Our experimental results obtained on several public datasets (82 distinct gesture types, 123 participants, ≈36k gesture samples) show that GATO predicts user-independent gesture production times that correlate rs > .9 with groundtruth, while delivering an average relative error of less than 10% with respect to actual measured times. With its accurate estimations of users' a priori time performance with stroke gesture input, GATO will help researchers to understand better users' gesture articulation patterns on touchscreen devices of all kinds. GATO will also benefit practitioners to inform highly effective gesture set designs.

We introduce $Q, a super-quick, articulation-invariant point-cloud stroke-gesture recognizer for mobile, wearable, and embedded devices with low computing resources. $Q ran up to 142X faster than its predecessor $P in our benchmark evaluations on several mobile CPUs, and executed in less than 3% of $P's computations without any accuracy loss. In our most extreme evaluation demanding over 99% user-independent recognition accuracy, $P required 9.4s to run a single classification, while $Q completed in just 191ms (a 49X speed-up) on a Cortex-A7, one of the most widespread CPUs on the mobile market. $Q was even faster on a low-end 600-MHz processor, on which it executed in only 0.7% of $P's computations (a 142X speed-up), reducing classification time from two minutes to less than one second. $Q is the next major step for the "$-family" of gesture recognizers: articulation-invariant, extremely fast, accurate, and implementable on top of $P with just 30 extra lines of code.

We introduce a new principled method grounded in the Kinematic Theory of Rapid Human Movements to automatically generate synthetic stroke gestures across user populations in order to support ability-based design of gesture user interfaces. Our method is especially useful when the target user population is difficult to sample adequately and, consequently, when there is not enough data to train gesture recognizers to deliver high levels of accuracy. To showcase the relevance and usefulness of our method, we collected gestures from people without visual impairments and successfully synthesized gestures with the articulation characteristics of people with visual impairments. We also show that gesture recognition accuracy improves significantly when using our synthetic gesture samples for training. Our contributions will benefit researchers and practitioners that wish to design gesture user interfaces for people with various abilities by helping them prototype, evaluate, and predict gesture recognition performance without having to expressly recruit and involve people with disabilities in long, time-consuming gesture collection experiments.

We contribute in this work on gesture recognition to improve the accessibility of touch screens for people with low vision. We examine the accuracy of popular recognizers for gestures produced by people with and without visual impairments, and we show that the user-independent accuracy of $P, the best recognizer among those evaluated, is small for people with low vision (83.8%), despite $P being very effective for gestures produced by people without visual impairments (95.9%). By carefully analyzing the gesture articulations produced by people with low vision, we inform key algorithmic revisions for the P recognizer, which we call P+. We show significant accuracy improvements of $P+ for gestures produced by people with low vision, from 83.8% to 94.7% on average and up to 98.2%, and 3x faster execution times compared to P.

This article identifies, catalogues, and discusses factors that are responsible for causing visual impairment of either a pathological or situational nature for touch and gesture input on smart mobile devices. Because the vast majority of interactions with touchscreen devices are highly visual in nature, any factor that prevents a clear, direct view of the mobile device’s screen can have potential negative implications on the effectiveness and efficiency of the interaction. This work presents the first overview of such factors, which are grouped in a catalogue of users, devices, and environments. The elements of the catalogue (e.g., psychological factors that relate to the user, or the social acceptability of mobile device use in public that relates to the social environment) are discussed in the context of current eye pathology classification from medicine and the recent literature in human–computer interaction on mobile touch and gesture input for people with visual impairments, for which a state-of-the-art survey is conducted. The goal of this work is to help systematize research on visual impairments and mobile touchscreen interaction by providing a catalogue-based view of the main causes of visual impairments affecting touch and gesture input on smart mobile devices.

Understanding users’ whole-body gesture performance quantitatively requires numerical gesture descriptors or features. However, the vast majority of gesture features that have been proposed in the literature were specifically designed for machines to recognize gestures accurately, which makes those features exclusively machine-readable. The complexity of such features makes it difficult for user interface designers, non-experts in machine learning, to understand and use them effectively (see, for instance, the Hu moment statistics or the Histogram of Gradients features), which reduces considerably designers’ available options to describe users’ whole-body gesture performance with legible and easily interpretable numerical measures. To address this problem, we introduce in this work a set of 17 measures that user interface practitioners can readily employ to characterize users’ whole-body gesture performance with human-readable concepts, such as area, volume, or quantity. Our measures describe (1) spatial characteristics of body movement, (2) kinematic performance, and (3) body posture appearance for whole-body gestures. We evaluate our measures on a public dataset composed of 5,654 gestures collected from 30 participants, for which we report several gesture findings, e.g., participants performed body gestures in an average volume of space of 1.0 m3, with an average amount of hands movement of 14.6 m, and a maximum body posture diffusion of 5.8 m. We show the relationship between our gesture measures and recognition rates delivered by a template-based Nearest-Neighbor whole-body gesture classifier implementing the Dynamic Time Warping dissimilarity function. We also release BOGArT, the Body Gesture Analysis Toolkit, that automatically computes our measures. This work will empower researchers and practitioners with new numerical tools to reach a better understanding of how users perform whole-body gestures and thus, to use this knowledge to inform improved designs of whole-body gesture user interfaces.

We present two contributions toward improving the accessibility of touch screens for people with motor impairments. First, we provide an exploration of the touch behaviors of 10 people with motor impairments, e.g., we describe how touching with the back or sides of the hand, with multiple fingers, or with knuckles creates varied multi-point touches. Second, we introduce Smart Touch, a novel template-matching technique for touch input that maps any number of arbitrary contact-areas to a user's intended (x,y) target location. The result is that users with motor impairments can touch however their abilities allow, and Smart Touch will resolve their intended touch point. Smart Touch therefore allows users to touch targets in whichever ways are most comfortable and natural for them. In an experimental evaluation, we found that Smart Touch predicted (x,y) coordinates of the users' intended target locations over three times closer to the intended target than the native Land-on and Lift-off techniques reported by the built-in touch sensors found in the Microsoft PixelSense interactive tabletop. This result is an important step toward improving touch accuracy for people with motor impairments and others for whom touch screen operation was previously impossible.

Elicitation studies, where users supply proposals meant to effect system commands, have become a popular method for system designers. But the method to date has assumed a within-subjects procedure and statistics. Despite the benefits of examining the relative agreement of independent groups (e.g., men versus women, children versus adults, novices versus experts, etc.), the lack of appropriate tools for between-subjects agreement rate analysis have prevented so far such comparative investigations. In this work, we expand the elicitation method to between-subjects designs. We introduce a new measure for evaluating coagreement between groups and a new statistical test for agreement rate analysis that reports the exact p-value to evaluate the significance of the difference between agreement rates calculated for independent groups. We show the usefulness of our tools by re-examining previously published gesture elicitation data, for which we discuss significant differences in agreement for technical and non-technical participants, men and women, and different acquisition technologies. Our new tools will enable practitioners to properly analyze their user-elicited data resulted from complex experimental designs with multiple independent groups and, consequently, will help them understand agreement data and verify hypotheses about agreement at more sophisticated levels of analysis.

We investigate in this work users' perceptions of interacting with invisible, zero-weight digital matter for smart mobile scenarios. To this end, we introduce the concept of a digital vibron as vibrational manifestation of a digital object located outside its container device. We exemplify gesture-based interactions for digital vibrons and show how thinking about interactions in terms of digital vibrons can lead to new interactive experiences in the physical-digital space. We present the results of a user study that showed high scores of users' perceived experience, usability, and desirability, and we discuss users' preferences for vibration patterns to inform the design of vibrotactile feedback for digital vibrons. We hope that this work will inspire researchers and practitioners to further explore and develop digital vibrons to design localized vibrotactile feedback for digital objects outside their smart devices toward new interactive experiences in the physical-digital space.

We address in this work the process of agreement rate analysis for characterizing the level of consensus between participants' proposals elicited during guessability studies. Two new measures, i.e., disagreement rate for referents and coagreement rate between referents, are proposed to accompany the widely-used agreement rate formula of Wobbrock et al. [37] when reporting participants' consensus for symbolic input. A statistical significance test for comparing the agreement rates of k>=2 referents is presented in analogy with Cochran's success/failure Q test [5], for which we express the test statistic in terms of agreement and coagreement rates. We deliver a toolkit to assist practitioners to compute agreement, disagreement, and coagreement rates, and run statistical tests for agreement rates at p=.05, .01, and .001 levels of significance. We validate our theoretical development of agreement rate analysis in relation with several previously published elicitation studies. For example, when we present the probability distribution function of the agreement rate measure, we also use it (1) to explain the magnitude of agreement rates previously reported in the literature, and (2) to propose qualitative interpretations for agreement rates, in analogy with Cohen's guidelines for effect sizes [6]. We also re-examine previously published elicitation data from the perspective of the agreement rate test statistic, and highlight new findings on the effect of referents over agreement rates, unattainable prior to this work. We hope that our contributions will advance the current knowledge in agreement rate analysis, providing researchers and practitioners with new techniques and tools to help them understand user-elicited data at deeper levels of detail and sophistication.

We present in this work GRASPhere, a prototype device that enables users to select and manipulate on-screen objects with grasping gestures. We demonstrate GRASPhere as an extension of a widely-employed force-sensing device, the Phantom OMNI. The hardware design of GRASPhere employs a minimum of components that cost less than $20, which makes our prototype easily replicable for practitioners interested in incorporating grasping gestures into their own force-feedback interactive applications. We discuss application opportunities for GRASPhere, such as exploring multimedia data with physical metaphors and providing assistance to people with visual impairments during indirect touch interaction.

We present a simple, two-step technique for recognizing multi-touch gesture input that is invariant to how users articulate gestures, i.e., by using one or two hands, one or multiple fingers, one or multiple strokes, synchronous or asynchronous stroke input. We introduce, for the first time in the gesture literature, a preprocessing step that is specific to multi-touch gestures (Match-Up) that clusters together similar strokes produced by different fingers, before running a gesture recognizer (Conquer). We report gains in recognition accuracy up to 10% leveraged by our new preprocessing step, which manages to construct a more adequate representation of multi-touch gestures in terms of key strokes. It is our hope that the Match-Up technique will add to the practitioners' toolkit of gesture preprocessing techniques, as a first step toward filling today's lack of algorithmic knowledge to process multi-touch input and leading toward the design of more efficient and accurate recognizers for touch surfaces.

We show that users are consistent in their assessments of the articulation difficulty of multi-touch gestures, even under the many degrees of freedom afforded by multi-touch input, such as (1) various number of fingers touching the surface, (2) various number of strokes that structure the gesture shape, and (3) single-handed and bimanual input. To understand more about perceived difficulty, we characterize gesture articulations captured under these conditions with geometric and kinematic descriptors computed on a dataset of 7,200 samples of 30 distinct gesture types collected from 18 participants. We correlate the values of the objective descriptors with users' subjective assessments of articulation difficulty and report path length, production time, and gesture size as the highest correlators (max Pearson's r=.95). We also report new findings about multi-touch gesture input, e.g., gestures produced with more fingers are larger in size and take more time to produce than single-touch gestures; bimanual articulations are not only faster than single-handed input, but they are also longer in path length, present more strokes, and result in gesture shapes that are deformed horizontally by 35% in average. We use our findings to outline a number of 14 guidelines to assist multi-touch gesture set design, recognizer development, and inform gesture-to-function mappings through the prism of the user-perceived difficulty of gesture articulation.

We introduce gesture heatmaps, a novel gesture analysis technique that employs color maps to visualize the variation of local features along the gesture path. Beyond current gesture analysis practices that characterize gesture articulations with single-value descriptors, e.g., size, path length, or speed, gesture heatmaps are able to show with colorful visualizations how the value of any such descriptors vary along the gesture path. We evaluate gesture heatmaps on three public datasets comprising 15,840 gesture samples of 70 gesture types from 45 participants, on which we demonstrate heatmaps' capabilities to (1) explain causes for recognition errors, (2) characterize users' gesture articulation patterns under various conditions, e.g., finger versus pen gestures, and (3) help understand users' subjective perceptions of gesture commands, such as why some gestures are perceived easier to execute than others. We also introduce chromatic confusion matrices that employ gesture heatmaps to extend the expressiveness of standard confusion matrices to better understand gesture classification performance. We believe that gesture heatmaps will prove useful to researchers and practitioners doing gesture analysis, and consequently, they will inform the design of better gesture sets and development of more accurate recognizers.

We show that large consensus exists among users in the way they articulate stroke gestures at various scales (i.e., small, medium, and large), and formulate a simple rule that estimates the user-intended scale of input gestures with 87% accuracy. Our estimator can enhance current gestural interfaces by leveraging scale as a natural parameter for gesture input, reflective of user perception (i.e., no training required). Gesture scale can simplify gesture set design, improve gesture-to-function mappings, and reduce the need for users to learn and for recognizers to discriminate unnecessary symbols.

Current measures of stroke gesture articulation lack descriptive power because they only capture absolute characteristics about the gesture as a whole, not fine-grained features that reveal subtleties about the gesture articulation path. We present a set of twelve new relative accuracy measures for stroke gesture articulation that characterize the geometric, kinematic, and articulation accuracy of single and multi-stroke gestures. To compute the accuracy measures, we introduce the concept of a gesture task axis. We evaluate our measures on five public datasets comprising 38,245 samples from 107 participants, about which we make new discoveries; e.g., gestures articulated at fast speed are shorter in path length than slow or medium-speed gestures, but their path lengths vary the most, a finding that helps understand recognition performance. This work will enable a better understanding of users' stroke gesture articulation behavior, ultimately leading to better gesture set designs and more accurate recognizers.

We investigate in this work the effect of bit depth on the performance of today's commonly used nearest-neighbor gesture recognizers. As current bit representations are typically an artifact of today's hardware and file formats, they are not reflective of the true cardinality of gesture data. We show that as few as 4-5 bits per gesture channel (x/y) are enough in order to attain peak recognition for Euclidean, Cosine, DTW, and Hausdorff distances. We also show how reduction in bit depth can lead to 85 times less memory for storing the training set without ruining recognition performance. The results will benefit practitioners of the next age of gesture sensing gadgets and devices that need to optimize speed, memory, and bit depth representation in their software and hardware designs.

Rapid prototyping of gesture interaction for emerging touch platforms requires that developers have access to fast, simple, and accurate gesture recognition approaches. The $-family of recognizers ($1, $N) addresses this need, but the current most advanced of these, $N-Protractor, has significant memory and execution costs due to its combinatoric gesture representation approach. We present $P, a new member of the $-family, that remedies this limitation by considering gestures as clouds of points. $P performs similarly to $1 on unistrokes and is superior to $N on multistrokes. Specifically, $P delivers >99% accuracy in user-dependent testing with 5+ training samples per gesture type and stays above 99% for user-independent tests when using data from 10 participants. We provide a pseudocode listing of $P to assist developers in porting it to their specific platform and a "cheat sheet" to aid developers in selecting the best member of the $-family for their specific application needs.

One Feature (1F) is a simple and intuitive pruning strategy that reduces considerably the amount of computations required by Nearest-Neighbor gesture classifiers while still preserving the high recognition rate. Performance results are reported for 1F by analyzing a large set of candidate features showing recognition rates of 99% with a peak reduction in computations of 70%. 1F is easy to implement, flexible with respect to the choice of the feature, and exploits the intuition of the designer by exposing clear inner workings.

We investigate in this work the effect of motion sampling rate over recognition accuracy and execution time for current template-based gesture recognizers in order to provide performance guidelines to practitioners and designers of gesture-based interfaces. We show that as few as 6 sampling points are sufficient for Euclidean and angular recognizers to attain high recognition rates and that a linear relationship exists between sampling rate and number of gestures for the dynamic time warping technique. We report execution times obtained with our controlled downsampling which are 10-20 times faster than shown by existing work at the same high recognition rates. The results of this work will benefit practitioners by providing important performance aspects to consider when using template-based gesture recognizers.