Current Virtual Reality (VR) Head-Mounted Displays (HMDs) offer limited shortcuts for rapid command access, which often requires users to navigate menus through precise visual targeting at multiple depths. This process can be slow and distracting, particularly during immersive gaming or productivity tasks. While marking menus have shown effectiveness as a shortcut command access mechanism, their performance in VR has not been adequately studied. Moreover, their potential integration with 6-degree-of-freedom (6-DoF) controllers and 2-DoF joysticks in VR environments remains largely unexplored. In this paper, we introduce the Drum Menu, a bimanual shortcut command access technique derived from the idea of traditional pie menus, featuring three input methods, designed for 4-item and 8-item layouts specifically for VR controller command access. Users can select commands by rotating the joystick, drawing a stroke, or pointing in different directions. Bimanual input enables simultaneous access to two menu levels. A controlled user study reveals that drum menus are faster than the unimanual versions for the 4-item layout. Additionally, users prefer the bimanual joystick drum menu with the 4-item layout given its short task time, low error rate, and low physical movement. For the 8-item layout, stroke drum menus are found to be less error-prone for expert users compared to the other techniques.

Teleportation - changing the point of view in 3D space by specifying a position - is one of the most common locomotion solutions in VR. However, it currently lacks a mechanism to adjust the height in 3D space, and it is difficult for users to predict the exact final view after the teleportation. Users are relocated to a place without knowing what the final view will look like. As a result, they often need to perform remedial interactions to achieve their ideal position, which can be time-consuming and effort-intensive. In this paper, we present Fly the Moon to Me (Locomoontion), a novel technique that enables users to bring their destination to themselves through object manipulation. Users first create a copy of the object they want to approach as a preview by selecting it, then bring it to an ideal position and direction using existing object manipulation techniques, and then snap the original object to the preview together with the rest of the world. A controlled experiment with 18 participants via a teleportation task reveals that Locomoontion is more effective than the traditional Point&Teleport technique with grabbing the world as a remedy to adjust the final positioning.

We compare the performance and level of distraction of expressive directional gesture input in the context of in-vehicle system commands. Center console touchscreen swipes and midair swipe-like movements are tested in 8-directions, with 8-button touchscreen tapping as a baseline. Participants use these input methods for intermittent target selections while performing the Lane Change Task in a virtual driving simulator. Input performance is measured with time and accuracy, cognitive load with deviation of lane position and speed, and distraction from frequency of off-screen glances. Results show midair gestures were less distracting and faster, but with lower accuracy. Touchscreen swipes and touchscreen tapping are comparable across measures. Our work provides empirical evidence for vehicle interface designers and manufacturers considering midair or touch directional gestures for centre console input.

Extracting underlying data from rasterized charts is tedious and inaccurate; values might be partially occluded or hard to distinguish, and the quality of the image limits the precision of the data being recovered. To address these issues, we introduce a semi-automatic system leveraging vector charts to extract the underlying data easily and accurately. The system is designed to make the most of vector information by relying on a drag-and-drop interface combined with selection, filtering, and previsualization features. A user study showed that participants spent less than 4 minutes to accurately recover data from charts published at CHI with diverse styles, thousands of data points, a combination of different encodings, and elements partially or completely occluded. Compared to other approaches relying on raster images, our tool successfully recovered all data, even when hidden, with a 78% lower relative error.

The growing live streaming economy and virtual reality (VR) technologies have sparked interest in VR streaming among streamers and viewers. However, limited research has been conducted to understand this emerging streaming practice. To address this gap, we conducted an in-depth thematic analysis of 34 streaming videos from 12 VR streamers with varying levels of experience, to explore the current practices, interaction styles, and strategies, as well as to investigate the challenges and opportunities for VR streaming. Our findings indicate that VR streamers face challenges in building emotional connections and maintaining streaming flow due to technical problems, lack of fluid transitions between physical and virtual environments, and not intentionally designed game scenes. As a response, we propose six design implications to encourage collaboration between game designers and streaming app developers, facilitating fluid, rich, and broad interactions for an enhanced streaming experience. In addition, we discuss the use of streaming videos as user-generated data for research, highlighting the lessons learned and emphasizing the need for tools to support streaming video analysis. Our research sheds light on the unique aspects of VR streaming, which combines interactions across displays and space.

Due to the proliferation of smart wearables, it is now the case that designers can explore novel ways that devices can be used in combination by end-users. In this paper, we explore the gestural input enabled by the combination of smart earbuds coupled with a proximal smartwatch. We identify a consensus set of gestures and a taxonomy of the types of gestures participants create through an elicitation study. In a follow-on study conducted on Amazon's Mechanical Turk, we explore the social acceptability of gestures enabled by watch+earbud gesture capture. While elicited gestures continue to be simple, discrete, in-context actions, we find that elicited input is frequently abstract, varies in size and duration, and is split almost equally between on-body, proximal, and more distant actions. Together, our results provide guidelines for on-body, near-ear, and in-air input using earbuds and a smartwatch to support gesture capture.

Pointing is an elementary interaction in virtual and augmented reality environments, and, to effectively support selection, techniques must deal with the challenges of occlusion and depth specification. Most of the previous techniques require two explicit steps to handle occlusion. In this paper, we propose Conductor, an intuitive, plane-ray, intersection-based, 3D pointing technique where users leverage bimanual input to control a ray and intersecting plane. Conductor allows users to use the non-dominant hand to adjust the cursor distance on the ray while pointing with the dominant hand. We evaluate Conductor against Raycursor, a state-of-the-art VR pointing technique, and show that Conductor outperforms Raycursor for selection tasks. Given our results, we argue that bimanual selection techniques merit additional exploration to support object selection and placement within virtual environments.

While many smartphones now include a waterfall (or curved edge) display, this edge is rarely exploited for input. In this paper, we introduce EdgeMark, a thumb-operated, one-handed, gestural menu to provide rapid and subtle access to commands. In a first study, we assess the movement range of the user during uni-manual input to define a spatial range for our EdgeMark menu. In a second study, we compare three potential EdgeMark designs to a marking menu variant to assess accuracy and performance. Our findings indicate that EdgeMark menus can serve as a reliable, effective mechanism for subtle, one-hand command invocation in modern, waterfall-edged smartphones.

Alongside vision and sound, hardware systems can be readily designed to support various forms of tactile feedback; however, while a significant body of work has explored enriching visual and auditory communication with interactive systems, tactile information has not received the same level of attention. In this work, we explore increasing the expressivity of tactile feedback by allowing the user to dynamically select between several channels of tactile feedback using variations in finger speed. In a controlled experiment, we show that a user can learn the dynamics of eyes-free tactile channel selection among different channels, and can reliable discriminate between different tactile patterns during multi-channel selection with an accuracy up to 90% when using two finger speed levels. We discuss the implications of this work for richer, more interactive tactile interfaces.

One common task when controlling smart displays is the manipulation of menu items. Given current examples of smart displays that support distant bare hand control, in this paper we explore menu item selection tasks with three different mappings of barehand movement to target selection. Through a series of experiments, we demonstrate that Positional mapping is faster than other mappings when the target is visible but requires many clutches in large targeting spaces. Rate-based mapping is, in contrast, preferred by participants due to its perceived lower effort, despite being slightly harder to learn initially. Tradeoffs in the design of target selection in smart tv displays are discussed.

Personal smart devices have demonstrated a variety of efficient techniques for pointing and selecting on physical displays. However, when migrating these input techniques to augmented reality, it is both unclear what the relative performance of different techniques will be given the immersive nature of the environment, and it is unclear how viewport-based versus world-based pointing methods will impact performance. To better understand the impact of device and viewing perspectives on pointing in augmented reality, we present the results of two controlled experiments comparing pointing conditions that leverage various smartphone- and smartwatch-based external display pointing techniques and examine viewport-based versus world-based target acquisition paradigms. Our results demonstrate that viewport-based techniques offer faster selection and that both smartwatch- and smartphone-based pointing techniques represent high-performance options for performing distant target acquisition tasks in augmented reality.

Delayed display of menu items is a core design component of marking menus, arguably to prevent visual distraction and foster the use of mark mode. We investigate these assumptions, by contrasting the original marking menu design with immediately-displayed marking menus. In three controlled experiments, we fail to reveal obvious and systematic performance or usability advantages to using delay and mark mode. Only in very constrained settings – after significant training and only two items to learn – did traditional marking menus show a time improvement of about 260~ms. Otherwise, we found an overall decrease in performance with delay, whether participants exhibited practiced or unpracticed behaviour. Our final study failed to demonstrate that an immediately-displayed menu interface is more visually disrupting than a delayed menu. These findings inform the costs and benefits of incorporating delay in marking menus, and motivate guidelines for situations in which its use is desirable.

This paper presents a qualitative multi-phase study seeking to identify patterns in users' anthropomorphized perceptions of conversational agents. Through a comparative analysis of behavioral perceptions and visual conceptions of three agents - Alexa, Google Assistant, and Siri - we first show that the perceptions of an agent's character are structured according to five categories: approachability, sentiment toward a user, professionalism, intelligence, and individuality. We then explore visualizations of the agents' appearance and discuss the specifics assigned to each agent. Finally, we analyze associative explanations for these perceptions. We demonstrate that the anthropomorphized behavioral and visual perceptions of agents yield structural consistency and discuss how these perceptions are linked with each other and system features.

Documents such as presentations, instruction manuals, and research papers are disseminated using various file formats, many of which barely support the incorporation of interactive content. To address this lack of interactivity, we present Chameleon, a system-wide tool that combines computer vision algorithms used for image identification with an open database format to allow for the layering of dynamic content. Using Chameleon, static documents can be easily upgraded by layering user-generated interactive content on top of static images, all while preserving the original static document format and without modifying existing applications. We describe the development of Chameleon, including the design and evaluation of vision-based image replacement algorithms, the new document-creation pipeline as well as a user study evaluating Chameleon.

As multi-touch displays grow in size and shrink in price, they are more commonly used as gaming devices. When co-located users play games on a single, large display, establishing and maintaining their physical and digital territories poses a social challenge to their interaction. To gain insight into the mechanisms of establishing and maintaining users' physical and digital territories, we analyze territorial interactions in cooperative and competitive multiplayer gameplay. Participants reported weighing each game interaction based on perceived intent to determine how socially acceptable they deemed each behaviour. In light of our observations, we contribute and discuss implications for the design of multi-user, large display, co-located, touchscreen games that consider display properties, digital and physical space, permeability of boundaries, and asymmetry of play to create interactions between players.

In head-mounted display (HMD) interaction, text entry is frequently supported via some form of virtual touch, controller, or ray casting keyboard. While these options effectively support text entry, they often incur costs of additional external hardware, awkward movements, and hand encumbrance. We propose STAT, a low-cost, mobile, touch typing technique that leverages a smartphone screen located at the thigh, to support both tap and word gesture text input for HMDs. Through a controlled laboratory study, we explore the efficacy of our technique – including a comparison of typing in and out of an enclosed pocket – and present design recommendations for the opportunistic use of a personal touchscreen device positioned at a user’s thigh for HMD text entry.

Intensive exploration and navigation of hierarchical lists on smartphones can be tedious and time-consuming as it often requires users to frequently switch between multiple views. To overcome this limitation, we present PinchList, a novel interaction design that leverages pinch gestures to support seamless exploration of multi-level list items in hierarchical views. With PinchList, sub-lists are accessed with a pinch-out gesture whereas a pinch-in gesture navigates back to the previous level. Additionally, pinch and flick gestures are used to navigate lists consisting of more than two levels. We conduct a user study to refine the design parameters of PinchList such as a suitable item size, and quantitatively evaluate the target acquisition performance using pinch-in/out gestures in both scrolling and non-scrolling conditions. In a second study, we compare the performance of PinchList in a hierarchal navigation task with two commonly used touch interfaces for list browsing: pagination and expand-and-collapse interfaces. The results reveal that PinchList is significantly faster than other two interfaces in accessing items located in hierarchical list views. Finally, we demonstrate that PinchList enables a host of novel applications in list-based interaction?

Many touch based interactions provide limited opportunities for direct tactile feedback; examples include multi-user touch displays, augmented reality based projections on passive surfaces, and mid-air input. In this paper, we consider distal feedback, through vibrotactile stimulation on a smart-watch placed on the user's non-dominant wrist, as an alternative feedback mechanism to interaction location vibrotactile feedback, under the user's finger. We compare the effectiveness of interaction location feedback vs. distal feedback through a Fitts's Law task completed on a smartphone. Results show that distal and interaction location feedback both reduce errors in target acquisition and exhibit statistically comparable performance, suggesting that distal vibrotactile feedback is a suitable alternative when interaction location feedback is not readily available.

In this paper, we explore an alternative form of volunteer work, PledgeWork, where individuals, rather than working directly for a charity, make indirect donations by completing tasks provided by a third party task provider. PledgeWork poses novel research questions on issues of user acceptance of on-line volunteerism, on quality and quantity of work performed as a volunteer, and on the benefits low-barrier volunteerism might provide to charities. To evaluate these questions, we conduct a mixed methods study that compares the quality and quantity of work between volunteer workers and paid workers and user attitudes toward PledgeWork, including perceived benefits and drawbacks. We find that PledgeWork can improve the quality of simple tasks and that the vast majority of our participants expressed interest in using our PledgeWork platform to contribute to a charity. Our interview also reveals current problems with volunteering and online donations, thus highlighting additional strengths of PledgeWork.

While mid-air gestures are an attractive modality with an extensive research history, one challenge with their usage is that the gestures are not self-revealing. Scaffolding techniques to teach these gestures are difficult to implement since the input device, e.g. a hand, wand or arm, cannot present the gestures to the user. In contrast, for touch gestures, feedforward mechanisms (such as Marking Menus or OctoPocus) have been shown to effectively support user awareness and learning. In this paper, we explore whether touch gesture input can be leveraged to teach users to perform mid-air gestures. We show that marking menu touch gestures transfer directly to knowledge of mid-air gestures, allowing performance of these gestures without intervention. We argue that cross-modal learning can be an effective mechanism for introducing users to mid-air gestural input.

Crowdworkers regularly support their work with scripts, extensions, and software to enhance their productivity. Despite their evident significance, little is understood regarding how these tools affect crowdworkers' quality of life and work. In this study, we report findings from an interview study (N=21) aimed at exploring the tooling practices used by full-time crowdworkers on Amazon Mechanical Turk. Our interview data suggests that the tooling utilized by crowdworkers (1) strongly contributes to the fragmentation of microwork by enabling task switching and multitasking behavior; (2) promotes the fragmentation of crowdworkers' work-life boundaries by relying on tooling that encourages a 'work-anywhere' attitude; and (3) aids the fragmentation of social ties within worker communities through limited tooling access. Our findings have implications for building systems that unify crowdworkers' work practice in support of their productivity and well-being.

In gesture recognition, one challenge that researchers and developers face is the need for recognition strategies that mediate between false positives and false negatives. In this article, we examine bi-level thresholding, a recognition strategy that uses two thresholds: a tighter threshold limits false positives and recognition errors, and a looser threshold prevents repeated errors (false negatives) by analyzing movements in sequence. We first describe early observations that led to the development of the bi-level thresholding algorithm. Next, using a Wizard-of-Oz recognizer, we hold recognition rates constant and adjust for fixed versus bi-level thresholding; we show that systems using bi-level thresholding result in significantly lower workload scores on the NASA-TLX and significantly lower accelerometer variance when performing gesture input. Finally, we examine the effect that bi-level thresholding has on a real-world dataset of wrist and finger gestures, showing an ability to significantly improve measures of precision and recall. Overall, these results argue for the viability of bi-level thresholding as an effective technique for balancing between false positives, recognition errors, and false negatives.

Despite the ubiquity of touch-based input and the availability of increasingly computationally powerful touchscreen devices, there has been comparatively little work on enhancing basic canonical gestures such as swipe-to-pan and pinch-to-zoom. In this paper, we introduce transient pan and zoom, i.e. pan and zoom manipulation gestures that temporarily alter the view and can be rapidly undone. Leveraging typical touchscreen support for additional contact points, we design our transient gestures such that they co-exist with traditional pan and zoom interaction. We show that our transient pan-and-zoom reduces repetition in multi-level navigation and facilitates rapid movement between document states. We conclude with a discussion of user feedback, and directions for future research.

Large displays are becoming commonplace at work, at home, or in public areas. However, interaction at a distance -- anything greater than arms-length -- remains cumbersome, restricts simultaneous use, and requires specific hardware augmentations of the display: touch layers, cameras, or dedicated input devices. Yet a rapidly increasing number of people carry smartphones and smartwatches, devices with rich input capabilities that can easily be used as input devices to control interactive systems. We contribute (1) the results of a survey on possession and use of smart devices, and (2) the results of a controlled experiment comparing seven distal pointing techniques on phone or watch, one- and two-handed, and using different input channels and mappings. Our results favor using a smartphone as a trackpad, but also explore performance tradeoffs that can inform the choice and design of distal pointing techniques for different contexts of use.

In mobile touchscreebn interaction, an important challenge is to find solutions to balance the size of individual widgets against the number of widgets needed during interaction. In this work, to address display space limitations, we explore the design of invisible off-screen toolbars (ether-toolbars) that leverage computer vision to expand application features by placing widgets adjacent to the display screen. We show how simple computer vision algorithms can be combined with a natural human ability to estimate physical placing to support highly accurate targeting. Our ether-toolbar design promises targeting accuracy approximating on-screen widget accuracy while significantly expanding the interaction space of mobile devices. Through two experiments, we examine off-screen content placement metaphors and off-screen precision of participants accessing these toolbars. From the data of the second experiment, we provide a basic model that reflects how users perceive mobile surroundings for ether-widgets and validate it. We also demonstrate a prototype system consisting of an inexpensive 3D printed mount for a mirror that supports ether-toolbar implementations. Finally, we discuss the implications of our work and potential design extensions that can increase the usability and the utility of ether-toolbars.

Many smartwatches have a digital crown as a complementary input modality to the touchscreen. While the crown eliminates issues of touchscreen occlusion and imprecision, i.e., the 'fat-finger' problem, the crown input is limited as it only supports bi-directional rotations along a single dimension. We present TiltCrown, a crown prototype that increases a smartwatch's input bandwidth through an isometric joystick. TiltCrown supports angular tilt, rotations and button press events for touchless smartwatch interaction. In a user study we explore how accurate and how fast users can perform item selection tasks using TiltCrown.

The popularity of head-worn displays (HWD) technologies such as Virtual Reality (VR) and Augmented Reality (AR) headsets is growing rapidly. To predict their commercial success, it is essential to understand the acceptability of these new technologies, along with new methods to interact with them. In this vein, the evaluation of social acceptability of interactions with these technologies has received significant attention, particularly from the performer's (i.e., user's) viewpoint. However, little work has considered social acceptability concerns from observers' (i.e., spectators') perspective. Although HWDs are designed to be personal devices, interacting with their interfaces are often quite noticeable, making them an ideal platform to contrast performer and observer perspectives on social acceptability. Through two studies, this paper contrasts performers' and observers' perspectives of social acceptability interactions with HWDs under different social contexts. Results indicate similarities as well as differences, in acceptability, and advocate for the importance of including both perspectives when exploring social acceptability of emerging technologies. We provide guidelines for understanding social acceptability specifically from the observers' perspective, thus complementing our current practices used for understanding the acceptability of interacting with these devices.

This paper explains the importance (1) of full user-space identification with categorization in requirements engineering (RE) and of ensuring that the categorization is a partition of the user space, (2) of the creation and application of user-space-covering personas in RE, (3) of the use of grounded analysis to do RE to produce a specification as a grounded theory, and (4) of privacy and security features in computer-based systems. Then it gives the steps of a grounded analysis method for doing user-space identification with categorization and producing personas as a grounded theory that is describing the classes of users for a computer-based system. The paper summarizes a case study of an iterative application of this method to arrive at a set of user-space-covering personas for privacy and security features in computer-based systems, and it shows how these personas can be used to inform RE for these features. The full case study and the descriptions of the personas are found in the appendices.

Free-space gestural systems are faced with two major issues: a lack of subtlety due to explicit mid-air arm movements, and the highly effortful nature of such interactions. With an ever-growing ubiquity of interactive devices, displays, and appliances with non-standard interfaces, lower-effort and more socially acceptable interaction paradigms are essential. To address these issues, we explore at-one's-side gestural input. Within this space, we present the results of two studies that investigate the use of side-gesture input for interaction. First, we investigate end-user preference through a gesture elicitation study, present a gesture set, and validate the need for dynamic, diverse, and variable-length gestures. We then explore the feasibility of designing such a gesture recognition system, dubbed WatchTrace, which supports alphanumeric gestures of up to length three with an average accuracy of up to 82%, providing a rich, dynamic, and feasible gestural vocabulary.

Why do we care if our teammates are not human? This study seeks to uncover whether or not the perception of other players as human or artificial entities can influence player experience. We use both deception and a between-participants blind study design to reduce bias in our experiment. Our qualitative results show that people do care about the perceived nature of other players, even though they are not always able to correctly identify them as human or as non-player character teammates. Interview data suggest believing that one is playing with other humans can positively affect a player's subjective experience. Furthermore, our qualitative results indicate that players view their non-player character teammates as humanized entities, but adopt a neo-feudalistic (i.e., an unequal rights) view of them. Based on our results, we establish game design guide- lines for non-player character teammates leading to stronger, emotional human-computer relationships in video games.

In 1997, Accot and Zhai presented seminal work analyzing the temporal cost and instantaneous speed profiles associated with movement along constrained paths. Their work posited and validated the emph{steering law}, which described the relationship between path constraint, path length and the temporal cost of path traversal using a computer input device (e.g. a mouse). In this paper, we argue that the steering law fails to correctly model constrained paths of varying, arbitrary curvature, propose a new form of the law that accommodates these curved paths, and empirically validate our model.

We explore how users approach and define personal space on large, public displays. Our results show that users of public displays use one of two strategies for visual search tasks: minimizers create a small window and work up close to the display, and maximizers expand content to its full resolution and work at a distance. We show that these interaction styles match predicted 'personal' and 'subtle' interaction zones, characterize typical width and height requirements for these interactions, and show that these requirements are independent of the on-screen content's dimensions. Finally, we suggest practical guidelines for defining workspaces during personal and subtle interaction on large, public displays.

Designing difficulty levels in platformer games is a challenge for game designers. It is important because design decisions that affect difficulty also directly affect player experience. Consequently, design strategies for balancing game difficulty are discussed by both academics and game designers. In this paper, we study how manipulating the following design decisions, commonly found in platformers, moderates difficulty: Scroll Speed, Target Size, Jump Task Complexity, and Perspective. Results for Scroll Speed and Target Size indicate that errors increase as speed increases and platform size decreases. However, results for jump task complexity demonstrate a separation of errors from task complexity. Specifically, while double-jump tasks are harder than single-jump tasks, triple-jump tasks appear to be as difficult as double-jump tasks. Additionally, the study demonstrates how changes in perspective affect the errors made by players in gameplay. The study results are applicable both to automatic level generation and dynamic difficulty adjustment in platformer games.

Bi-level thresholding is a motion gesture recognition technique that mediates between false positives, and false negatives by using two threshold levels: a tighter threshold that limits false positives and recognition errors, and a looser threshold that prevents repeated errors (false negatives) by analyzing movements in sequence. In this paper, we examine the effects of bi-level thresholding on the workload and acceptance of end-users. Using a wizard-of-Oz recognizer, we hold recognition rates constant and adjust for fixed versus bi-level thresholding. Given identical recognition rates, we show that systems using bi-level thresholding result in significant lower workload scores on the NASA-TLX and accelerometer variance. Overall, these results argue for the viability of bi-level thresholding as an effective technique for balancing between false positives, recognition errors and false negatives.

While machine learning is a powerful tool for the analysis and classification of complex real-world datasets, it is still challenging, particularly for developers with limited expertise, to incorporate this technology into their software systems. The first step in machine learning, data labeling, is traditionally thought of as a tedious, unavoidable task in building a machine learning classifier. However, in this paper, we argue that it can also serve as the first opportunity for developers to gain insight into their dataset. Through a Label-and-Learn interface, we explore visualization strategies that leverage the data labeling task to enhance developers' knowledge about their dataset, including the likely success of the classifier and the rationale behind the classifier's decisions. At the same time, we show that the visualizations also improve users' labeling experience by showing them the impact they have made on classifier performance. We assess the visualizations in Label-and-Learn and experimentally demonstrate their value to software developers who seek to assess the utility of machine learning during the data labeling process.

In information retrieval and information visualization, hierarchies are a common tool to structure information into topics or facets, and network visualizations such as knowledge graphs link related concepts within a domain. In this paper, we explore a multi-layer extension to knowledge graphs, hierarchical knowledge graphs (HKGs), that combines hierarchical and network visualizations into a unified data representation. Through interaction logs, we show that HKGs preserve the benefits of single-layer knowledge graphs at conveying domain knowledge while incorporating the sense-making advantages of hierarchies for knowledge seeking tasks. Specially, this paper describes our algorithm to construct these visualizations, analyzes interaction logs to quantitatively demonstrate performance parity with networks and performance advantages over hierarchies, and synthesizes data from interaction logs, inter- views, and thinkalouds on a testbed data set to demonstrate the utility of the unified hierarchy+network structure in our HKGs.

Multi-touch interaction on mobile devices typically consists of taps and swipes Simple gestures that don't exploit the full range of technical and human capabilities. Some devices offer more sophisticated, expert-level gestures, but these haven't yet been widely adopted. To determine what factors might be preventing widespread adoption of expert-level gestures, we examine the use of multi-tasking features on the iPad as representative of widely available, but optional, expert-level gestures. We survey a group of 106 iPad users about their device habits, awareness of expert-level gestures, and how they are used in everyday tasks. We determine that users have a fairly high awareness of expert-level gestures, and willingness to perform them, but often find them difficult to discover and challenging to learn. We conclude with design implications for future research.

In the age of the internet, the ability to fully control online information about oneself is no longer possible; once information is online, it is easily disseminated and quickly becomes part of the internet archive. In this paper, we explore data from a set of in-person questionnaires and contextual interviews with both searchers and searched individuals and an additional data set from an online survey of internet users to identify strategies for vetting, managing, and interpreting content online. We see a desire for agency, a contextualization of information based on social context and temporal context, and a desire to constrain the exploration of online content to a relevant subset of information when searching others, i.e. a desire for selectiveness. We offer an evocative design sketch to highlight these issues and foster a discussion of emerging socio-cultural norms around online search behavior.

We describe the design and evaluation of a freehand, smartwatch-based, mid-air pointing and clicking interaction technique, called Watchpoint. Watchpoint enables a user to point at a target on a nearby large display by moving their arm. It also enables target selection through a wrist rotation gesture. We validate the use of Watchpoint by comparing its performance with two existing techniques: Myopoint, which uses a specialized forearm mounted motion sensor, and a camera-based (Vicon) motion capture system. We show that Watchpoint is statistically comparable in speed and error rate to both systems and, in fact, outperforms in terms of error rate for small (high Fitts's ID) targets. Our work demonstrates that a commodity smartwatch can serve as an effective pointing device in ubiquitous display environments.

One challenge with modern smartwatches is text input. In this paper we explore the use of gestural interaction with a smartwatch to support text input. The inertial measurement unit of a smartwatch is used to capture gestural interaction by a user, and an external display is used to provide feedback. We examine two specific variants of gesture keyboards: the swype keyboard common on modern smartphones and the cirrin keyboard, a gestural keyboard that supports character input via directional gestures. We show, first, that freearm gestural input as sensed by a smartwatch exhibits similar efficiency as freearm gestural input sensed by motion capture systems. As well, we show that the smartwatch inertial measurement unit can support text input on ubiquitous computing displays.

A primary goal of research in usable security and privacy is to understand the differences and similarities between users. While past researchers have clustered users into different groups, past categories of users have proven to be poor predictors of end-user behaviors. In this paper, we perform an alternative clustering of users based on their behaviors. Through the analysis of data from surveys and interviews of participants, we identify five user clusters that emerge from end-user behaviors-Fundamentalists, Lazy Experts, Technicians, Amateurs and the Marginally Concerned. We examine the stability of our clusters through a survey-based study of an alternative sample, showing that clustering remains consistent. We conduct a small-scale design study to demonstrate the utility of our clusters in design. Finally, we argue that our clusters complement past work in understanding privacy choices, and that our categorization technique can aid in the design of new computer security technologies.

This paper looks at the activities of environmentally minded technology designers and provides an account of how these designers think, behave and differ. In contrast to traditional designers, green designers appear to: (1) Bias decisions (2) Design from a deep, personal ethos; (3) Accept 'not knowing' as a part of the design process; (4) Rely on alternative ways of knowing, and; (5) Shift roles as needed throughout the design process. While a superficial treatment of these differences might seem to disenfranchise green designers, we show that an analysis of green design as a discursive practice highlights how, through engagement with others, green design can enhance pro-environmental dialog and enact meaningful change.

We describe an empirical study that explores how users establish and use personal space around large public displays (LPDs). Our study complements field studies in this space by more fully characterizing interpersonal distances based on coupling and confirms the use of on-screen territories on vertical displays. Finally, we discuss implications for future research: limitations of proxemics and territoriality, how user range can augment existing theory, and the influence of dis- play size on personal space.

In exploratory search, how information is presented to the user and how the user interacts with the presented information heavily influence the user's success. In this paper, we examine two different spatial representations of search results: knowledge graphs and hierarchical trees. Through interaction logs we show that knowledge graphs can effectively reduce the need to read source content with no reduction in the quality of the information gathered by the user. Through qualitative interviews and thinkalounds we explore factors that influence user perception of different search results representations including biases, task, perceived structure of the data, and problem-solving approach. Overall, these results enhance our understanding of the role each of these representations can play in information seeking.

Clutching is usually assumed to be triggered by a lack of physical space and detrimental to pointing performance. We conduct a controlled experiment using a laptop trackpad where the effect of clutching on pointing performance is dissociated from the effects of control-to-display transfer functions. Participants performed a series of target acquisition tasks using typical cursor acceleration functions with and without clutching. All pointing tasks were feasible without clutching, but clutch-less movements were harder to perform, caused more errors, required more preparation time, and were not faster than clutch-enabled movements.

When it comes to environmental sustainability, the time that electricity is consumed matters. For example, using an air conditioner on a hot summer afternoon as the power grid is strained necessitates the use of more polluting sources to meet demand. In this paper, we analyze end-user response to two utility-driven conservation programs in Ontario, Canada: Time-of-Use pricing and the peaksaver program. We find that time-of-use pricing encourages shifting some electricity demand, but only when it is convenient. We also find that while potentially effective at a larger scale, the peaksaver program in its current form is unattractive to participants. These results are discussed in the context of Fogg's Behavior Model for Persuasive Design, which allows us to explore the design space for improvement to these programs and ground our design implications for the design of technologies to encourage reduction of peak electricity demand.

We develop and formally evaluate a metaphor for smartphone interaction with 3D environments: Tiltcasting. Under the Tiltcasting metaphor, users interact within a rotatable 2D plane that is "cast" from their phone's interactive display into 3D space. Through an empirical validation, we show that Tiltcasting supports efficient pointing, interaction with occluded objects, disambiguation between nearby objects, and object selection and manipulation in fully addressable 3D space. Our technique out-performs existing target agnostic pointing implementations, and approaches the performance of physical pointing with an off-the-shelf smartphone.

Recent work by Mandryk and Lough demonstrated that the movement time of Fitts-style pointing tasks varies based on intended use of a target, suggesting major implications for HCI research that models pointing using Fitts' Law. We replicate the study of Mandryk and Lough to determine exactly how and why observed movement times vary. We demonstrate that any variation in movement time is the result of differences in additive factors (a in Fitts' equation) and can be attributed to changes in the time a user spends over their primary target.

Although touchscreen mobile phones are widely used for recording informal text notes (e.g., grocery lists, reminders and directions), the lack of efficient mechanisms for combining informal graphical content with text is a persistent challenge. In this paper, we present InkAnchor, a digital ink editor that allows users to easily create ink-based notes by finger drawing and writing on a mobile phone touchscreen. InkAnchor incorporates flexible anchoring, focus-plus-context input, content chunking, and lightweight editing mechanisms to support the capture of informal notes and annotations. We describe the design and evaluation of InkAnchor through a series of user studies, which revealed that the integrated support enabled by InkAnchor is a significant improvement over current mobile note taking applications on a range of mobile note-taking tasks.

When using motion gestures, 3D movements of a mobile phone, as an input modality, one significant challenge is how to teach end users the movement parameters necessary to successfully issue a command. Is a simple video or image depicting movement of a smartphone sufficient? Or do we need three-dimensional depictions of movement on external screens to train users? In this paper, we explore mechanisms to teach end users motion gestures, examining two factors. The first factor is how to represent motion gestures: as icons that describe movement, video that depicts movement using the smartphone screen, or a Kinect-based teaching mechanism that captures and depicts the gesture on an external display in three-dimensional space. The second factor we explore is recognizer feedback, i.e. a simple representation of the proximity of a motion gesture to the desired motion gesture based on a distance metric extracted from the recognizer. We show that, by combining video with recognizer feedback, participants master motion gestures equally quickly as end users that learn using a Kinect. These results demonstrate the viability of training end users to perform motion gestures using only the smartphone display.

Despite its popularity, the classic pinch-to-zoom gesture used in modern multi-touch interfaces has drawbacks: specifically, the need to support an extended range of scales and the need to keep content within the view window on the display can result in the need to clutch and pan. In two formative studies of unimanual and bimanual pinch-to-zoom, we found patterns: zooming actions follows a predictable ballistic velocity curve, and users tend to pan the point-of-interest towards the center of the screen. We apply these results to design an enhanced zooming technique called Pinch-to-Zoom-Plus (PZP) that reduces clutching and panning operations compared to standard pinch-to-zoom behaviour.

Classroom response systems (CRS) have been shown to dramatically improve a variety of learning outcomes in science and engineering education. Their adoption in research university classrooms has been very slow. This paper postulates a contributing factor to the lack of widespread adoption of CRS in that context and introduces a new CRS designed to mitigate the identified factor. The CRS, NetClick, is built on the premise that the requirement of authoring new content prevents many Professors from using existing CRS. After describing the operation and advantages of NetClick to other CRS, we describe initial feedback from Computer Science Professors on its use for converting existing teaching slides into interactive content.

Handwriting recognizers occasionally misinterpret digital ink input, requiring users to compare their ink input and the recognizer output to identify and correct errors. Technologies like Anoto pens can make this error discovery and correction task more difficult, because verification of recognizer output may occur many hours after data input and may involve the verification of many documents. In this paper, we explore the design space for error discovery aids geared toward "out-of-the-moment" verification. We present three discovery techniques, a visual proximity technique, a multimodal technique, and a character manipulation technique, and analyze the performance of our techniques. Experimental results show that the visual proximity technique outperforms all others in number of errors caught, and is also significantly better than a control technique. This paper is the first experimental study of techniques that aid in the discovery of handwriting recognition errors.

Computer Algebra Systems and matrix-based mathematics packages provide sophisticated functionality to assist with mathematical problem solving. However, despite their widespread adoption, little work in the human–computer interaction community has examined the extent to which these computational tools support expert problem solving. In this article, we report findings from a qualitative study comparing and contrasting the work practices and software use of practicing researchers in mathematics and engineering who share the goal of developing and defending new mathematical formulations. Our findings indicate that although computational tools are used by both groups to support their work, current mathematics software plays a relatively minor, somewhat untrusted role in the process. Our data suggest that five primary factors limit the applicability of current mathematics software to expert work practices: (a) a lack of transparency in how current software derives its computed results; (b) the lack of clearly defined operational boundaries indicating whether the system can meaningfully operate on the user's input (whether expressions or data); (c) the need for free-form two-dimensional input to support annotations, diagrams, and in-place manipulation of objects of interest; (d) the potential for transcription problems when switching between physical and computational media; and (e) the need for collaboration, particularly in early stages of problem solving. Each of these issues suggests a concrete direction for future improvement of mathematics software for experts. These findings also have more general implications for the design of computational systems intended to support complex problem solving.

We investigate behaviours on, and around, large vertical displays during concurrent usage. Using an observational field study, we identify fundamental patterns of how people use existing public displays: their orientation, positioning, group identification, and behaviour within and between social groups just-before, during, and just-after usage. These results are then used to motivate a controlled experiment where two individuals or two pairs of individuals complete tasks concurrently on a simulated large vertical display. Results from our controlled study demonstrates that vertical surface territories are similar to those found in horizontal tabletops in function, but their definitions and social conventions are different. In addition, the nature of use-while-standing systems results in more complex and dynamic physical territories around the display. We show that the anthropological notion of personal space must be slightly refined for application to vertical displays.

Smartphones are frequently used in environments where the user is distracted by another task, for example by walking or by driving. While the typical interface for smartphones involves hardware and software buttons and surface gestures, researchers have recently posited that, for distracted environments, benefits may exist in using motion gestures to execute commands. In this paper, we examine the relative cognitive demands of motion gestures and surface taps and gestures in two specific distracted scenarios: a walking scenario, and an eyes-free seated scenario. We show, first, that there is no significant difference in reaction time for motion gestures, taps, or surface gestures on smartphones. We further show that motion gestures result in significantly less time looking at the smartphone during walking than does tapping on the screen, even with interfaces optimized for eyes-free input. Taken together, these results show that, despite somewhat lower throughput, there may be benefits to making use of motion gestures as a modality for distracted input on smartphones.

Designers of motion gestures for mobile devices face the difficult challenge of building a recognizer that can separate gestural input from motion noise. A threshold value is often used to classify motion and effectively balances the rates of false positives and false negatives. We present a bi-level threshold recognition technique designed to lower the rate of recognition failures by accepting either a tightly thresholded gesture or two consecutive possible gestures recognized by a relaxed model. Evaluation of the technique demonstrates that the technique can aid in recognition for users who have trouble performing motion gestures. Lastly, we suggest the use of bi-level thresholding to scaffold the learning of gestures.

We propose spatial correspondence targeting to support interaction between devices in multi-device environments when network connectivity fails. In spatial correspondence targeting, for a given target on surface A, an end-user envisions the relative position of that target on surface B and interacts on surface B without any visual depiction of the target on surface B. The targeting task relies on human spatial visualization ability, i.e. the ability to relate the spatial position of objects on one display to their scale-invariant position on another display. We provide experimental evidence that demonstrates that users may be able to target up to 25 discrete targets using a smartphone screen even in the absence of a depiction of the target on the smartphone screen. We argue that the accuracy of spatial correspondence targeting is sufficient for the technique to have many practical applications.

Modern smartphones contain sophisticated sensors to monitor three-dimensional movement of the device. These sensors permit devices to recognize motion gestures - deliberate movements of the device by end-users to invoke commands. However, little is known about best-practices in motion gesture design for the mobile computing paradigm. To address this issue, we present the results of a guessability study that elicits end-user motion gestures to invoke commands on a smartphone device. We demonstrate that consensus exists among our participants on parameters of movement and on mappings of motion gestures onto commands. We use this consensus to develop a taxonomy for motion gestures and to specify an end-user inspired motion gesture set. We highlight the implications of this work to the design of smartphone applications and hardware. Finally, we argue that our results influence best practices in design for all gestural interfaces.

Recognizing hand drawn mathematical matrices on tablet computers has proven to be a particularly challenging task. While individual expression recognition can be simplified by assuming the entire content is a single semantic construct, a single math expression, a matrix is composed of multiple expressions arranged in rows and columns. These expressions must first be segmented into matrix elements, and then each individual matrix element expression must be recognized. In this work, we show how a simple algorithm on in-air (i.e. non-inking) strokes can be used to analyze the drawing order of a matrix. Once the drawing order is recognized, we show how outlier analysis on in-air packets gives rapid, reliable segmentation of matrix elements.

Both handwriting recognition systems and their users are error prone. Handwriting recognizers make recognition errors, and users may miss those errors when verifying output. As a result, it is common for recognized documents to contain residual errors. Unfortunately, in some application domains (e.g. health informatics), tolerance for residual errors in recognized handwriting may be very low, and a desire might exist to maximize user accuracy during verification. In this paper, we present a technique that allows us to measure the performance of a user verifying recognizer output. We inject artificial errors into a set of recognized handwritten forms and show that the rate of injected errors and recognition errors caught is highly correlated in real time. Systems supporting user verification can make use of this measure of user accuracy in a variety of ways. For example, they can force users to slow down or can highlight injected errors that were missed, thus encouraging users to take more care.

Target expansion is a pointing facilitation technique where the user's target, typically an interface widget, is dynamically enlarged to speed pointing in interfaces. However, with densely packed (tiled) arrangements of widgets, interfaces cannot expand all potential targets; they must, instead, predict the user's desired target. As a result, mispredictions will occur which may disrupt the pointing task. In this paper, we present a model describing the cost/benefit of expanding multiple targets using the probability distribution of a given predictor. Using our model, we demonstrate how the model can be used to infer the accuracy required by target prediction techniques. The results of this work are another step toward pointing facilitation techniques that allow users to outperform Fitts' Law in realistic pointing tasks.

Computer Algebra Systems (CAS) provide sophisticated functionality to assist with mathematical problem solving. Despite their widespread adoption, however, little work in the HCI community has examined the extent to which these computational tools support domain experts. In this paper, we report findings from a qualitative study investigating the work practices and tools of nine mathematicians in a research setting. Counter to our expectations, our data suggests that computational tools play only a minor role in their workflow, with the limited use of CAS owing primarily to four factors: (1) the need for transparency in CAS's reasoning to explain computed results; (2) the problem of rigidity and formality in CAS's input/output style dialogue; (3) the need for 2D input to support a wide range of annotations, diagrams, and in-place manipulation of objects of interest; and (4) the need for collaboration, particularly in early stages of problem solving. While grounded in the study of mathematicians, these findings (particularly the first) have implications for the design of computational systems intended to support complex problem solving.

Today's handheld gaming systems allow players to engage in multiplayer games via ad-hoc, wireless networking. They are also now sufficiently commonplace that it is possible to study how portability and ad-hoc wireless networking have affected the social gaming practices of owners of these systems. In this paper, we report findings from a qualitative study investigating the collocated multiplayer gaming practices of Nintendo DS owners. Based on interviews of nine DS owners and observations of three organized gaming events, we identified three major themes surrounding the social, multiplayer gaming practices of Nintendo DS users: renegade gaming, or the notion that users reappropriate contexts traditionally hostile to game play; pragmatic and social barriers to the formation of ad-hoc pick-up games, despite a clear desire for multiplayer, collocated gaming; and private gaming spheres, or the observation that the handheld device's form factor creates individual, privatized gaming contexts within larger social contexts. These findings lead to a set of implications for the design of future handheld gaming systems.

Many on-line (interactive) mathematics recognition systems allow the creation of typeset equations, normally in LaTeX, but they do not support mathematical problem solving. In this paper, we present MathBrush, a system that allows users to draw math input using a pen-input device on a tablet computer, recognizes the math expression, and then supports mathematical transformation and problem solving using back-end Computer Algebra Systems (CAS). We describe the architecture of the MathBrush system, which includes modules that support symbol recognition, semantic analysis, the transfer of recognized expressions to back-end CAS, and interface techniques for interacting with CAS output. We also identify unique challenges associated with recognition for math problem solving, such as the need for deeper semantic analysis than is required by LATEX, and the need to deal with ambiguities in user input. Our experiences serve to inform researchers seeking to design interactive mathematics recognition systems geared toward mathematical problem solving.

This paper introduces kinematic templates, an end-user tool for defining content-specific motor space manipulations in the context of editing 2D visual compositions. As an example, a user can choose the "sandpaper" template to define areas within a drawing where cursor movement should slow down. Our current implementation provides templates that amplify or dampen the cursor's speed, attenuate jitter in a user's movement, guide movement along paths, and add forces to the cursor. Multiple kinematic templates can be defined within a document, with overlapping templates resulting in a form of function composition. A template's strength can also be varied, enabling one to improve one's strokes without losing the human element. Since kinematic templates guide movements, rather than strictly prescribe them, they constitute a visual composition aid that lies between unaided freehand drawing and rigid drawing aids such as snapping guides, masks, and perfect geometric primitives.

Collocation preferences represent the commonly used expressions, idioms, and word pairings of a language. Because collocation preferences arise from consensus usage, rather than a set of well-defined rules, they must be learned on a case-by-case basis, making them particularly challenging for non-native speakers of a language. To assist non-native speakers with these parts of a language, we developed AwkChecker, the first end-user tool geared toward helping non-native speakers detect and correct collocation errors in their writing. As a user writes, AwkChecker automatically flags collocation errors and suggests replacement expressions that correspond more closely to consensus usage. These suggestions include example usage to help users choose the best candidate. We describe AwkChecker's interface, its novel methods for detecting collocation errors and suggesting alternatives, and an early study of its use by non-native English speakers at our institution. Collectively, these contributions advance the state of the art in writing aids for non-native speakers.

Recently proposed novel interaction techniques such as cursor jumping [1] and target expansion for tiled arrangements [13] are predicated on an ability to effectively estimate the endpoint of an input gesture prior to its completion. However, current endpoint estimation techniques lack the precision to make these interaction techniques possible. To address a recognized lack of effective endpoint prediction mechanisms, we propose a new technique for endpoint prediction that applies established laws of motion kinematics in a novel way to the identification of motion endpoint. The technique derives a model of speed over distance that permits extrapolation. We verify our model experimentally using stylus targeting tasks, and demonstrate that our endpoint prediction is almost twice as accurate as the previously tested technique [13] at points more than twice as distant from motion endpoint.

PocketPad is an information management system geared toward university students. The system is designed to support the capture, storage, browsing, editing and organization of handwritten notes via the complementary use of digital pens to capture information, handheld computers to browse and store prior information, and digital pens and handheld computers in combination to edit and organize information. Desktop computer software synchronizes the information from the digital pen and Pocket PC during editing and reorganization. Through the combined use of different hardware components supported by our software, we describe a system that bridges the paper-electronic information divide in mobile contexts. The system relies on human-in-the-loop techniques coupled with stroke timing to simplify coordination of content from different sources, rather than resorting to complex and often error-prone computer document recognition algorithms.

This paper presents the synergistic use of handheld computers and desktop computers as tools for grade school teachers to record observational data from their classrooms. The system uses handheld devices to tersely log events in situ, and allows later reflection and elaboration through annotation of the logged events on desktop computer. The terse, ephemeral capture of events coupled with post-hoc annotation has been novelly exploited to design a system that supports teachers' development as reflective practitioners and enables richer capture of grade school student behaviour and performance.

In pen-tablet input devices modes allow overloading of the electronic stylus. In the case of two modes, switching modes with the non-preferred hand is most effective [12]. Further, allowing temporal overlap of mode switch and pen action boosts speed [11]. We examine the effect of increasing the number of interface modes accessible via non-preferred hand mode switching on task performance in pen-tablet interfaces. We demonstrate that the temporal benefit of overlapping mode-selection and pen action for the two mode case is preserved as the number of modes increases. This benefit is the result of both concurrent action of the hands, and reduced planning time for the overall task. Finally, while allowing bimanual overlap is still faster it takes longer to switch modes as the number of modes increases. Improved understanding of the temporal costs presented assists in the design of pen-tablet interfaces with larger sets of interface modes.

Novel key mappings, including chording, character prediction, and multi-tap, allow the use of fewer keys than those on a conventional keyboard to enter text. In this paper, we explore a text input method that makes use of rhythmic mappings of five keys. The keying technique averages 1.5 keystrokes per character for typical English text. In initial testing, the technique shows performance similar to chording and other multi-tap techniques, and our subjects had few problems with basic text entry. Five-key entry techniques may have benefits for text entry in multi-point touch devices, as they eliminate targeting by providing a unique mapping for each finger.

The design of new diagram recognition systems remains a challenging problem. Ideally, recognition systems should accept real-world input, perform robustly, fail gracefully, and be implemented in a timely manner. In reality, the intricacy involved in implementing recognition systems for diagram notations makes this a challenging open problem. One solution to these challenges is the design of middleware to speed the development of robust applications. Middleware takes the form of a framework or toolkit for the creation of applications. This paper describes a retargetable framework which can be used to speed the development of robust interactive sketch recognition systems. The system includes a drawing surface to capture interactively created drawings, a set of generic segmentation routines, a character recognizer, and a common interface for integrating domain-specific components. The framework has been used to construct systems for the recognition of UML, math, and molecular diagrams. Work is on-going on the design of additional generic recognizers of logical structure and spatial layout of diagrams.

This paper offers a solution to the mode problem in computer sketch/notetaking programs. Conventionally, the user must specify the intended "draw" or "command" mode prior to performing a stroke. This necessity has proven to be a barrier to the usability of pen/stylus systems. We offer a novel Inferred-Mode interaction protocol that avoids the mode hassles of conventional sketch systems. The system infers the user's intent, if possible, from the properties of the pen trajectory and the context of the trajectory. If the intent is ambiguous, the user is offered a choice mediator in the form of a pop-up button. To maximize the fluidity of drawing, the user is entitled to ignore the mediator and continue drawing. We present decision logic for the inferred mode protocol, and discuss subtleties learned in the course of its development. We also present results of initial user trials validating the usability of this interaction design.

Unified Modeling Language (UML) diagrams are widely used by software engineers to describe the structure of software systems. Early in the software design cycle, software engineers informally sketch initial UML diagrams on paper or whiteboards. The information provided by these UML diagrams needs to be made available to computer assisted software engineering (CASE) tools. In order to smooth this transition from paper to electronic form, we have developed an online recognition system for UML diagrams. The system accepts input from an electronic whiteboard, a data tablet or a mouse. Efforts have been made to separate the domain-independent and domain-specific parts of the recognition system. The kernel of the system is retargetable, providing a general front end for online recognition of any glyph-based diagram notation. The kernel is extended with UML-specific routines for segmentation, recognition of glyphs, and recognition of glyph relationships.

N-grams are a well-structured knowledge representation that has proven useful in the recognition of textual documents. In this paper, we propose that n-grams can be extended to the domain of graphical documents as well. This has the advantage of providing a regular, easily-maintainable knowledge representation for local constraints, specifically symbol-interaction knowledge, in the graphical domain. To extend the definition of an n-gram from the textual to the graphical domain, we must resolve how to handle directional considerations, how to treat the variety of relations that can occur between image primitives, and how to treat multiple neighbours. We have implemented a prototype system for the application of n-gram knowledge to the recognition of sketch maps. Early results are encouraging. We look forward to further refining this system. N-grams appear to be a promising tool for representing spatial constraints in a form that is useful to the recognition of graphical documents.